One of the most valuable plug-ins I was running under FreeNAS was Crashplan which was responsible for ensuring I had an additional off site backup of my primary data sets. Under FreeNAS you have to run Crashplan in the officially unsupported ‘headless mode’ which until recently has been reliable. Recent changes to FreeNAS and Crashplan have contributed to the point where the stability and reliability of this critical service no longer met my requirements and was a major contributor to my decision to move away from FreeNAS jails altogether.
This guide will go through the steps to create a Linux virtual machine in Proxmox and install & configure Crashplan for reliable operation. Although we will be running a Linux desktop to enable us to interact with the Crashplan client, we will keep the resource requirements down to a reasonable minimum to reduce the burden on the host machine.
I’ll also include how to adopt an existing backup from Crashplan to prevent having to re-upload all your data.
This guide will build on my Proxmox 4.2 baseline installation which ensures the underlying host configuration is configured correctly to support reliable usage including UPS initiated shut-downs and email reporting etc.
Moving away from the headless setup required a desktop to support the Crashplan client. I wanted to keep resource usage down and therefore required a lightweight Linux desktop. I settled on CrunchBangPlusPlus which is an open-source, lightweight Linux desktop built upon Debian Jessie and configured out of the box with Openbox window manager and a number of other useful additions. I looked at more extreme pre-configured desktops such as TinyCore, however the additional effort of configuring TinyCore wasn’t worth the additional effort in terms of additional resource saving.
Download the CrunchBangPlusPlus ISO image from here.
Before we can create a VM based upon CrunchBangPlusPlus, we need to upload the ISO file to Proxmox.
You can use as many cores as your machine offers, because Crashplan has low CPU overheads because I don’t compress or dedupe my uploads, I’ve allocated 2 cores which is plenty. If you make use of compression, it may be worth increasing the core count.
Crashplan is fairly memory intensive, even without dedupe enabled. I’ve allocated 4GB which is enough to manage my 2TB archive. Monitor the Proxmox dashboard summary to see if there is a need to increase the memory made available to the VM.
As per my baseline installation guide, I use a Intel x520 10gig network card for all of my VM hosts to communicate over and hence why I select vmbr1 here. If you are making use of a single connection, this should be set to vmbr0.
Confirm details and Click Finish to proceed.
After a few seconds your virtual machine should have been created. Navigate to ServerView > proxmox > 103 (crashplan) to see a Summary of your new VM.
In the VM header, click Start to start the VM.
You can then click on Console and you should be presented with another window with the Debian installer boot menu.
CrunchBangPlusPlus is built on top of Debian, we’ll run through the baseline install of Debian.
Select Graphical Install
Select your preferred installation language and click Continue
Select your location and click Continue
Select your keymap and click Continue
Some files will now be copied to your boot disk and hardware will be detected and configured…
Enter a hostname, in this case I used ‘crashplan’. Click Continue to proceed.
Enter full name of the user, in my case I use ‘cp’ as the system user and click Continue
Select username for the user we just created, again I’ll use ‘cp’. Click Continue to proceed.
Select a strong password and enter it twice. Click Continue to proceed.
Set desired timezone and click Continue to proceed.
The system disk will now be detected and we can commence with partitioning. Given this is a simple Linux install only running the Crashplan service, there’s no need for anything fancy. Given we only allocated 8GB to this disk we can go ahead and use all of the disk. A simple guided install will take care of creating the primary partition and a small swap space. if you start to notice swap being used it is likely worth increasing the amount of system ram to reduce this.
Select Guided - use entire disk and click Continue to proceed
Our system only has one disk so go ahead and select ‘vda’ and click Continue to proceed.
We’ll go ahead and install all the files into a single partition. There’s no need to complicate this setup given its limited use. Select ‘All files in one partition’ and click Continue to proceed.
Check you are happy with the partitioning information and if so, click Continue to proceed. You can see the system has provided a small 401.6MB swap space and the rest of the disk space is assigned to the primary partition.
Highlight ‘Finish partitioning and write changes to disk’ and click Continue to proceed.
A final confirmation before the disk contents are destroyed, select ‘Yes’ and click Continue to proceed.
Partitioning will now proceed… After a short wait we can move n to configure the package manager.
We’ll need to keep our system updated and install some programs so configure the package manager by highlighting an area and click Continue to proceed.
and again, select the Debian archive mirror close to you.
Click Continue to proceed.
In order for our VM to boot, we need to install a grub boot loader on the primary system disk. As this is the only OS, we can go ahead and install it to the Master Boot Record. Select ‘Yes’ and click Continue to proceed.
Highlight our only drive, ‘/dev/vda’ and click Continue to proceed to write the grub loader to our drive.
Congratulations, you now have a basic working Linux VM. Click Continue to reboot, CrunchBangPlusPlus should load through a login screen where you can enter your username and password to login to the desktop environment.
At this stage we have a functioning Linux desktop. Feel free to play around with it to familiarise yourself with it. You can see in the system info display on the background CrunchBangPlusPlus impressively runs in less than 200MB of RAM and takes 2.2GB of disk space. It is possible ot shrink it further but its so lightweight I didn’t think it was worth the effort.
Close the cpbb-welcome window, or enter q to quit it. The configurator offers some neat additional configuration steps but largely they are superfluous for our Crashplan needs. The important steps are already captured in this guide.
Right click on the CBPP desktop and open a terminal prompt. We’ll do the majority of our configuration in the terminal.
Lets start off by updating our Debian base install with the following commands.
sudo -apt-get update sudo apt-get dist-upgrade
Now lets configure our server so it pulls the correct time from our local NTP server, in my case my pfSense router.
In the terminal window, enter
sudo nano /etc/ntp.conf and edit the section which details the NTP servers. Comment out the default servers by adding a hash in front of them and add your local source, in my case this is my pfSense router which is at IP address 192.168.20.1.
# pool.ntp.org maps to about 1000 low-stratum NTP servers. Your server will # pick a different set every time it starts up. Please consider joining the # pool: # server 0.debian.pool.ntp.org iburst dynamic # server 1.debian.pool.ntp.org iburst dynamic # server 2.debian.pool.ntp.org iburst dynamic # server 3.debian.pool.ntp.org iburst dynamic server 192.168.20.1 iburst dynamic
Start the NTP service with
sudo /etc/init.d/ntp start and verify it started correctly with
sudo /etc/init.d/ntp status.
Verify your server is set correctly by entering
ntpq -p, for example
$ ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== *192.168.20.1 18.104.22.168 2 u 10 256 377 0.213 1.114 0.127
The Promxmox host needs the ability to shutdown virtual machines correctly and safely. This is accomplished via ACPI commands. Without ACPI being installed and configured Proxmox cant initiate a safe shutdown of the VM, either automatically from a UPS event, or from user interacting with the Proxmox interface. Without ACPI correctly configured you will see errors in the VM task list such as
TASK ERROR: command '/bin/nc6 -l -p 5900 -w 10 -e '/usr/sbin/qm vncproxy 103 2>/dev/null'' failed: exit code 1
Enter the command
sudo apt-get install acpi-support to install ACPI support.
We will edit the power button event to initiate the shutdown command. Enter
sudo nano /etc/acpi/events/powerbtn-acpi-support and edit the file so it contains.
event=button[ /]power action=/sbin/shutdown -h -P now
then start the service by entering
$ sudo /etc/init.d/acpid start [ ok ] Starting acpid (via systemctl): acpid.service.
you can check it started correctly by
$ sudo /etc/init.d/acpid status ● acpid.service - ACPI event daemon Loaded: loaded (/lib/systemd/system/acpid.service; disabled) Active: active (running) since Sat 2016-07-23 18:01:55 EDT; 50s ago Main PID: 443 (acpid) CGroup: /system.slice/acpid.service └─443 /usr/sbin/acpid Jul 23 18:01:55 crashplan acpid: starting up with netlink and the input layer Jul 23 18:01:55 crashplan acpid: 27 rules loaded Jul 23 18:01:55 crashplan acpid: waiting for events: event logging is off
You should try to initiate a VM client shutdown from the Proxmox user interface and verify everything works correctly. Hopefully the VM will shutdown promptly and cleanly reporting ‘OK’ in the event log rather than the previous ‘Error, VM quit/powerdown fail’.
Due to the importance of proper shutdown procedures in time of power outage, I recommend verifying the full UPS shutdown stack.
I originally mounted my shares via fstab however this caused me some issues. In my experience, it appears fstab runs before the network is completely up and ready to accept connection attempts and therefore some folders may not mount correctly. Assuming you have created the root folder for the mount to attach to, Crashplan views this as an empty folder and politely removes all the associated data from the cloud assuming you have deleted it locally!
I now use autofs to mount drives which has been a lot more reliable.
I have a number of FreeNAS folders shared which I back up daily to Crashplan’s cloud. I’ll focus on one specific folder called ‘Data’ in this guide but it should be simple to extrapolate to all the folders you need to backup.
Install autofs, CIFS and smbclient packages with the following command.
sudo apt-get install autofs smbclient cifs-utils
I like to verify SMB by running
smbclient -L <hostname> where hostname is the name of your server hosting the shares, for example.
$ smbclient -L freenas Domain=[WORKGROUP] OS=[Windows 6.1] Server=[Samba 4.3.6-GIT-UNKNOWN] Sharename Type Comment --------- ---- ------- Data Disk IPC$ IPC IPC Service (FreeNAS Server) Domain=[WORKGROUP] OS=[Windows 6.1] Server=[Samba 4.3.6-GIT-UNKNOWN] Server Comment --------- ------- FREENAS FreeNAS Server NGUVUWIN8 Win8 Workgroup Master --------- ------- WORKGROUP FREENAS
Edit autofs master configuration file by entering
sudo nano /etc/auto.master
Add the following to the bottom of the file,
so the file looks like this
# Sample auto.master file # This is an automounter map and it has the following format # key [ -mount-options-separated-by-comma ] location # For details of the format look at autofs(5). # #/misc /etc/auto.misc # # NOTE: mounts done from a hosts map will be mounted with the # "nosuid" and "nodev" options unless the "suid" and "dev" # options are explicitly given. # #/net -hosts # # Include /etc/auto.master.d/*.autofs # +dir:/etc/auto.master.d # # Include central master map if it can be found using # nsswitch sources. # # Note that if there are entries for /net or /misc (as # above) in the included master map any keys that are the # same will not be seen as the first read key seen takes # precedence. # +auto.master /mnt /etc/auto.smb-shares
To avoid storing the username and passwords of the shared folders in the configuration file, we’ll create a credential file by entering
sudo nano ~/.smbcredentials
this file should be populated only with the username and password used for accessing the shared folder, no spaces either side of the ‘=’ sign. For example
Reduce the access rights to this file by entering
sudo chmod 600 ~/.smbcredentials.
You will need a credential file for each share you access which doesn’t share the same username and password.
We’ll now create the configuration file that specifies the folders we want autofs to mount for us by entering
sudo nano /etc/auto.smb-shares
Enter the following line into this file
Data -fstype=cifs,ro,credentials=/home/cp/..smbcredentials ://freenas/Data
Data is the name of the folder the line refers to.
fstype is the share filesystem type, in our case, CIFS
ro mounts the share as read-only
credentials specifies the file containing the username and password to use for authorisation.
://freenas/Data is the network location of the share
It should be simple enough to extrapolate form that line into the other shares you would like to access.
Start autofs by entering
sudo /etc/init.d/autofs start and verify its status with
sudo /etc/init.d/autofs status
$ sudo /etc/init.d/autofs start [ ok ] Starting autofs (via systemctl): autofs.service. $ sudo /etc/init.d/autofs status ● autofs.service - LSB: Automounts filesystems on demand Loaded: loaded (/etc/init.d/autofs) Active: active (running) since Sun 2016-07-24 00:04:54 EDT; 31min ago CGroup: /system.slice/autofs.service └─1998 /usr/sbin/automount --pid-file /var/run/autofs.pid Jul 24 00:04:54 crashplan autofs: Starting automount....
Now ensure we can access the directory we setup,
$ cd /mnt/Data $ ls -l total 0 drwxrwxrwx 283 root root 0 Nov 15 2015 apps drwxrwxrwx 2 root root 0 Jul 17 19:43 FreeNAS drwxrwxrwx 7 root root 0 Oct 18 2014 Linux drwxrwxrwx 122 root root 0 Jul 17 19:41 Macintosh Apps drwxrwxrwx 4 root root 0 Jul 17 19:45 pfSense drwxrwxrwx 3 2001 root 0 Jul 17 19:33 Proxmox
The /mnt/Data folder will unmount itself after a period of activity but will remount whenever it is required by a user or an application.
If you need to have further visibility into what’s going on with autofs, you can turn verbose logging on by editing the
/etc/default/autofs file and setting the LOGGING line to
From Proxmox, Navigate to our Crashplan VM via ServerView > DataCener > Proxmox > 103 (crashplan)
Click on Console to start the VM, enter the username and password to login to the desktop.
Right click on the desktop and open a Web Browser.
Navigate to www.crashplan.com and go go downloads.
Click on ‘Download Crashplan’ which will download the Linux tar file to your downloads folder.
Right click on the desktop and open Terminal. Follow these commands to unpack the archive and commence installation.
$ cd ~/downloads $ ls -l -rw-r--r-- 1 cp cp 47470950 Jul 24 07:43 CrashPlan_4.7.0_Linux.tgz
Unzip the tgz file with
tar -xzvf <filename>, for example
$ tar -xvzf CrashPlan_4.7.0_Linux.tgz crashplan-install/install.sh crashplan-install/uninstall.sh crashplan-install/ crashplan-install/scripts/ crashplan-install/CrashPlan_4.7.0.cpi crashplan-install/EULA.txt crashplan-install/INSTALL crashplan-install/README crashplan-install/install.defaults crashplan-install/scripts/CrashPlan.desktop crashplan-install/scripts/CrashPlanDesktop crashplan-install/scripts/CrashPlanEngine crashplan-install/scripts/crashplan crashplan-install/scripts/run.conf
cd into the extracted file folder and execute the install script.
$ cd crashplan-install $ ./install.sh
Installation will now commence. The default options are ok and summarised below for completeness.
Would you like to switch users and install as root? (y/n) Y
Read the EULA…
Do you accept and agree to be bound by the EULA? (yes/no) yes
What parent directory do you wish to install CrashPlan into? [/usr/local] return to accept defaults
/usr/local/crashplan does not exist. Create /usr/local/crashplan? (y/n) y
What directory do you wish to link the CrashPlan executable to? [/usr/local/bin] return to accept defaults
What directory do you wish to store incoming backup data? [/usr/local/var/crashplan] return to accept defaults
/usr/local/var/crashplan does not exist. Create /usr/local/var/crashplan? (y/n) y
What directory contains your SYSV init scripts? [/etc/init.d] return to accept defaults
What directory contains your runlevel init links? [/etc/rc5.d] return to accept defaults
Is this correct? (y/n) y
Crashplan will now install files, configure Java and start the Crashplan engine.
Back in the CBPP desktop, open the file manager and navigate to
/usr/local/crashplan/bin. Double click on the
As we are going to deal with adopting an existing Crashplan backup, enter your existing username and password and log into the Crashplan client.
Click on ‘adopt’ at the top of the page and select your archive when presented with the option.
Click on archive to adopt the backup. At this stage you will be logged out and you will have to log back in again. Assuming you followed best practice and set a encryption key to encrypt any data uploaded to Crashplan’s cloud, you’ll be notified that the backup isn’t able to run not knowing this code.
Enter your 448bit passphrase and Click Save. Crashplan will now verify the local files with those in the Crashplan cloud.
One of the most common complaints about Crashplan is slow uploading of larger datasets. This I believe is largely affected by the deduplication of data. Thanks to NetworkRockstar it became widely understood what the impact of this deduplication process was and also how to disable it. Crashplan address the topic here themselves and it is worth reading before you consider disabling it. I’ve been running without deduplication and compression for 18 months now without any adverse effects.
To disable deduplication,
nano /usr/local/crashplan/conf/my.service.xml and edit the
dataDeDupAutoMaxFileSizeForWan setting to
1 as follows
Restart Crashplan and you should see improved upload speeds. My most recent backup ran at around 100mbps (my upload cap is 150mbps) for a significant proportion of the 2TB uploaded.