Crashplan just crashed my production server

I use a mix of Crashplan and Dropbox. They both have their uses and are great. Until recently that is. My headless linux boxes stopped backing up to Crashplan earlier this year. A local JVM instance wouldn’t run. The documentation for a headless client is poor if you run into any issues. I ended up installing the 12.04LTS java-common package a few months ago which is still v1.6.

Importantly, to do this you must change /usr/local/crashplan/install.vars
From JAVACOMMON=/usr/local/crashplan/jre/bin/java
To JAVACOMMON=/usr/bin/java

I will say that I didn’t install the java package on two other linux servers and they subsequently just worked themselves out. I think Code42 (Crashplan’s creator) was inundated with failures as they released some software which essentially broke their linux based backbone. Their newer software versions, at least after 4.4.1, seem to upgrade the native jre package properly and require little intervention. So, before you go down the road of manual jre installations, look into this first.

Backups started working again after the manual jre upgrade, some hassle, a lot of searching, finding logs, etc. It was a mess. Fast forward to today. Server went down around 1:30am. We look and find there is zero space left on the drive. Damn. We find that in our /usr/local/crashplan/upgrade directory there are effectively an unlimited number of time stamped update folders. Crashplan was creating a new folder every 30 minutes and then finding that v1.6 of the ubuntu java-common package was installed and then failing out of the process. This continued long enough to spool up to our entire drive size. Code42, you need a reasonability check here!

I resolved by uninstalling and finding all crashplan install files and locations. Once removed, I installed java using the guide here and the useful first comment for default install. http://linuxg.net/how-to-install-oracle-java-jdk-678-on-ubuntu-13-04-12-10-12-04/ . I went through the install process again. During the installation process java-common was updated with compliant java files as Crashplan does not work with the Oracle version apparently.

Then we are back up again. I’ve lost a huge time commitment fixing these issues brought about by Crashplan’s upgrade processed from 3.x to 4.x. Prior upgrades were seamless and required no intervention. I only noticed this problem when our backups stopped working and the subsequent failure when we had a production server go down.

The new version also required me to open some ports in the Ubuntu Firewall. ufw allow from xxx.xxx.xxx.xxx to any port 4242 is the command I used to open a port back to my office for backups to go back and forth between servers. I keep a local copy of crashplan data here on a file server for rapid restores if needed.

3 thoughts on “Crashplan just crashed my production server

  1. Pingback: Crashplan resolution | James Ballenger

  2. Thanks for documenting this. CrashPlan just killed my server too – Arch Linux with the crashplan package installed from AUR. CrashPlan appears to have been attempting to update itself about every 30 seconds, using 33M each time.

    Here’s my /opt/crashplan/upgrade directory size and file listing: https://gist.github.com/ZimbiX/ef05e08ee76a83de58ecde5749c8ff46

    I’ve deleted the contents for now. Investigations continue…

  3. I had a moment of panic as I read your comment, realizing that I haven’t checked this on the affected servers for a while!

    The servers in question all have 6 files now for the upgrade process that are at that 33MB mark. Jan 6, Mar 2, Mar 20 for the .jar containers which is similar to yours. I’ll have to come back to this later to see if it continues to grow but I was hoping that the package would self-manage after the major update.

Leave a Reply

Your email address will not be published. Required fields are marked *