Box Backup

An open source, completely automatic on-line backup system for UNIX.

Project status: Stable, but not feature complete

More information: what it is, and how it works.

Comparisons to other backup systems

Distributed under a BSD license. Please read the license and commentary now.

Author

Ben Summers, ben@fluffy.co.uk

Project status

This project should be considered as stable, but not feature complete. Over 6 months of testing suggests it will be reliable enough to will back up your data, and make it available for restoration.

It has always been possible to retrieve old versions of files. Planned features will implement marked versions of the store to emulute backing up onto sets of tapes in a controlled manner. This will make restoration of old versions much easier, although in the case of retrieving a single old file will not make much difference to the usability as the FTP-like utility is already effective.

Given that this is a backup system, I am being very conservative about when I consider it ready for production use. That said, I trust my data to it! However, as this is quite a young system compared to the alternatives, and operates in a slightly different manner, for production use I must recommend that you regularly verify your backups.

The documentation is not yet complete, and the error messages you get when things go wrong are not as clear as they should be. (see the troubleshooting page.) Please follow the instructions carefully and read everything to avoid problems -- following the instructions on this web site will result in a working installation.

Why online backup?

Because tape is not a reliable solution, unless you...

which is hardly likely to happen in real life.

Backing up to media such as CDROMs is not really a solution either. It needs to be something which just happens without user intervention, and without media which is liable to be corrupted.

So, for the same price as a decent tape backup system, you could buy a server with three big hard discs. And for the money you spend on tapes per year (you do replace them regularly, don't you?), you could host it in a nice data centre somewhere.

We all have internet connections these days, so connectivity isn't really a problem. Although it might be best to archive those static files, like MP3s and images, to CDROM or something.

The costs are even less if you pool resources amongst a group of friends. Remember, you don't have to trust the server admin to not look at your files as they're encrypted. You just have to trust them to make the files available when you need them.

Offer of server space

I would like as many people as possible to test this software. To help, I have a large amount of disc space on a server, and am likely to give out a few Gb to people who ask nicely.

As I'm sure I have mentioned, everything is encrypted so you aren't trusting me with your data. You will of course wish to review the code to check my claims are true.

Mailing list

Please join the project mailing list, boxbackup@fluffy.co.uk, for announcements of new versions and discussion of the system. Join at the sign up page.

Many thanks to Alaric for hosting this list.

Implementation

There are three main elements

Running the store server is a multi-step process, but the backup client is easy.

TLS (SSL revised) is used to encrypt connections, and more importantly, to authenticate servers and clients with both server and client side certificates. Scripts are provided to generate and manage these certificates.

Stored files are encrypted using AES for file data and Blowfish for metadata. This does mean that the one thing you do need to back up off-site and look after is a 1k file containing your keys -- the data on the server is useless without it. But it never changes, so that's OK.

Platforms

I develop under Darwin and deploy on OpenBSD. The software is designed to take advantage of BSD features where useful, but not to rely on them. It should be easily portable to other UNIX like operating systems. I have done ports to the following platforms:

On these platforms, the software compiles and the tests pass. I do all my deployment on OpenBSD, but I know of several people who have had good results under Linux.

More platforms will arrive over time. It takes in general a couple of hours to get it compiling on a BSD-like UNIX, and a bit longer on something like Linux which deviates from the BSD way of doing things.

I welcome patches for other platforms, and volunteers to look after ports for a particular platform. Programmers notes describing how everything works are included in the notes directory within the distribution archive.

Download

Source only distribution. See documentation on this web site for compilation and installation notes. See notes directory inside distribution archive for programmers notes.

boxbackup-0.08.tgz (486k, released 23 September 2004)
(download via SourceForge)

Notes on upgrading from 0.06 to 0.07, 0.05 to 0.06, 0.04 to 0.05

A 0.08 or later server requires all clients to be version 0.08 or later too. Existing backup store accounts are compatible with 0.08, and do not require conversion.

Changes:

0.08 (23 September 2004)
Minor bug fixes
Server stores old versions of files as patches from the current version
Adjust behaviour of client
Works under 64bit systems
Client recovers cleanly from corruption of its working files
Improvements to build system
Add contributed code for building RPMs
0.07 (28 June 2004)
Minor bug fixes
Improvements to build process, including better detection of environment on Linux, especially SuSE 9.x
Fix wrong space used sizes on store with non-RAID (see notes)
Be more paranoid about crypto startup
Extra checking for file modifications
SyncAllowScript to allow temporary disabling of syncs, eg only backup from a laptop on certain network connections
Add force-sync command to bbackupctl, like sync but ignores AllowSyncScript
0.06 (21 May 2004)
Bug fixes
Use AES for file data
Cygwin port of client
Add usage command to bbackupquery
Add delete [yes] command to bbstoreaccounts
Add check [fix] [quiet] command to bbstoreaccounts
Deleted locations in conf file get deleted from server
0.05 (10 Apr 2004)
Bug fixes
Added configurable "max diffing time" parameter, improved speed of diffing algorithm in edge cases
Added support for old versions of OpenSSL -- experimental only (but appears to work fine)
Read errors on files and directories are reported neatly, and no longer abort the backup scan
NetBSD port fixed
Server more tolerant of being unexpectedly aborted
0.04 (16 Mar 2004)
Bug fixes and code clean up
Change to standard BSD license
Expanded error messages
Improved flexibility with backup timing, including the ability to do snapshot backups
Exclude files explictly or with regular expressions
More work arounds for memory issues with STL in gcc 2.95.x (the gcc 3 version is fine)
In bbackupquery, add a 'ls' alias, and include 'help' command to document all commands
The *-config programs are improved and validates options and the configuration more to catch common problems
bbackupd-config excludes keys file if it might be backed up
Add bbackupctl program for controlling bbackupd daemon
bbstoreaccounts takes sizes in blocks, Mb or Gb with unit suffix
Start and end of last sync recorded in bbackupd working dir
bbackupquery compares check last modification time to see if this explains any difference
bbackupd will run a script run to alert administator that store is full (the default script sends an email)
0.03 (27 Jan 2004)
Improvements to the build system, especially configuring for random Linux systems.
Worked round an issue with the gcc 2.95.x libraries which meant that the backup daemon slowly used more and more memory.
Efficiency improvements.
0.02 (22 Jan 2004)
First public release

Documentation

This is basic documentation, which will be improved. Note that configuration uses a number of scripts to do the hard work -- these all output a lot of detailed instructions on exactly what to do. Please read it.

Programmers notes on how the system works can be found within the notes directory inside the distribution archive. In particular, read notes/backup_encryption.txt for details of how the encryption scheme works, and where to look in the code to verify it.

Planned future work

 

© Ben Summers, 2003, 2004

 

Web design by Joe Gillespie