Making full backups in no time every hour
You can use this method to make a backup every hour or every ten minutes if you like. There are many many features you can tune or configure to your own taste – excluding files that are larger than 1GB for example.
So, here the command to make the backup:
rsync -aP –link-dest=PATHTO/$PREVIOUSBACKUP $SOURCE $CURRENTBACKUP
Lets go through the parameters step by step.
-a means Archive and includes a bunch of parameters to recurse directories, copy symlinks as symlinks, preserve permissions, preserve modification times, preserve group, preserve owner, and preserve device files. You usually want that option for all your backups.
-P allows rsync to continue interrupted transfers and show a progress status for each file. This isn’t really necessary but I like it.
–link-dest this is a neat way to make full backups of your computers without losing much space. rsync links unchanged files to the previous backup (using hard-links, see below if you don’t know hard-links) and only claims space for changed files. This only works if you have a backup at hand, otherwise you have to make at least one backup beforehand.
PATHTO/$PREVIOUSBACKUP is the path to the previous backup for linking. Note: if you delete this directory, no other backup is harmed because rsync uses hard-links and the operating system (or filesystem) takes care of releasing space if no link points to that region anymore.
$SOURCE is the directory you’d like to backup.
$CURRENTBACKUP is the directory to which you’d like to make the backup. This should be a non-existing directory.
As said earlier, rsync has many many features. To exclude files over a certain size for example, use the option –max-size (unfortunately this is not available on the rsync version shipped with Mac OS X Leopard). The man page or the documentation can give you plenty of ideas in this direction.
So much for the theory of the most important command for our purpose. Here a simple script that makes an incremental backup every time you call it:
rsync -aP –link-dest=$HOME/Backups/current /path/to/important_files $HOME/Backups/back-$date
rm -f $HOME/Backups/current
ln -s back-$date $HOME/Backups/current
The script creates a file called “back” appended by the current date and time, for example back-2007-11-13T22:03:32 which contains the full backup. Then there is a symbolic link called “current” which points to the most recent directory. This directory-link is used for the –link-dest parameter.
You should look at the –exclude parameter (or better, –exclude-from= parameter) and learn how to exclude certain files or directories from the backup (you shouldn’t backup your backup for example).
The script above only works on the local machine because making links on a remote machine needs some extra work. But not much:
rsync -azP –link-dest=PATHTOBACKUP/current $SOURCE $HOST:PATHTOBACKUP/back-$date
ssh $HOST “rm -f PATHTOBACKUP/current && ln -s back-$date PATHTOBACKUP/current”
The -f parameter for the rm command is used to supress error messages if the current directory is not present, which would in turn prevent the link to be created.
To get that working you either use a public/private key authentication scheme or something else to avoid typing in your password. Another possibility is, of course, to mount the remote file-system on the local computer using the above script.
On my setup the script takes about 6 seconds to synchronize 46968 files and 29GB – this takes 20MB for the file structure (with no actual files to transfer of course). But afterwards, I have a complete backup of my system in a new directory.
On a much bigger setup (1.2 million files and 50GB of data) the backup takes about 30 minutes and takes about 3GB of space (just for links!), so it isn’t exactly free, but very convenient.
The space needed for the backup is determined by the shape of your directory structure. On the larger setup I have lots of Maildirs and a very deep directory structure so it takes much more space than my home-directory backup above. 3GB is quite a lot, but 20MB doesn’t hurt.