This script is used to control the backup procedure for backing up my SVN repositories. A post detailing the software needed for this script to run can be seen here. This script is originally made by Damon Timm and can be found here. I will not detail how to use it, since Damon has done a good job at it, so take a look at his site.

I have made some small modifications to the script. Most important is the possibility to use European buckets with S3.

Edit 23. June 2009: The number of log files in the log file directory started to annoy me, since every file is put in the same directory and the script is run once a day. I have now changed the script to save the log files in subfolders based on year and month.

The Script

More...

This is how I solved my first backup task. An overview of my backup project is found here.

The task is to make incremental backup’s of my subversion repository and store the backup on Amazon S3. I found a blog post by Damon Timm describing how to do incremental backups to S3 using a piece of software called Duplicity. Damon’s is based on OS X or Ubuntu, so it is quite similar to my install, since my server runs Debian. There are a few differences however, such as no sudo command and i had to generate the gpg key on another machine.

Installation of the software

First thing first. Log in as root and issue an apt-get update:

su root
apt-get update

Time to install Duplicity. The latest version at time I installed the software was 0.5.16. Check Duplicity’s download page to see if there has been a new version made available:

apt-get build-dep duplicity
aptitude install python-boto ncftp
wget http://savannah.nongnu.org/download/duplicity/duplicity-0.5.16.tar.gz
tar xvzf duplicity-0.5.16.tar.gz
cd duplicity-0.5.16/
python setup.py install

We also need a piece of software called s3cmd, so that is installed next

apt-get install s3cmd

Encryption

As mentioned earlier i couldn’t get the server to generate a new key for me. It just kept hanging with a message that it needed more entropy. To overcome this problem i generated the key on a different machine, exported it, imported it to the server and changed the trust, so I could use it.

Generate and export the key

I booted Ubuntu 9.04 from a Live CD and once logged in I generated the key with the default choices:

gpg --gen-key

To export the key run the following command, substituting NAME with the name from the key and FILE with the file to save the key in.

gpg -a --export-secret-keys NAME > FILE

Import the key

After copying the key to the server, it is loaded with the following command,  substituting  FILE as described above.

gpg --allow-secret-key-import --import FILE

Next is to change the trust level on the key, otherwise Duplicity will complain and quit, rendering encrypted use of Duplicity useless. Substitute NAME with the name from the key.

gpg --edit-key NAME

This will put you in a console. Type “trust” and choose 5, I trust ultimately. After the trust have been set, you leave the console again by typing “quit”.

Passphrase

The last thing to consider is the passphrase used for the key. I have my key stored in a file that has been chmod’ed to 0600.

nano ~/.gnupg/.gpg-passphrase

Enter the passphrase and press Ctrl^O to save and then Ctrl^X to exit. Finally the rights on the file has to be modified.

chmod 0600 ~/.gnupg/.gpg-passphrase

The Script

Now that all the software bits have been set up, it is time to make a script that can take care of the backup procedure. And that script is described here .

With the increasing amount of data that i have stored in various places, it has become quite clear that i need to have backup of important things. And it should not just be backup on other disks at my home, it should be an online backup somewhere in the cloud.

After doing some research into what options are available i have settled on using Amazon’s S3 for storing the backups. There are no limits on the amount of data to store and you only pay for what you use. And it is not too expensive either.

Backup Requirements

Before I define what i would like to have backup of, i will start by making some requirements that the backup procedure should adhere to.

• Automatic. The backup procedure must be completely automated. If it aren't automatic, i will probably forget a step or two along the way and finally end up not doing any backup at all.
• Controlled by my server. The backup procedures must be controlled by my server, since it is the only machine i have that is always turned on.
• Online backup. The backup’s must be stored online, since a backup on other disks in house is void if the house burns down.
• Free! The backup procedures has to be based on free and/or open source software. It is enough to be paying for the use of S3.

Backup Tasks

So far i have identified 6 different backup tasks that i would like to address. The tasks are as follows:

1. SVN repository backup. This should be straightforward. Do incremental backup of the folder containing all my repositories. Should be run once a day.
2. Picture backup. This is the big one. Currently i have more than 20 GB worth of pictures. I won’t be doing incremental backup with the pictures, since i don’t think the content of a picture will change. Even though there is a lot of data, i will prefer to have this one be a mirror of my pictures folder. The backup procedure should be run once a week.
3. Linux web domain file backup. Incremental backup of various files on my Linux domain. Primarily my phpbb installations. Should be per app backup, for example separate backup of each of my phpbb installations. This procedure should be run once a week.
4. Linux web domain DB backup. Incremental backup of the MySQL database belonging to my Linux domain. Per app backup just as for the files. Should be run once a week.
5. Windows web domain file backup. Incremental backup of various files on my Windows domain. Backups should be per sub domain and/or app. This procedure should be run once a week.
6. Windows web domain DB backup. Incremental backup of the SQL Server databases belonging to the windows domain. Backups should be per database and/or application. Should be run once a week.

As like many other Computer Science persons, i have my own server running at home. I use it primarily as an svn server and to do backup of various things. Being a student, one of the demands I had for server was that it was cheap to run, such that my electrical bill wouldn’t be too insane. After some researching into the matter i discovered the perfect little device: the Linksys NSLU2

The Slug

The NSLU2 is nicknamed Slug. It contains a 266 MHz ARM processor, 32 MB RAM and 8 MB Flash ROM. Being small the only connectors on it are 2 USB ports, an ethernet port and the power. A nice feature of the device is that it is passively cooled, so it is dead quiet . The OS on the slug is a custom made version of Linux.

Modding

Being a Linuc device, there are some nifty people that have managed to replace the firmware on the slug with their own version, making it capable to do a lot more than it was designed to. A good source for information regarding what you can do with a slug is http://www.nslu2-linux.org/.

By changing the firmware it is possible to install the OS on an external disk, making it possible to install a lot more programs on it than can be kept in the 8 MB onboard flash. My first attempt was with Unslung, which is a good starting point, since it only extends the original firmware and still keeps the interface from the original. After a while though, i decided to change it to Debian to get the power of a full linux distro.

My Setup

My Slug runs Debian 5.0, which there is a nice installation guide for here, from a 30 GB external 2.5” harddrive. I have set up ssh with key authentication on it, which is my primary means of interacting with it. I also have Sambato make a little network drive. I have subversion installed for having my own versioning control system. I also have some backup solutions running that stores the backup on Amazon S3, i will describe those setups in other blog posts.