install nc and pv

If you are on a mac, you can use MacPorts to install this easily enough:

• $ sudo port install netcat pv

If you are on Ubuntu, you just need to get pv installed (nc is already there):

• $ sudo aptitude install pv

Once you have nc and pv installed, it’s really simple.  On one machine, run the following command:

• $ nc -ulp 5000 > /dev/null

On the second machine run the following command (you need the IP address of the first machine):

• $ pv < /dev/zero | nc -u ip.addy.of.other.machine 5000

And you should get some output with a little <=> sign moving across the screen that resembles this (static):

1.15GB 0:00:19 [ 218MB/s] [             <=>                  ]

This will show you the number of MB/s the connection is averaging; if you watch if for a bit, you can get an idea of where things stand. Here are some of the results I got between my PowerMac, PowerBook, and EEE PC:

localhost:                      218MB/s
gigabit Ethernet:               143MB/s
100MB ethernet:                 11.5MB/s
wireless (poor connection):     141kB/s

If you read the original post on slashdot, it seems you can us pv to measure a lot of other stuff as well — I haven’t played with it yet but should, one day.

before you begin

Before you can start backing things up off-site in a secure fashion, you’ll need to get a few pieces of the puzzle in place. Namely, you’ll need software installed (duplicity), a GPG key (for encryption), and an Amazon S3 account setup (for storage), and then use a backup script that can be run automatically (for laziness’s sake!).

Getting an Amazon S3 account is easy to do: head over to http://aws.amazon.com/s3/ and sign up; grab your “Access Key ID” and “Secret Access Key” and you are ready to go. There is a lot you can do with S3 (and a lot of ways to access it), but for our purposes, this is pretty much all you need.

Lastly, I would recommend spending a little time reading about duplicity (see the man page) as well as GnuPG (man page). There is a lot to consider, and I just picked the options I thought would work best for me.

Install the Software: Duplicity

For this to work we need duplicity installed with all the correct dependencies. The easiest way to do this on your Mac is to simply use MacPorts, which has an up-to-date version in the repositories (see Installing MacPorts if you don’t have it installed already). If you already have MacPorts installed, all you should have to do is run the following from the Terminal:

• $ sudo port install duplicity py25-socket-ssl py25-boto

If you are using Ubuntu, you could simply run sudo aptitude install duplicity to install the program (it is in the repositories); however, if you want to make sure you are using the latest version (which may not be available there yet), you can try this:

• $ sudo apt-get build-dep duplicity
• $ sudo aptitude install python-boto ncftp
• $ wget http://savannah.nongnu.org/download/duplicity/duplicity-0.5.07.tar.gz
• $ tar xvzf duplicity-0.5.07.tar.gz
• $ cd duplicity-0.5.07/
• $ sudo python setup.py install

If you ever want to upgrade again, just download and untar the latest version and run the last setup line again.  It will install the newest version for you.

If everything has installed correctly, you can do a test run pretty easily on your local machine by backing up a folder to another local folder (first command) and then restoring it to a different folder (second command). If you look inside this /test/backup-location/ you’ll see what duplicity looks like:

• $ duplicity --no-encryption /test/folder/ file:///test/backup-location/
• $ duplicity --no-encryption file:///test/backup-location/ /test/restore-location/

Setting Up Encryption

For duplicity to really shine, it needs to have a gpg key to encrypt your files. If you don’t already have one, you can create it by running the following (read the documentation for more information):

• $ gpg --gen-key

I used all the defaults when setting up my key and chose my own passphrase. Unfortunately, in order to make this work without user input (as an automatic cron job), the passphrase is going to have to be stored somewhere on your computer locally, so, I wouldn’t use one of your usual passwords (something really long and unique would be better).  Also, if you already have a gpg key (or want to use one for other purposes), I would recommend making a different one for the Amazon S3 backups — because, in the end, your password has to be stored somewhere on your computer for it to work auto-magically.

Once you have your gpg key created you can check it out by running:

• $ gpg --list-keys

This shows your new key, which probably looks something like this:

pub   1024D/CA4ZA320 2008-11-15
uid                  Damon Timm (thornomad) email@example.com
sub   2048g/C1E64A4F 2008-11-15

Note the public key identifier “CA4FA320″ (yours will be different); we will need that to go in our script.

final step: using a backup script

So, everything is on your system and, hopefully, working.  Now, to run a backup takes a lot of typing (on the command line) and the easiest way to avoid this chore is to run a backup script.  A script can store your Amazon and GPG key information and make it so you don’t have to type anything ever again!

Backing things up is a very personal task, and everyone is going to want to do it a little differently. I created my own backup script which you are happy to check out — if you have any neat features you add to suggestions, I would love to hear them and incorporate them.

Good luck!

Most importantly, you can easily backup the script and your gpg key in a convenient passphrase-encrypted file. This comes in in handy if/when your machine ever does go belly up.

how to use

To get the latest latest code in the script you can download a zip copy of the source or clone the git repository like so:

• git clone git://github.com/thornomad/dt-s3-backup.git

You’ll also need to have a number of things in place in order to utilize this script, specifically: gpg, duplicity, an Amazon S3 account, and (optionally) s3cmd. If you need help getting all these in order, I wrote another post about putting it all together. It’s not all that difficult, but does take a few pieces of the puzzle to be in order.

Once you have the script, you will need to fill out the foobar variables with your own specific information.  I suggest testing the script on a small directory of files and a local directory for your destination first to make sure it is working.

Usage

From the README file:

COMMON USAGE EXAMPLES
=====================

* View help:
    $ dt-s3-backup.sh

* Run an incremental backup:
	$ dt-s3-backup.sh --backup

* Force a one-off full backup:
    $ dt-s3-backup.sh --full

* Restore your entire backup:
	$ dt-s3-backup.sh --restore
    You will be prompted for a restore directory

	$ dt-s3-backup.sh --restore /home/user/restore-folder
    You can also provide a restore folder on the command line.

* Restore a specific file in the backup:
    $ dt-s3-backup.sh --restore-file
    You will be prompted for a file to restore to the current directory

    $ dt-s3-backup.sh --restore-file img/mom.jpg
    Restores the file img/mom.jpg to the current directory

    $ dt-s3-backup.sh --restore-file img/mom.jpg /home/user/i-love-mom.jpg
    Restores the file img/mom.jpg to /home/user/i-love-mom.jpg

* List files in the remote archive
	$ dt-s3-backup.sh --list-current-files

* Verify the backup
    $ dt-s3-backup.sh --verify

* Backup the script and gpg key (for safekeeping)
    $ dt-s3-backup.sh --backup-script

Changes

You can view the changelog at github.

This is because, apparently, OpenSSL has a license that is incompatible with Debian’s GPL. Regardless: clear text is bad; encryption is good. And since Ubuntu doesn’t package netatalk with the appropriate encryption support, one must do it oneself.

Updated 05.08.09: Just tested this with Jaunty (09.04) and the package in the repositories works with no extra steps. If you are using an older version of Ubuntu, however, you will want to follow these instructions. Tested with Intrepid Ibex (8.10) as well as: 6.06, 7.04, 7.10, and 8.06.

about this guide

When I first found that Ubuntu’s netatalk package didn’t support encrypted authentication, I tried to compile netatalk from the source. I didn’t get very far. Throwing up my hands in frustration, I spent some more time on google and found some ideas at the Ubuntu Forums. Pulling it all together, with ideas and fixes from comments (below), this is what I came up with (which I think is a lot easier than building from source).

steps to follow

NOTE: If you have already installed netatalk you should remove it before proceeding with a sudo aptitude purge netatalk before you get going.

• sudo aptitude update
• mkdir -p ~/src/netatalk
• cd ~/src/netatalk
• sudo aptitude install cracklib2-dev libssl-dev
• apt-get source netatalk
• sudo apt-get build-dep netatalk
• cd netatalk-2.0.3
• sudo DEB_BUILD_OPTIONS=ssl dpkg-buildpackage -us -uc
• sudo debi
• echo "netatalk hold" | sudo dpkg --set-selections

The basic trend of this set of operations is to: create a directory where all the messy files can be stored, download necessary packages, get the netatalk source, compile the source with the ssl option, install the package, then tell Ubuntu never to update the package (because if it did, it would break).

Settings for the netatalk service can be found on your Ubuntu machine at /etc/netatalk/. There are a couple configuration files in there with instructions. Good luck.

configuration files

One of the first changes I make after installing netatalk is to disable some of the services that I don’t need running (and start those that I do). To do this, I edit: /etc/default/netatalk, changing the daemons that run so that it ends up looking like this (which allows netatlk to restart a lot quicker without the atalkd daemon — which is a holdover from pre-OS X times):

# Set which daemons to run (papd is dependent upon atalkd):
ATALKD_RUN=no
PAPD_RUN=no
CNID_METAD_RUN=no
AFPD_RUN=yes
TIMELORD_RUN=no
A2BOOT_RUN=no

These are the settings I am using since I only need the afp file server — one thing to note, however, is that if you want to use the dbd databashe scheme rather than cdb, you need to set CNID_METAD_RUN to yes.  cdb is supposed to be faster, while dbd is supposed to be “corruption-proof”.  You can read it about in the docs.

After you’ve saved changes to this configuration file, run the following to restart netatalk:

• sudo /etc/init.d/netatalk restart

other tips and tricks

Here are a couple other thoughts and pointers that I’ve picked up over the years …

multiple afp servers running on the same network

I never thought much of it, but I did notice: if you have two different servers on your network running netatalk, you are unable to login to both of them at the same time. JET posted a solution to this and it works flawlessly. It has changed my life.

multiple network interfaces causing errors

Update (9/24/07 & 10/22/07): I’ve noticed a few people mentioning they get an error when compiling and/or starting netatalk (from ubuntuforums.org as well). Folks with more than one available network adapter (like eth1 and eth2, or virtual adapters created by vmware) seem to run an error when they compile and during runtime . During compile time you might have have an error that ends in:

• dpkg: error processing netatalk (--install):
• subprocess post-installation script returned error exit status 1
• Errors were encountered while processing:
• netatalk
• debi: debpkg -i failed

After this, you would probably get an error at runtime that looked like:

• Starting Netatalk services (this will take a while): nbp_rgstr: Connection timed out

Tim Pope wrote a suggested fix in the comments below that should eliminate the conflict between the multiple adapters. I only have one adapter myself (and don’t use vmware) so I haven’t had a chance to try it yet myself. Let me know if this works for you as well.

Special Thanks: goes to user llamakc from ubuntuforums.org for helping me with this one night in this thread; also, can find Ubuntu’s SSHFS documentation here.)

install the sshfs software and mount

After some trial and error on my part, I found that only a few simple steps are needed to get everything up and running:

First, get the sshfs software (which is based on FUSE); if you have Ubuntu, this is easy because it is an included package available for easy install. After the package is installed, you need to add your username to the fuse group. On Ubuntu, you would open a terminal window and perform the following:

• sudo aptitude update
• sudo apititude install sshfs
• sudo adduser yourusername fuse

After, restart your machine. (I have tried just logging in and logging out, but I kept getting permissions errors — all of which disappeared after a restart.)

The next step it so to create an empty directory that will serve as the “window” into the SFTP server. I created a folder on my desktop called sftp. Once the folder has been created, simply run sshfs using the appropriate login information (host username and IP), the host and local directories, and the SFTP connection is mounted on a folder on my desktop.

• mkdir ~/Desktop/sftp
• sshfs HOSTuser@remote.host.or.ip:/host/dir/to/mount ~/Desktop/sftp

This folder will work like any other folder on your system; when you restart your computer (or logout and log back in) you will have to go through the last step of the process again (calling the sshfs program) to enable the folder on your desktop (save time by creating a bash alias).

possible errors and workarounds

When I restarted my system the first time (using Ubuntu 6.06), after having so cleverly got sshfs to work, and tried to run my sshfs command I got an error:

• sshfs HOSTuser@remote.host.or.ip:/host/dir/to/mount ~/Desktop/sftp
fusermount: failed to open /dev/fuse: No such file or directory


A quick search on google brought me to the sourceforge FAQ for sshfs and there, lo and behold, the following was suggested to rectify the situation:

• sudo mknod -m 666 /dev/fuse c 10 229


After running this command, I was able to mount my SFTP directory. I never received this error using Ubuntu 6.10.

If you get any permission denied warnings, be sure you have added your username to the fuse group and also restarted your system.

create a bash alias to save time and typing

To save time, I created a bash alias that would remember all the details for me.

First, make make sure my system reads from the ~/.bash_aliases file (it may not be default). Open ~/.bashrc and ensure the following lines are uncommented:

• # Alias definitions.
• # You may want to put all your additions into a separate file like
• # ~/.bash_aliases, instead of adding them here directly.
• # See /usr/share/doc/bash-doc/examples in the bash-doc package.
• if [ -f ~/.bash_aliases ]; then
• . ~/.bash_aliases
• fi

Next, create (or modify if you already have one) your ~/.bash_aliases file.

• nano ~/.bash_aliases

I added the following single line of code to the document (first call the mknod, if you are getting the error, then the sshfs):

• alias sftp='sudo mknod -m 666 /dev/fuse c 10 229; sshfs HOSTuser@remote.host.or.ip:/host/dir/to/mount ~/Desktop/sftp'


Now, when I open the terminal, I just type:

• sftp

And everything loads correctly. Is very fast and very nice. I like it.

finally:

Changes to your ~/.bash_aliases file will only take effect after you have reopened the terminal or called:

• . ~/.bash_aliases

If you ever want to unmount the directory without logging out or restarting, use the following:

• fusermount -u ~/Desktop/sftp

Benefit: I can login remotely with my user name (henry) and have direct access to the web documents root folder. This makes it easy to use FTP or SSH or AFP to move files around and I don’t have to fool with user groups or permissions.

If you want to do this, login as the user you want to host the web pages and run the following from the terminal:

• $ mkdir ~/www
• $ sudo mv /var/www /var/www-backup
• $ sudo ln -s /home/henry/www /var/www

This will create a www folder in your user’s home directory, change the name of Apache’s default www directory, and create a symbolic link in its place that points to the directory you recently created.

Drop a index.html file in your new ~/www directory and you are good to go!