how to set it up

The only way I have found to circumvent this annoyance is to play a little trick on iMovie. And that trick is to move the original ~/Movies folder under a different name and put a symlink in its place (to the network folder of your choice).

Here are the steps I took from the Terminal to do this (you will need to have Administrator privileges and you will need to know where your network folder is attached):

• sudo mv ~/Movies/ ~/Movies-original
• ln -s /Volumes/my-network-volume/Movies/ ~/Movies

Now, when I load up iMovie it will use the networked volume. If you don’t like this, you can always move your ~/Movies-original folder back where it was.

remaining issues

Unfortunately, this isn’t a perfect fix. There are two caveats or issues I have found.

One, is that now my hard drive (called Papa Bear) is listed twice in the Project Library:

And two, is that when you delete an event or project from within iMovie, it moves the project into a temporary folder rather than actually deleting it. The next time you open iMovie, it will show up again. I’m sure this is because it thinks it is working with a bona fide Trash system but its not (because it is network storage. See this example:

In order to get around this annoyance, I created a Python script which removes the temporary files for me and created a cron job to run the script from time to time (cleaning up the mess iMovie leaves behind).

#!/usr/bin/env python
'''
Simple script to remove all directories that match the name:
    /iMovie Temporary Items */  [note the star!]

'''
import os
import fnmatch
import shutil

# tuple of directories to search through (you can add more than one)
DIRS = ('/Volumes/leaker-damon/tps/Videos/',)

for directory in DIRS:
    for root, dirnames, filenames in os.walk(directory):
        for directory in dirnames:
            if fnmatch.fnmatch(directory,'iMovie Temporary Items *'):
                shutil.rmtree(os.path.join(root,directory))

make this more better

Am interested to hear how other people have been handling this — this approach certainly isn’t perfect, but it seems to work (for now).

install nc and pv

If you are on a mac, you can use MacPorts to install this easily enough:

• $ sudo port install netcat pv

If you are on Ubuntu, you just need to get pv installed (nc is already there):

• $ sudo aptitude install pv

Once you have nc and pv installed, it’s really simple.  On one machine, run the following command:

• $ nc -ulp 5000 > /dev/null

On the second machine run the following command (you need the IP address of the first machine):

• $ pv < /dev/zero | nc -u ip.addy.of.other.machine 5000

And you should get some output with a little <=> sign moving across the screen that resembles this (static):

1.15GB 0:00:19 [ 218MB/s] [             <=>                  ]

This will show you the number of MB/s the connection is averaging; if you watch if for a bit, you can get an idea of where things stand. Here are some of the results I got between my PowerMac, PowerBook, and EEE PC:

localhost:                      218MB/s
gigabit Ethernet:               143MB/s
100MB ethernet:                 11.5MB/s
wireless (poor connection):     141kB/s

If you read the original post on slashdot, it seems you can us pv to measure a lot of other stuff as well — I haven’t played with it yet but should, one day.

before you begin

Before you can start backing things up off-site in a secure fashion, you’ll need to get a few pieces of the puzzle in place. Namely, you’ll need software installed (duplicity), a GPG key (for encryption), and an Amazon S3 account setup (for storage), and then use a backup script that can be run automatically (for laziness’s sake!).

Getting an Amazon S3 account is easy to do: head over to http://aws.amazon.com/s3/ and sign up; grab your “Access Key ID” and “Secret Access Key” and you are ready to go. There is a lot you can do with S3 (and a lot of ways to access it), but for our purposes, this is pretty much all you need.

Lastly, I would recommend spending a little time reading about duplicity (see the man page) as well as GnuPG (man page). There is a lot to consider, and I just picked the options I thought would work best for me.

Install the Software: Duplicity

For this to work we need duplicity installed with all the correct dependencies. The easiest way to do this on your Mac is to simply use MacPorts, which has an up-to-date version in the repositories (see Installing MacPorts if you don’t have it installed already). If you already have MacPorts installed, all you should have to do is run the following from the Terminal:

• $ sudo port install duplicity py25-socket-ssl py25-boto

If you are using Ubuntu, you could simply run sudo aptitude install duplicity to install the program (it is in the repositories); however, if you want to make sure you are using the latest version (which may not be available there yet), you can try this:

• $ sudo apt-get build-dep duplicity
• $ sudo aptitude install python-boto ncftp
• $ wget http://savannah.nongnu.org/download/duplicity/duplicity-0.5.07.tar.gz
• $ tar xvzf duplicity-0.5.07.tar.gz
• $ cd duplicity-0.5.07/
• $ sudo python setup.py install

If you ever want to upgrade again, just download and untar the latest version and run the last setup line again.  It will install the newest version for you.

If everything has installed correctly, you can do a test run pretty easily on your local machine by backing up a folder to another local folder (first command) and then restoring it to a different folder (second command). If you look inside this /test/backup-location/ you’ll see what duplicity looks like:

• $ duplicity --no-encryption /test/folder/ file:///test/backup-location/
• $ duplicity --no-encryption file:///test/backup-location/ /test/restore-location/

Setting Up Encryption

For duplicity to really shine, it needs to have a gpg key to encrypt your files. If you don’t already have one, you can create it by running the following (read the documentation for more information):

• $ gpg --gen-key

I used all the defaults when setting up my key and chose my own passphrase. Unfortunately, in order to make this work without user input (as an automatic cron job), the passphrase is going to have to be stored somewhere on your computer locally, so, I wouldn’t use one of your usual passwords (something really long and unique would be better).  Also, if you already have a gpg key (or want to use one for other purposes), I would recommend making a different one for the Amazon S3 backups — because, in the end, your password has to be stored somewhere on your computer for it to work auto-magically.

Once you have your gpg key created you can check it out by running:

• $ gpg --list-keys

This shows your new key, which probably looks something like this:

pub   1024D/CA4ZA320 2008-11-15
uid                  Damon Timm (thornomad) email@example.com
sub   2048g/C1E64A4F 2008-11-15

Note the public key identifier “CA4FA320″ (yours will be different); we will need that to go in our script.

final step: using a backup script

So, everything is on your system and, hopefully, working.  Now, to run a backup takes a lot of typing (on the command line) and the easiest way to avoid this chore is to run a backup script.  A script can store your Amazon and GPG key information and make it so you don’t have to type anything ever again!

Backing things up is a very personal task, and everyone is going to want to do it a little differently. I created my own backup script which you are happy to check out — if you have any neat features you add to suggestions, I would love to hear them and incorporate them.

Good luck!

Most importantly, you can easily backup the script and your gpg key in a convenient passphrase-encrypted file. This comes in in handy if/when your machine ever does go belly up.

how to use

To get the latest latest code in the script you can download a zip copy of the source or clone the git repository like so:

• git clone git://github.com/thornomad/dt-s3-backup.git

You’ll also need to have a number of things in place in order to utilize this script, specifically: gpg, duplicity, an Amazon S3 account, and (optionally) s3cmd. If you need help getting all these in order, I wrote another post about putting it all together. It’s not all that difficult, but does take a few pieces of the puzzle to be in order.

Once you have the script, you will need to fill out the foobar variables with your own specific information.  I suggest testing the script on a small directory of files and a local directory for your destination first to make sure it is working.

Usage

From the README file:

COMMON USAGE EXAMPLES
=====================

* View help:
    $ dt-s3-backup.sh

* Run an incremental backup:
	$ dt-s3-backup.sh --backup

* Force a one-off full backup:
    $ dt-s3-backup.sh --full

* Restore your entire backup:
	$ dt-s3-backup.sh --restore
    You will be prompted for a restore directory

	$ dt-s3-backup.sh --restore /home/user/restore-folder
    You can also provide a restore folder on the command line.

* Restore a specific file in the backup:
    $ dt-s3-backup.sh --restore-file
    You will be prompted for a file to restore to the current directory

    $ dt-s3-backup.sh --restore-file img/mom.jpg
    Restores the file img/mom.jpg to the current directory

    $ dt-s3-backup.sh --restore-file img/mom.jpg /home/user/i-love-mom.jpg
    Restores the file img/mom.jpg to /home/user/i-love-mom.jpg

* List files in the remote archive
	$ dt-s3-backup.sh --list-current-files

* Verify the backup
    $ dt-s3-backup.sh --verify

* Backup the script and gpg key (for safekeeping)
    $ dt-s3-backup.sh --backup-script

Changes

You can view the changelog at github.

Install MacPorts

First, we’ll need to make sure you have “XCode Tools” installed on your Mac — XCode Tools install all the programs needed to build applications from source. It comes standard with all Macs, however, it isn’t installed by default, so you’ll need your installation DVD (or you can download them). From the DVD, go to the “Optional Install” folder and then “XCode Tools” and run the installer package.

Second, we’ll have to download and install the MacPorts from: http://macports.org/install.php. Choose a .dmg disk image for whichever version of OS X you are using.

Third, after MacPorts has been installed, we’ll need to update our PATH variable in the Terminal to be able to find the MacPorts program (which is called port). If you have your own way of doing this, go for it — however, I created a ~/.bash_profile file to accomplish this task. If you don’t already have a profile file, you can do this by entering the following in the terminal (I also added two other common places programs get installed on the command line that save me trouble down the road):

• $ echo "export PATH=/usr/local/bin/:/usr/local/sbin/:/opt/local/bin/:/opt/local/sbin/:$PATH" >> ~/.bash_profile

Restart the Terminal program for these changes to take effect. If this has been accomplished successfully, then you should be able to run (again in the Terminal) the following for a self-update and it should work:

• $ sudo port -v selfupdate

If nothing happens and, instead, you get an error that says: -bash: port: command not found, that means your PATH variable wasn’t set correctly. If this is the first time you are running a command using sudo you’ll get a warning saying something to the effect of: if you don’t know what you are doing, turn back now. Ignore this warning.

What’s Next

With MacPorts installed by itself, nothing much is going to change. But you’ll be able to use it to install open source applications that are very exciting (like lame, flac, ffmpeg, gimp, etc).

Tested on: Ubuntu 6.06.1, 6.10, 10.04.

install libtrash and test it out

The version of libtrash in the repository is not the latest — I prefer to grab it from source which, in this case, is very easy:

• sudo aptitude install build-essential
• wget http://pages.stern.nyu.edu/~marriaga/software/libtrash/libtrash-latest.tgz
• tar xzf libtrash-latest.tgz
• cd libtrash-3.2/
• make
• sudo make install

Unlike a lot of software on linux, installing it isn’t quite enough to get it running. I think this is because it is a lib package and not an actual program — it seems to be meant to be used by other programs in the background and not directly interact with the user (in the way that I want it to).

To test it out briefly, run the following to start the libtrash engine in your terminal prompt:

• export LD_PRELOAD=/usr/local/lib/libtrash.so

After the above command has run, you can do a little test by creating a file (the touch command doesn’t seem to work, so create a real file) and then delete it. It should show up in your ~/Trash folder.

If it isn’t working at this point, check out the documentation at: /usr/share/doc/libtrash/

To make sure libtrash is running every time you login you need to add the export command to the top of /etc/profile:

# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).

export LD_PRELOAD=/usr/local/lib/libtrash.so

[...]

#!/bin/bash
export LD_PRELOAD=/usr/local/lib/libtrash.so
/usr/lib/openssh/sftp-server

#Subsystem sftp /usr/lib/openssh/sftp-server
Subsystem sftp /usr/local/lib/libtrash-sftp-server

case "$1" in
	start)
        export LD_PRELOAD=/usr/local/lib/libtrash.so
		if [ "x$ATALK_BGROUND" = "xyes" ]; then
			echo "Starting Netatalk services in the background."
			atalk_startup >/dev/null &
		else
			echo -n "Starting Netatalk services (this will take a while): "
			atalk_startup
			echo "."
		fi
	;;

# m h  dom mon dow   command
30 01 * * * /usr/sbin/strash --age 7d