As a self-taught Python user I am still looking for insight on the most pythonic and programmatically-friendly way of accomplishing a given task. In this case, I have written a script that will perform a “clean bzip2″ of a directory (or directories). Mac OS X (via AFP and netatalk, in my case) tends leaves a bunch of ugly files/directories hanging around and I would rather not include them in my compressed tar file.

In writing the script, though, I ran into some questions and I am not sure what the recommended approach would be. The script works, as it is, but I feel its a little hacked together and also a little limited in its application. There is something to be said for programs that “just work” (this does) but I want to take it a little further as an educational endeavor and would like it to appear robust, future-thinking, and pythonic.

My initial questions are:

1. Is there a better way to implement a --quiet flag? [using logging]
2. I am not very clear on the use of Exceptions (or even if I am using it in a good way here) — is what I have done the right approach?
3. Finally, in general: any feedback on how to improve this? (I am thinking, just now, that the script is only suitable for a command line usage, and couldn’t be imported by another script, for example.)

Any feedback is greatly appreciated. Writing a script like this is a good learning tool (for me, at least).

#! /usr/bin/env python

'''Script to perform a "clean" bzip2 on a directory (or directories).  Removes
extraneous files that are created by Apple/AFP/netatalk before compressing.
'''

import os
import tarfile
import logging
from optparse import OptionParser

# Default files and directories to exclude from the bzip tar
IGNORE_DIRS = ('.AppleDouble',)
IGNORE_FILES = ('.DS_Store',)

class DestinationTarFileExists(Exception):
    '''If the destination tar.bz2 file already exists.'''

def ignore_walk(directory, ignore_dirs=None, ignore_files=None):
    '''Ignore defined files and directories when doing the walk.'''

    # TODO: this does not currently take wild cards into account.  For example,
    # if you wanted to exclude *.pyc files ... should fix that.  Perhaps
    # consider moving this entirely into the below function (or making it more
    # reusable for other apps).
    for dirpath, dirnames, filenames in os.walk(directory):
        if ignore_dirs:
            dirnames[:] = [dn for dn in dirnames if dn not in ignore_dirs]
        if ignore_files:
            filenames[:] = [fn for fn in filenames if fn not in ignore_files]
        yield dirpath, dirnames, filenames

def tar_bzip2_directory(directory, ignore_dirs=IGNORE_DIRS,
                                   ignore_files=IGNORE_FILES ):
    '''Takes a directory and creates a tar.bz2 file (based on the directory
    name).  You can exclude files and sub-directories as desired.'''

    file_name = '-'.join(directory.split(' '))
    tar_name = file_name.replace('/','').lower() + ".tar.bz2"

    if os.path.exists(tar_name):
        msg = ("The file %s already exists. " +
                "Please move or rename it and try again.") % tar_name
        raise DestinationTarFileExists(msg)

    tar = tarfile.open(tar_name, 'w:bz2')

    for dirpath, dirnames, filenames in ignore_walk(directory, ignore_dirs,
            ignore_files):
        for file in filenames:
            logging.info(os.path.join(dirpath, file))
            tar.add(os.path.join(dirpath, file))

    tar.close()

def main(args=None, callback=None):
    directories = []

    for arg in args:
        if os.path.isdir(arg):
            directories.append(arg)
        else:
            logging.ERROR("Ingoring: %s (it's not a directory)." % arg)

    for dir in directories:
        try:
            tar_bzip2_directory(dir)
        except DestinationTarFileExists, e:
            print e

if __name__ == "__main__":

    parser = OptionParser(usage="%prog [options: -q ] [dir1] [...dir2]")
    parser.add_option("-q", "--quiet", action="store_true", dest="quiet")
    options, args = parser.parse_args()

    if options.quiet:
        logging.basicConfig(level=logging.ERROR, format='%(message)s')
    else:
        logging.basicConfig(level=logging.INFO, format='%(message)s')

    main(args)

Most importantly, you can easily backup the script and your gpg key in a convenient passphrase-encrypted file. This comes in in handy if/when your machine ever does go belly up.

how to use

To get the latest latest code in the script you can download a zip copy of the source or clone the git repository like so:

• git clone git://github.com/thornomad/dt-s3-backup.git

You’ll also need to have a number of things in place in order to utilize this script, specifically: gpg, duplicity, an Amazon S3 account, and (optionally) s3cmd. If you need help getting all these in order, I wrote another post about putting it all together. It’s not all that difficult, but does take a few pieces of the puzzle to be in order.

Once you have the script, you will need to fill out the foobar variables with your own specific information.  I suggest testing the script on a small directory of files and a local directory for your destination first to make sure it is working.

Usage

From the README file:

COMMON USAGE EXAMPLES
=====================

* View help:
    $ dt-s3-backup.sh

* Run an incremental backup:
	$ dt-s3-backup.sh --backup

* Force a one-off full backup:
    $ dt-s3-backup.sh --full

* Restore your entire backup:
	$ dt-s3-backup.sh --restore
    You will be prompted for a restore directory

	$ dt-s3-backup.sh --restore /home/user/restore-folder
    You can also provide a restore folder on the command line.

* Restore a specific file in the backup:
    $ dt-s3-backup.sh --restore-file
    You will be prompted for a file to restore to the current directory

    $ dt-s3-backup.sh --restore-file img/mom.jpg
    Restores the file img/mom.jpg to the current directory

    $ dt-s3-backup.sh --restore-file img/mom.jpg /home/user/i-love-mom.jpg
    Restores the file img/mom.jpg to /home/user/i-love-mom.jpg

* List files in the remote archive
	$ dt-s3-backup.sh --list-current-files

* Verify the backup
    $ dt-s3-backup.sh --verify

* Backup the script and gpg key (for safekeeping)
    $ dt-s3-backup.sh --backup-script

Changes

You can view the changelog at github.