One of the most often read postings on this blog is the one about my getopts-enabled shell script template. So I figured, a similar template in Python would also be useful. Like the shell script, it’s quite trivial, but it might still save some time.

The template is an example for a typical unix utility. It connects two aspects: An optparse-based command line parser and iterating over multiple input sources using the fileinput module.

I’ve used command line parsers in several programming languages, but from all of them, I liked Python’s optparse module best (for those who need it, the Python’s standard library still provides a classic getopt). The fileinput module is simple but useful. Beyond its main functionality, it supports more advanced features like transparent decompressing (not really Unix-like though) and in-place filtering of files, but I rarely ever use them.

Here’s the template in my online Mercurial repository and here it is for download.

Since Python 2.5, creating histograms has become easier. Instead of dict, we can now use defaultdict which is similar in behavior to awk’s associative arrays. Instead of raising a KeyError for undefined keys, defaultdict adds a user-defined item and returns it.

I’ll demonstrate this with a simple program that analyzes line length distribution in a file. In older Python versions, you’d typically write code like this:

hist = { }
for line in open(filename):
    hist[len(line)] = hist.get(len(line), 0) + 1

The code using defaultdict is much clearer and more elegant (although an additional import is needed):

from collections import defaultdict

hist = defaultdict(int)
for line in open(filename):
    hist[len(line)] += 1

Note that defaultdict’s constructor expects a factory function that initializes unset items on request.

Unless you dump the contents of hist to gnuplot or similar, you might want to sort the dict by value. There are several ways to do this, but I learned from a related blog posting that this is the most efficient way:

from operator import itemgetter
sorted(hist.iteritems(), key=itemgetter(1))

The min and max builtins support the key parameter, too, by the way.

Usually, I download all my email via fetchmail, sort it into folders using procmail and read the mails with mutt. This works reasonably well and I’ve been using this mail setup without major changes for about 10 years. Since I recently bought a netbook, I also wanted to read mails with it, but without touching my existing mail setup. The netbook should only read mails directly on the server without downloading anything to local disk.

My mail service provider 1&1 supports IMAP and a webmail client that I sometimes use, so I decided to follow the webmail client’s conventions regarding folder names. This way, I can use mutt on the netbook and the webmail client interchangeably.

Based on a mutt configuration for gmail, I ended up with the following IMAP-related settings:

set from = "your.address@example.invalid"
set realname = "Your Real Name"

set imap_user = "matthias@mafr.de"
set imap_pass = "*******"

set smtp_url = "smtp://$imap_user@smtp.1und1.de:587/"
set smtp_pass = $imap_pass

set folder = "{imap.1und1.de/ssl}"
set spoolfile = "+INBOX"
set record = "+Gesendete Objekte"
set postponed = "+Entwürfe"
set trash = "+Papierkorb"

set move = no

mailboxes "+INBOX"

set header_cache = "~/.cache/mutt/headers"

The most important account settings are imap_user, imap_pass (optional, mutt will prompt for it), and folder. Folder names prefixed with a + sign are relative to folder. Note that I disabled moving read mails to the mbox folder; I want to keep everything in my inbox. Because of the record setting, all sent mails are recorded in a folder on the IMAP server.

To speed things up, I enabled the header cache. You could also cache entire messages, but since mutt doesn’t expire them yet, I don’t use this feature.

What doesn’t work: Neither fetchmail nor the 1&1 server propagate the IMAP "seen" flag of the emails to a BSD mbox-style "Status: RO" header. Because of this, I have to mark all downloaded mails as read manually.

Once in a while, I install Python packages from source using distutils. The distutils-powered setup.py script found in many packages installs software in /usr/local/ – quite useful because it doesn’t interfere with packages managed by your distribution. Since Ubuntu Jaunty and Python 2.6, this doesn’t work anymore.

Previously, your usual command python setup.py install installed packages in /usr/local/lib/pythonX.Y/site-packages/, depending on your Python version. As of Ubuntu Jaunty, Python 2.6 no longer has this directory in sys.path. Instead, it expects local packages in /usr/local/lib/python2.6/dist-packages/, so your locally installed packages are no longer found.

I learned from this posting that distutils now supports an --install-layout=deb switch that installs the package in /usr/local/lib/python2.6/dist-packages. The switch is accepted in distutils from Python 2.4 and 2.5, too, without having any effect, so you can always include it in your command line.

To sum things up, the new command line for installing python packages in /usr/local/ is

  python setup.py install --install-layout=deb

This probably applies to Debian systems as well, judging from the name choice of --install-layout’s parameter. I updated my posting on stow to reflect this change.

Everybody knows that Unix shell utilities are powerful. Even though they’re text-based, you can build a lot of useful things outside of the text domain. Today I’ll show you how to implement set operations. All we need are sorted files as input, with each file representing a set.

Let’s create some input files A and B:

$ echo a b d e | tr ' ' '\n' > A
$ echo a c e f | tr ' ' '\n' > B

Note: It’s an important precondition is that all input files are sorted! To avoid ugly surprises, the files mustn’t contain duplicates, but you would expect that from a set.

Since the input files are sorted, equality is trivial:

$ cmp A B

For use in scripts, the silent switch -s is useful.

Intersection:

$ comm -12 A B

The comm utility outputs three columns per default: The lines unique to the first file, the lines unique to the second file and the lines appearing in both files. We turn off the first two columns leaving just the intersection.

Union:

$ sort -m A B | uniq

The merge switch -m isn’t necessary for correctness, but it improves runtime for large files quite a bit. Unfortunately, not all sort implementations have it.

Difference:

$ comm -23 A B

This only displays the lines unique to A, leaving the result of A minus B.

You can use the difference to get the complement. This requires a universe U first that we create using our union construct. The we subtract A from U:

$ sort -m A B | uniq | comm -23 - A

With only two files, this is just an inefficient way of subtracting A from B.

Contains can be implemented using grep, but we have to be careful about false positives:

$ grep -Fx e A

The quiet switch -q is useful in scripts and may also reduce runtime because it exits as soon as the entry is found. This is still far less efficient than it could be if we had fixed size records and used interval halving.

More advanced operations like testing if one set is a subset of another are possible, too. Often you will also have to compare line counts using wc -l, but I’ll leave that as an exercise to the reader :)

Although one could have guessed otherwise, I’ve always been a happy Subversion user. Which former CVS user hasn’t? But now I’m ready to take the next step. For some of my personal stuff I’ve been using Mercurial for more than a year, so I thought it was time to take the plunge and switch my JSysTest project to Mercurial.

Google Code recently added Mercurial hosting, so I didn’t have to switch the hosting provider. The conversion process is quite simple: There are good instructions available that worked without any problems. From what I’ve seen so far, all functionality I care about is available (basically everything I used Subversion for).

One thing surprised me though: Converting a two dozen revisions took roughly 20 minutes, so for larger repositories you will have to take a long coffee break.

Recently, I had the opportunity to work with Subversion 1.5 on a medium sized project. Since there were more than 20 developers working on the project with some of them in a different country, there was no other way than to use feature branches extensively. We thought Subversion’s merge tracking would reduce typical merging errors by simplifying the process. But, well, we were wrong.

The sad truth is that few developers know exactly what their revision control system does behind their backs when merging branches. With Subversion prior to 1.5, things were pretty simple and there’s little magic involved. You have to be careful to state the correct revisions and if you don’t, you can still roll back your changes.

With 1.5, as long as everyone plays by the rules and sticks to the basic use cases, everything is fine. But if things go wrong (and they will), you’re really screwed. In a large team, there will always be someone who uses a buggy Subversion client, or someone who accidentally deletes merge properties (there’s often a high stress level before releases). And as soon as this has happened, merges may leave out revisions (or merge revisions twice) and you need in-depth knowledge to repair the damage. In some cases, even rolling back changes on trunk wasn’t enough if someone created a feature branch at the wrong time.

In a project consisting mostly of source files in a compiled language like Java or C#, obvious mistakes typically show up early. If you have lots of HTML, CSS, and XML configuration, or you’re working with an interpreted language, you don’t have this safety net. Your QA department (if you’re lucky enough to have one) will be the first to discover the problems.

So, if you’re a mostly happy Subversion 1.4 user and you’re considering migration to 1.5 or later for merge tracking, then my advice is: Don’t bother. Maybe recent versions fixed some of the problems, but I wouldn’t take chances. You don’t want to spend your nights fixing your source repository.

One of my pet peeves in programming is that few people use guard clauses. A guard clause is an almost trivial concept that greatly improves readability. Inside a method, handle your special cases right away and return immediately.

Have a look at the following example:

private int doSomething() {
    if (everythingIsGood()) {

        /*
         * Lots and lots of code here, but that's a different story.
         */

        return SOME_VALUE;
    } else {
        return ANOTHER_VALUE;  // a special case
    }
}

You can easily rewrite it using the Replace Nested Conditional with Guard Clauses refactoring from Martin Fowler’s Refactoring:

private int doSomething() {
    if (! everythingIsGood()) // <-- this is your guard clause
        return ANOTHER_VALUE;

    /*
     * Lots and lots of code here, but that's a different story.
     */

    return SOME_VALUE;
}

Once you’ve read past the conditional(s) at the beginning of the method, you know that your world is in order and you don’t have to worry about special cases anymore.

I love unit tests and use them whenever possible. But in the end, there’s no substitute for a full scale system test. Only after system testing, you can be sure that everything works as intended. When I was looking for a way to test a REST-style JSON web service, I decided to create a small testing framework to simplify the task.

The result is the JSysTest framework written in Java. Using JSysTest, you can create functional web service tests based on JUnit 4 and its Hamcrest-based assertion DSL. With the help of some custom matchers, a test can look as concise as this:

  assertThat(get("/some-resource"), has(status(404)));

Have a look at the Documentation for a few more examples.

The framework is still in its early stages of development and is likely to change. I still don’t know how to make HTTP POST and PUT methods as elegant as the GET case, but I’m sure a pattern will emerge at some point.

I think the most important lesson to take away from this little experiment is that you can simplify your tests significantly if you invest a bit of time for building a highly specialized DSL.

Exciting news for Java developers: The first 1.0 release candidate of Google Collections has been released. For almost a year I’ve been waiting for this, and now it seems the waiting will soon be over.

Basically, Google Collections is a better, more consistent version of commons-collections with generics and a few cool things that let you almost use a functional style in Java (at least as far as this is possible). I think this library will quickly become a standard dependency in my Java projects.