One of the most often read postings on this blog is the one about my getopts-enabled shell script template. So I figured, a similar template in Python would also be useful. Like the shell script, it’s quite trivial, but it might still save some time.

The template is an example for a typical unix utility. It connects two aspects: An optparse-based command line parser and iterating over multiple input sources using the fileinput module.

I’ve used command line parsers in several programming languages, but from all of them, I liked Python’s optparse module best (for those who need it, the Python’s standard library still provides a classic getopt). The fileinput module is simple but useful. Beyond its main functionality, it supports more advanced features like transparent decompressing (not really Unix-like though) and in-place filtering of files, but I rarely ever use them.

Here’s the template in my online Mercurial repository and here it is for download.

Since Python 2.5, creating histograms has become easier. Instead of dict, we can now use defaultdict which is similar in behavior to awk’s associative arrays. Instead of raising a KeyError for undefined keys, defaultdict adds a user-defined item and returns it.

I’ll demonstrate this with a simple program that analyzes line length distribution in a file. In older Python versions, you’d typically write code like this:

hist = { }
for line in open(filename):
    hist[len(line)] = hist.get(len(line), 0) + 1

The code using defaultdict is much clearer and more elegant (although an additional import is needed):

from collections import defaultdict

hist = defaultdict(int)
for line in open(filename):
    hist[len(line)] += 1

Note that defaultdict’s constructor expects a factory function that initializes unset items on request.

Unless you dump the contents of hist to gnuplot or similar, you might want to sort the dict by value. There are several ways to do this, but I learned from a related blog posting that this is the most efficient way:

from operator import itemgetter
sorted(hist.iteritems(), key=itemgetter(1))

The min and max builtins support the key parameter, too, by the way.

I’m back to playing with the Django web development framework again. Since I’m close to putting something online (it’s only a matter of years, actually), I’m also looking into hosting options. Due to the framework choice, Google App Engine appeared on my radar and I had a quick look at it.

Running your web application on Google’s infrastructure is certainly an intriguing idea. The official into video didn’t tell me too much, so I watched Guido van Rossum’s talk Rapid Development with Python, Django and Google App Engine on YouTube. Based on the information given there I decided against App Engine.

App Engine uses Google’s high performance distributed database system Bigtable. That’s pretty cool, but of course it works a lot different than a relational database. I’d have to port my application to Google’s persistence layer (not that much work yet) but I’d later be tied to a single hosting provider.

With the Django ORM framework out of the way, Django’s built-in generic admin site no longer works. This means, one of the framework’s biggest assets is lost and that’s the real show stopper for me.

If I were to create a highly scalable web application, the advantages would probably outweigh the disadvantages. But in my case, it means going back to comparing classic web hosters.

I’ve got a nice MP3 player from Samsung with 2 GB of flash memory. That’s very little and I can never decide which files to keep and which to delete. Fortunately, the player also supports Ogg Vorbis files that require less space for the same quality. To make the most of my space, I transcode MP3s to Ogg using a simple python script.

The files in my music collection have all been ripped from CD and encoded to MP3 using lame’s "–r3mix" high quality VBR option. Because of that, the transcoding process doesn’t lose too much quality. I don’t hear a difference so I’m happy.

My transcoding script uses GStreamer, a media framework that supports all kinds of audio formats and transformations via its pipes and filters architecture. The framework supports file metadata (like ID3 tags), too, so it’s the perfect fit for transcoding files with little effort.

To transcode a set of MP3 files, let’s say an album that I just bought and ripped, I use the following command:

to-ogg.py -d /media/disk/ARTIST/ALBUM /PATH/TO/*.mp3

This takes the given MP3 files and transcodes them to Ogg Vorbis. The new files are created in the /media/disk/ARTIST/ALBUM/ directory, with ID3 tags converted to Ogg comments. The directory is created on the fly, if necessary.

The script is flexible enough to accept all source formats that GStreamer supports. So, if you can play a file using rhythmbox (which also uses GStreamer as its backend), you can also convert the file to Ogg Vorbis.

Writing long documents in a browser input box isn’t the most pleasant thing to do. The WordPress builtin editor makes things even more inconvenient by forcing you to write your articles in HTML. It would be pretty cool if you could edit articles in a simple, readable format using your favorite text editor. Fortunately, there’s ReStructured Text, a document format frequently used in the Python community.

ReStructured Text (rst) is a pleasant plain text format that can be converted into different formats like HTML or LaTeX. The format is pretty intuitive and well suited for typical blog postings with a bit of formatting and a few links (see this example). Of course, there are also more powerful features, but I don’t use many of them in my articles.

The rst distribution comes with several converter scripts, but none of them is perfectly suited for WordPress. The HTML syntax WordPress uses is slightly simplified and we need just the body, not a full HTML document. Using the rst2s5 converter as a reference I wrote my own converter: rst2wp.

The rst distribution contains a library that parses the text format and turns it into a document tree. A writer class then translates the document tree into a writer-specific format (HTML, LaTeX, S5, …). All I had to do was to inherit from the standard HTMLTranslator class and override all parts of the document I wasn’t interested in. The HTMLTranslator follows the visitor pattern so it’s pretty easy to understand.

Using rst2wp my workflow for posting an article works like this:

1. Write the article in rst format.
2. Check it into my Mercurial repository.
3. Run rst2wp to turn the article into HTML.
4. Paste the output into the browser’s input box.

The script works well for me, but I didn’t test the more advanced features of rst so there’s a good chance that they won’t work. Feel free to tweak the script to your needs, the code is pretty simple. A possible next step would be to write a tool for posting the article directly via WordPress’ web service. But I’ll leave that for another day.

My interest in Grid Computing over the last weeks begins to show. After reading the Google MapReduce paper, I tried my fingers on a client side toy problem.

For formatting purposes, I was interested in the size of the longest string in a sequence. There are lots of ways to do this, but I wanted to try Python’s map and reduce functions.

This is my sample input sequence:

  values = ('abc', 'xy', 'abcdef', 'xyz')

The first step is to transform the values sequence into a new list which contains the lengths of each string:

  mapped = map(lambda x: len(x), values)

After that, the mapped list is reduced to a single value:

  max_len = reduce(lambda x, y: max(x, y), mapped)

Both user-defined map and reduce functions are associative and commutative, so this task can be parallelized easily. In fact, I could in theory hand each map operation off to a different node in a grid. Because of the max function’s properties, the reduce could be executed in multiple steps on a grid, too.

BTW: While map/reduce was a nice experiment, I used a different implementation in the end, using list comprehension and the powerful max() function:

  max( [len(x) for x in values] )

If I had been interested in the longest string itself, I would have used the following (python-2.5 only):

  max(values, key=lambda x: len(x))

Copying files from one host in your home network to another is easy using protocols like NFS or SMB. But if you just want to transfer a couple of files to a friend’s computer in a foreign network, you have a lot of setup work to do (at least on Linux boxes). It would be much easier to just run a web server and let the other host download the files via HTTP in a web browser. Unfortunately, web servers aren’t trivial to setup either.

Gladly, Python’s standard library comes with lots of useful modules: Among them are BaseHTTPServer and SimpleHTTPServer which provide everything needed to implement a simple web server in pure python. The SimpleHTTPServer.SimpleHTTPRequestHandler class is a handler which serves the current working directory in a style similar to apache. Via your browser, you can click through directories and download individual files.

It was a matter of minutes to write the servehttp script. The script accepts parameters to set the server’s port and for the directory to serve. Feel free to adjust it to your needs.

By the way: More advanced servers can be written using Twisted, an extremely flexible tool kit for networking applications.

Mozilla-based browsers like Firefox have inherited a useful feature from the good old Netscape Navigator: Controlling a running browser instance via the command line. Using command line options, you can open URLs or files inside your browser window without having to enter them in the address bar.

Suppose you’re working in a terminal and want to open an HTML file in your current working directory:

  firefox -remote "openurl(file://`pwd`/filename.html)"

Your browser should then open the file in a new tab.

Unfortunately, this is a lot to type, so I wrote a python script called b (for “browser”) which simplifies this. The command above can now be written like this:

  b filename.html

Any number of files may be given on the command line. The script supports absolute and relative file names, directories, and URLs. Whitespace and unprintable characters are handled correctly, too.

See the documentation at mozilla.org for more information about the remote control feature.

On GNOME desktops, the D-Bus IPC standard has superseded the CORBA-based approach. Using D-Bus, the operating system can notify the desktop about hardware changes and applications can communicate with each other in a standardized, simple manner. Using signals it is also possible to get a notification in case there are changes in a remote application.

In this posting I’ll give a simple example on how to control the Rhythmbox audio player remotely using the python bindings.

Similar to CORBA’s IDL, there is an XML-based interface description language. Using this language, applications can announce the methods (including signatures) they provide for others to call. To execute a method, you have to know four things:

1. The service (in this case org.gnome.Rhythmbox)
2. The name of the remote object. Rhythmbox registers three objects for us to use: /org/gnome/Rhythmbox/Player, /org/gnome/Rhythmbox/Shell, and /org/gnome/Rhythmbox/PlaylistManager
3. The interface (org.gnome.Rhythmbox.Player in this example)
4. The method and its signature (getPlayingUri())

Strictly speaking, the interface isn’t necessary, but an object is allowed to provide multiple methods with the same name and signature. In this case, the interface can be used to the access the desired method.

Let’s use python to print the URI of the currently playing song:

#! /usr/bin/env python
import dbus

session_bus = dbus.SessionBus()

proxy_obj = session_bus.get_object(
    'org.gnome.Rhythmbox', '/org/gnome/Rhythmbox/Player')

player = dbus.Interface(proxy_obj, 'org.gnome.Rhythmbox.Player')

print player.getPlayingUri()

This is pretty straight forward. We obtain a dbus.Bus instance (additionally to the SessionBus there is also a SystemBus) and use it to get an object reference to the player. Like in CORBA, a proxy object is used which takes care of all the marshalling and unmarshalling for us. The first argument of get_object() is the name of the service, the second one names the object we’re interested in using an object path.

We select the interface (the object supports more, but we don’t care about them) and then call the getPlayingUri method. Pretty simple, hm?

Rhythmbox offers a lot more than that: You can pause the player, adjust the volume, or print the name of the currently playing song. The D-Bus interface definitions can be found in XML files in the source distribution. For more information about D-Bus see the D-Bus Tutorial.

The interactive python interpreter is a highly useful tool. When writing python code, I always run it in a terminal to test things and to access the documentation. Unfortunately, the built-in line editing capabilities are limited: There’s no history, tab-completion, or other advanced editing features that shells like bash provide. Few people seem to know that it’s simple to add all of that using a few lines of code.

Like bash, the python interpreter can use readline on Unix systems:

try:
	import readline
except ImportError:
	print 'Module readline not available.'
else:
	import rlcompleter
	readline.parse_and_bind('tab: complete')

This code from the rlcompleter module’s documentation has to be executed at startup, so python’s PYTHONSTARTUP mechanism comes to help. Save it to a file (it doesn’t have to be executable) and let the PYTHONSTARTUP environment variable point to its location. On my system, I created a file ~/.pythonstartup.py and set the environment variable in my shell’s profile:

export PYTHONSTARTUP=~/.pythonstartup.py

To test this, start the python interpreter, and import the sys module. Then type sys. and hit the tab key. A list of attributes and functions available in the module should be displayed.

That’s pretty cool already, but it’s not all there is to it. See the readline module’s documentation and the example there for what’s possible.