Everybody knows that Unix shell utilities are powerful. Even though they’re text-based, you can build a lot of useful things outside of the text domain. Today I’ll show you how to implement set operations. All we need are sorted files as input, with each file representing a set.

Let’s create some input files A and B:

$ echo a b d e | tr ' ' '\n' > A
$ echo a c e f | tr ' ' '\n' > B

Note: It’s an important precondition is that all input files are sorted! To avoid ugly surprises, the files mustn’t contain duplicates, but you would expect that from a set.

Since the input files are sorted, equality is trivial:

$ cmp A B

For use in scripts, the silent switch -s is useful.

Intersection:

$ comm -12 A B

The comm utility outputs three columns per default: The lines unique to the first file, the lines unique to the second file and the lines appearing in both files. We turn off the first two columns leaving just the intersection.

Union:

$ sort -m A B | uniq

The merge switch -m isn’t necessary for correctness, but it improves runtime for large files quite a bit. Unfortunately, not all sort implementations have it.

Difference:

$ comm -23 A B

This only displays the lines unique to A, leaving the result of A minus B.

You can use the difference to get the complement. This requires a universe U first that we create using our union construct. The we subtract A from U:

$ sort -m A B | uniq | comm -23 - A

With only two files, this is just an inefficient way of subtracting A from B.

Contains can be implemented using grep, but we have to be careful about false positives:

$ grep -Fx e A

The quiet switch -q is useful in scripts and may also reduce runtime because it exits as soon as the entry is found. This is still far less efficient than it could be if we had fixed size records and used interval halving.

More advanced operations like testing if one set is a subset of another are possible, too. Often you will also have to compare line counts using wc -l, but I’ll leave that as an exercise to the reader :)

A while ago, I blogged about my self-made backup solution, a simple shell script using rsync to create snapshot backups. This wasn’t my first attempt at establishing backups for my private workstation but the most successful one so far. The backup takes less than 5 minutes for about 10 GB of data in my home directory. It’s a quick and painless process, and a repeating appointment in my calendar reminds me not to forget it.

A friend of mine (hey Mike!) liked the idea, bought an external USB drive and started to use the script, too. Being an experienced Unix user and hacker, he sensed room for improvement and we tweaked the script to handle errors more gracefully and to provide better diagnostics in case things go wrong. The script has about tripled in size since its original release, but with its 80 lines and comprehensive comments it’s still easy to extend and understand. Thanks a lot, Mike!

You can get the script and the rsync filter file from the usual place.

One could assume that I have run out of material and retreat to safer ground, but far from it. From now on I’ll just throw in a few quick shell hacks hoping some of you don’t know them yet. So, here it goes …

Unix pipes are cool, but it’s quite a limitation that a program can only read from a single pipe. Sure, there are named pipes, but it’s a real pain to use them and we’d be leaving oneliner zone. Fortunately, recent versions of bash support implicit FIFOs that are created on demand by the shell. All you need is a bit of additional syntax.

Suppose you want to concatenate two compressed CSV files. The minor complication here is that you only want the header of the first file. With implicit FIFOs that’s easily possible:

$ zcat data_1.csv.gz | cat - <(zcat data_2.csv.gz | sed 1d) > out.csv

The sed(1) command deletes the first line of its input stream removing the CSV file’s header (one header should be enough in most cases). If you like symmetry, you can use this equivalent version:

$ cat <(zcat data_1.csv.gz) <(zcat data_2.csv.gz | sed 1d) > out.csv

In another use case you might want to check which files are in directory dir1 but not in dir2:

$ comm -23 <(ls dir1 | sort) <(ls dir2 | sort)

Try the following if you’re interested in where the FIFO is created:

$ echo <(echo test)

There are implicit FIFOs for output, too, but I never found a good use case for them (maybe they’re useful for complex tee(1) constructs). For more information see section "Process Substitution" in your bash(1) man page.

In programs written in C, parsing of command line arguments has always been done using the getopt(3) library function. This function has set the standards Linux/Unix users have come to expect from command line interfaces. Fortunately, there’s a getopt(3) equivalent for almost every programming language and the shell is no exception.

The getopts command available in all POSIX-compliant Bourne shell derivates (like bash, dash, or ksh) provides convenient command line parsing capabilities. It is a builtin accepting POSIX-style argument lists (as opposed to GNU-style, which is a bit more fancy) and should not be confused with the getopt utility.

For my shell scripts, I use the following template to implement command line parsing:

#! /bin/sh

USAGE="Usage: `basename $0` [-hv] [-o arg] args"

# we want at least one parameter (it may be a flag or an argument)
if [ $# -eq 0 ]; then
	echo $USAGE >&2
	exit 1
fi

# parse command line arguments
while getopts hvo: OPT; do
    case "$OPT" in
	h)	echo $USAGE
		exit 0
		;;
	v)	echo "`basename $0` version 0.1"
		exit 0
		;;
	o)	OUTPUT_FILE=$OPTARG
		;;
	\?)	# getopts issues an error message
		echo $USAGE >&2
		exit 1
		;;
    esac
done

shift `expr $OPTIND - 1`

# access additional parameters through $@ or $* as usual or using this loop:
for PARAM; do
    echo $PARAM
done

# EOF

More command line switches can be added to the loop. If you want to add an -x switch requiring an argument (-x arg) and a y flag without an argument, you would have to change the getopts call to getopts hvo:x:y OPT and add two case labels to the loop. Note that the colon indicates that an argument is required for a flag.

This week at work I had the opportunity to build an RPM for some third-party software package.

My previous experiences with RPM date back to SuSE 5 almost ten years ago and I haven’t touched it ever since. Packaging for Debian or even NetBSD is mostly a painless process, so I hoped today’s RPM wouldn’t be much different. Things worked out fine, except for one thing: RPM uses a well-known base directory (/usr/src/packages on SuSE) which only root can write to. Sure, you can pass ownership to another user, but without root access in the first place you’re out of luck.

Using ~/.rpmmacros, users can define their own private staging area, but that’s no elegant solution either. Not even The RPM Guide came up with a usable receipe for this.

After some unsuccessful experiments, I decided to have a look at how python’s distutils solve this problem. With distutils, you can build RPMs from python packages (python setup.py bdist_rpm) without superuser privileges. That’s why I figured it had to be possible.

Distutils uses rpmbuild’s --define switch (undocumented in the man page) and the _topdir macro:

  rpmbuild -ba --define "_topdir /path/to/staging_dir"
     /path/to/staging_dir/SPECS/package.spec

Note that /path/to/staging_dir has to be an absolute path, leading to the directory which contains the same directory structure /usr/src/packages has. `pwd` can be used to avoid hardcoding the path.

From a script, I create the directory structure in my local working directory, copy source tarball and spec file to SOURCES and SPECS and execute rpmbuild.

The created RPM can then be found in the RPMS directory.

When you work with the interactive shell, there’s a common pattern that is used a lot: Copying or moving files to a directory and then changing to the target directory using cd. In these cases, you find yourself typing the name twice:

  $ cp file1 file2 file3 /very/long/path/name
  $ cd /very/long/path/name

Tab completion helps, but for long names it’s still tedious.

One of my favourite tools, the bash function g (“go”), shortens this to the following:

  $ cp file1 file2 file3 /very/long/path/name
  $ g

It works by getting the previous command from the bash history, treating the last argument as a directory and then going there. If the last argument is a file name, the g function tries to resolve it to a directory name by using the basename or stripping off common archiver suffixes like tar, tar.gz or zip.

The function is one of the most executed commands in my shell history. I also had a companion function which changed back to the previous directory, but since cd - is pretty short, too, it didn’t get used that often. To use it, download the function and simply copy it into your ~/.bashrc file.

Note that this function works on bash only. I used to have a Korn shell compatible function, but it didn’t work as nicely as this one.

To control a D-Bus-enabled application, you need the interface definition to find out which methods are offered and which parameters they expect.

Of course, you can get the relevant interface definition from the application’s source distribution. But there’s an easier way: Ask the object in question via D-Bus to give you the interface description. All objects implementing the org.freedesktop.DBus.Introspectable interface offer an Introspect() method that returns an XML document describing the methods and their signatures.

Instead of python, we’ll use the dbus-send command line utility this time. The following command dumps the definition of rhythmbox’ /org/gnome/Rhythmbox/Player object:

  dbus-send --session --type=method_call --print-reply
      --dest=org.gnome.Rhythmbox
      /org/gnome/Rhythmbox/Player
      org.freedesktop.DBus.Introspectable.Introspect

The service to query is org.gnome.Rhythmbox and the object’s interface is org.freedesktop.DBus.Introspectable. Note that for dbus-send, the method’s name (Introspect here) has to be appended to the interface. See the dbus-send manpage for more information, like calling methods with parameters.