I love unit tests and use them whenever possible. But in the end, there’s no substitute for a full scale system test. Only after system testing, you can be sure that everything works as intended. When I was looking for a way to test a REST-style JSON web service, I decided to create a small testing framework to simplify the task.

The result is the JSysTest framework written in Java. Using JSysTest, you can create functional web service tests based on JUnit 4 and its Hamcrest-based assertion DSL. With the help of some custom matchers, a test can look as concise as this:

  assertThat(get("/some-resource"), has(status(404)));

Have a look at the Documentation for a few more examples.

The framework is still in its early stages of development and is likely to change. I still don’t know how to make HTTP POST and PUT methods as elegant as the GET case, but I’m sure a pattern will emerge at some point.

I think the most important lesson to take away from this little experiment is that you can simplify your tests significantly if you invest a bit of time for building a highly specialized DSL.

Exciting news for Java developers: The first 1.0 release candidate of Google Collections has been released. For almost a year I’ve been waiting for this, and now it seems the waiting will soon be over.

Basically, Google Collections is a better, more consistent version of commons-collections with generics and a few cool things that let you almost use a functional style in Java (at least as far as this is possible). I think this library will quickly become a standard dependency in my Java projects.

Configuring servers is tedious work if you operate a cluster of more than a few machines (if you’re a regular reader, you’ve heard about it). I’ve created a simple deployment framework that helps with building deployment packages based on central configuration templates. It is ant-based, so people from the Java world should have no problem to adjust it to their needs. Read on to find out how it works.

The Server Deployment Framework (SDF)

Based on a server template, a bit of configuration and your application, this little framework creates a self-contained deployment package ready for installation. The package contains both the server itself and the application, so you always start from a clean slate. There’s no cruft left between installations.

With SDF, there’s a separation of responsibilities between developers and admins. Developers provide the application in a .war file. The .war file has no environment-specific configuration. Admins are responsible for configuring the application for the target environment. The general ideas is that the same application artifact that is handed to the QA department is also installed on the production systems.

This is how it works:

1. Create a server template in the src directory. Replace important configuration values with wildcards. Rename all files containing wildcards to filename.in (think autoconf).
2. Place your web applications in the base directory’s webapps directory.
3. Create/edit the environment configuration in the conf directory, ie. ENV-NAME.properties.
4. Run ant package -Dapp.environment=ENV-NAME.
5. You’ll find your pre-configured deployment package in the target directory.

Repeat steps 2-5 any time you want to make a new release.

The Bundled Tomcat

A few words on the bundled Tomcat instance: It is for illustration only and hasn’t been configured for production use. There are a couple of interesting ideas though.

All ports settings have a @PORT_PREFIX@ wildcard. That way you can run multiple Tomcat instances on one host by changing the prefix parameter. This is useful for rollouts: You can start up and check the new revision while the old one is still running. Using loadbalancer settings, you can switch over to the new application and later shut down the old revision.

In conf/context.xml, an example context parameter has been set. This is a simple way of passing environment-specific parameters to your application without having to change anything inside the .war file.

As a general rule, it’s a good idea to run just one web application per Tomcat instance. This way, it’s easier to figure out which application is misbehaving. In large operations with high performance requirements, this is the normal case anyway. When you follow the single application rule, you can as well turn off hot deployment.

Since applications tend to crash sometimes (OutOfMemory exceptions happen to the best of us), it makes sense to restart the server automatically without admin intervention. One easy solution to do this is to run Tomcat inside a JavaServiceWrapper that makes sure your JVM is in a healthy state. Remote restarts and easy JMX access are helpful additional features.

Most people eventually get it that building release artifacts using their IDEs is not the way to nirvana. Builds have to stay stable and reproducible between IDE revisions, and being able to execute them in a command line environment or especially continuous integration servers is key to agile development.

Fortunately, there are open source build tools for Java. Less fortunately, they’re not quite perfect. Let’s see what we’ve got.

The Sorry State of Affairs

There’s Ant, the undying classic, but pretty much in maintenance mode, I understand. Ant-based build systems usually start small and innocent but quickly turn into a tangled mess. Every Ant script is a unique piece of modern art that has to be deciphered on its own. They are the cockroaches of Java development, and I guess we’ll have to deal with them again in 30 years when we’ll be porting ancient Java applications to the next big platform.

Maven 2 promises relief and introduces convention over configuration, dependency management, lifecycle support and a few other things that aren’t strictly relevant for a build system. Great concepts, but the actual implementation makes you want to cry. The state of documentation is poor, the XML syntax is unnecessarily bulky, and trivial one-off tasks aren’t possible without a proper plugin (go and write one if you’re feeling brave). Not to mention that stunts like multi-module projects are necessary to build a simple EAR file. And no, Eclipse doesn’t like them either.

Despite all its shortcomings, a lot of people including me tried hard to love Maven, but it’s not easy.

The Winds of Change

Where does that leave us?

I don’t have much hope for Ant. It is what it is and lacks too many things a build system needs, most importantly built-in conventions and higher level abstractions. Granted, there’s Ivy, a pretty nice dependency manager that adds a couple of interesting features, but that’s about it (Ivy looks heavily inspired by Maven, but it’s funny how they changed all the terminology).

Maven 2’s future looks a little bit brighter since it seems the project is opening up to common sense, which maybe be a direct effect of recent discussions and probably Don Brown’s patches. For example, the XML syntax in 2.1.0 will be much more concise and there are fixed default plugin versions in 2.0.9 so that it’s easier to have reproducible builds. Unfortunately, the public repository is a swamp of misplaced software and crappy metadata, so there may be no alternative to setting up a local repository.

At the moment, I’m taking a closer look at Buildr, a ruby-based build system for Java that’s currently under incubation at the ASF. Apparently they’re creating a better Maven, taking the proven concepts but none of its implementation. The jury’s still out on whether the average coder will be able to create effective, maintainable build scripts, but we’ll see. Having to learn Ruby may be a problem to some and an incentive to others.

So?

Poor build systems and development environments in general are a barrier for new project members and a frequent cause for frustration. I’ve worked on projects where all you had to do was to check something out from Subversion, type "mvn package" and you were ready to go. For other projects (both commercial and open source!) you needed a specialist’s support, a couple of days time and a lot of hand-holding to actually build the product and run it locally.

Unfortunately, only few development shops realize that a decent development environment is an asset and invest in their build infrastructure. A good project setup takes time, and if there’s no time or budget available, your build tool of choice won’t do any wonders on its own.

A while ago I posted an article that showed how to generate a database schema from JPA annotations. Since I didn’t get the hibernate3 maven plugin working back then I used the antrun plugin as a workaround. Thanks to the help of a reader the plugin works now, so an update is in order.

[Update 2009-06-01: The plugin is now available in version 2.2 from the central maven repository, so I removed the plugin repository configuration. I also switched from annotationconfiguration to jpaconfiguration.]

Since the last article the scenario hasn’t changed: I want to generate DDL scripts for creating a database schema from JPA annotations. The created schema can then be tweaked as necessary and used in the production environment. Like the old antrun solution, the maven plugin uses the ant task from Hibernate Tools, so the resulting schema is the same.

This is how it works. Edit your pom.xml and add the plugin configuration:

<build>
  <plugins>
    <!-- other plugins ... -->
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>hibernate3-maven-plugin</artifactId>
      <version>2.2</version>
      <executions>
        <execution>
          <phase>process-classes</phase>
          <goals>
            <goal>hbm2ddl</goal>
          </goals>
        </execution>
      </executions>
      <configuration>
        <components>
          <component>
            <name>hbm2ddl</name>
            <implementation>jpaconfiguration</implementation>
          </component>
        </components>
        <componentProperties>
          <persistenceunit>Default</persistenceunit>
          <outputfilename>schema.ddl</outputfilename>
          <drop>false</drop>
          <create>true</create>
          <export>false</export>
          <format>true</format>
        </componentProperties>
      </configuration>
    </plugin>
  <plugins>
</build>

Don’t forget to set the persistence unit to whatever you declared in persistence.xml.

Schema creation is executed in the process-classes phase, so you’ll find the schema in target/hibernate3/sql/schema.ddl after running mvn package (or any later phase). An alternative is to remove the <executions> element and to run the hbm2ddl goal from the command line:

  mvn hibernate3:hbm2ddl

The plugin offers a lot more than just schema creation. See the plugin documentation for more information.

On a GridGain cluster, you sometimes want to execute your jobs on only a subset of the nodes available: those nodes meeting a given condition. Let’s say some nodes run an expensive piece of thirdparty software that is (fortunately) only needed for a couple of tasks. At another time, the jobs should be executed on a different subset, or maybe on all nodes of the cluster.

There are several ways to do this: Filtering out nodes in your map() method would work, but you usually don’t want that if you’re using split() already. Or you could put the nodes on different multicast groups, thus filtering via DiscoverySpi. In that case it’s more difficult to run a task’s job on all nodes: You can no longer use the default discovery service provider; you’d have to pick a different one or even implement your own.

Luckily, there’s the TopologySpi. It is used by the framework to filter the set of nodes returned by discovery. Any strategy for filtering can be implemented, providing a perfect hook for a criteria based filter. For example, the service provider shipped with GridGain, GridBasicTopologySpi, makes it possible to execute jobs only on the local node or only on remote nodes.

To solve the problem described above, I developed a simple group concept based on user attributes that can be assigned to each grid node in its configuration. A node may be part of one or more groups. This is added to the configuration file of the worker grid nodes like this:

  <property name="userAttributes">
    <map>
      <entry key="grid.groups">
        <set>
          <value>foo</value>
          <value>bar</value>
        </set>
      </entry>
    </map>
  </property>

A custom topology provider is used on the node where my tasks are deployed. It’s derived from GridBasicTopologySpi, but you can also specify which groups a node has to be part of to qualify for the task. The provider is called GroupTopologySpi (the “i” is used consistently in GridGain, although it’s counter-intuitive) and is activated in the master’s configuration:

  <property name="topologySpi">
    <bean class="de.mafr.grid.GroupTopologySpi">
      <property name="localNode" value="true"/>
      <property name="remoteNodes" value="true"/>
      <property name="requiredGroups">
        <set>
          <value>foo</value>
        </set>
      </property>
    </bean>
  </property>

If the requiredGroups property isn’t given or if it contains the empty set, the service provider works exactly the same as GridBasicTopologySpi, so it can be used as a drop-in replacement. In the example only grid nodes in the group foo will take part in the task.

The implementation is simple and consists of one short class and an interface: The actual topology provider, GroupTopologySpi, and the MBean interface.

This weekend I finally had the time to take a look at GridGain, a computational grid package written in and for Java. GridGain is a an open source product licensed under LGPL-2.1 (the same as JBoss) with minor portions under the Apache 2.0 license, so use in commercial products is possible. Since we’re doing medium-sized number crunching at work, I definitely wanted to give it a try. The following article is no in-depth evaluation but rather a first impression after a few days of reading and experimenting with version 1.6.1.

Like I said, GridGain is a computational grid, which means it focuses on the computational part. Getting meta data (job input parameters etc.) to your workers is easily possible, of course, but transfer of large amounts of input data is entirely up the user. GridGain is no data grid, but it provides optional integration with Oracle Coherence, a commercial, closed source data grid and caching package.

The programming paradigm behind GridGain is Map/Reduce, a divide and conquer approach which has been made popular by Google. The underlying idea is a simple one: Split your problem into smaller parts and execute them in parallel on network nodes (the map part). When the nodes are done, aggregate the individual results to obtain the final result (reduce). If you’re able to express your problem in terms of map and reduce operations, you can scale out to many nodes and solve large problems.

GridGain is a pretty young product, so there isn’t much to be found about it on the web. That leaves the official documentation (and the source code) as the only sources of information. Fortunately, the documentation (Javadoc and the read-only wiki) is in a good state already and gets you up to speed quickly.

In retrospect, I spent too much time reading the wiki documentation though. It’s excellent to get a basic grasp of GridGain’s concepts and features, but there’s no substitute for actually writing a simple application yourself. As soon as I did that, a lot of things suddenly cleared up and and I gained much more confidence. I quickly implemented a GridTask (containing the map and reduce functionality) and a GridJob (containing the actual processing logic) and distributed it to nodes on the local network. I even added an MBean for monitoring the entire task just because it was so simple to do. The comprehensive API documentation helped a lot in the process.

To make things easier for new users, I think it would help to restructure the tutorials a bit. In my opinion, the @Gridify annotation (a way to grid-enable a method) is a bit too magic for a beginner in his or her first two hours with GridGain, as it stands in the way of understanding the basics: The simple and elegant GridTask and GridJob abstractions. I’d also move the anonymous GridJob implementations out of the GridTask in the examples. That would make the code easier to read and understand.

From what I’ve seen, GridGain’s source code is clean and well-written. The authors obviously have a great deal of design experience which results in a modular, flexible system. Like many applications using the Spring Framework, GridGain is made up of several services implementing service provider interfaces (SPIs). For example, finding grid nodes on a network is done using a discovery service, which implements the DiscoverySpi interface. The default is a multicast-based discovery (with lots of other implementations available), but users could create their own service easily. It is possible to configure many aspects of the provided services or use your own services to change policies. For the start, I didn’t have to do that, however, since things generally worked out of the box.

Over the next few months it will be quite interesting to see if GridGain is able to build a community around their software. A great step towards this is the public online forum. Employees from GridGain Systems (the company behind GridGain) answer questions in a polite and helpful manner. The forum has a good search function, which proved to be very useful when I had questions.

Personally, I’m also interested in a public maven proxy – something many users would benefit from – and also a public source code repository. I know that companies are often hesitant to go this far, but it has proven to be a great asset to open source projects, since it allows advanced users to track the project’s progress in detail. I’d also appreciate build instructions (from what I’ve seen, the build system isn’t shipped in version 1.6.1), in case I’d have to do emergency bug fixes to GridGain. Experience shows that things like that are necessary in any software product from time to time.

To sum things up, my overall impression is a very positive one: The package is really a lot of fun to work with, as the website promised. I think everybody who considers doing serious number crunching on the Java platform should take the time to evaluate GridGain.

Handling configuration data in Java isn’t as convenient as it could be. The old and popular Properties mechanism has an awkward syntax and you have to take care of type safety yourself. The new Preferences mechanism supports XML but still isn’t everything I’d like it to be.

Recently, I discovered an alternative while playing with GridGain: Spring Beans. Despite all the hype over the last years I’ve never worked with Spring before, so I had some catch-up to do. Spring provides – among many other things – an Inversion of Control (IoC) container, similar in style to packages like PicoContainer/NanoContainer or Hivemind. The main service of these containers is Dependency Injection, a non-intrusive way of obtaining dependencies for objects (data sources are a popular example). In this article, I’m focusing on Spring and the configuration aspect.

Spring provides a simple XML format that we can use for configuration files:

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">

  <bean id="config" class="de.mafr.demo.Config">
    <property name="hostname" value="mafr.de"/>
    <property name="port" value="8080"/>
  </bean>

</beans>

In the example, we declare a Java Bean (de.mafr.demo.Config) and assign values to its properties. The Config bean is a POJO that follows the Java Bean conventions and isn’t tied to Spring in any way. Here it is:

package de.mafr.demo;

public class Config {
	private String hostname;
	private short port;

	public String getHostname() {
		return hostname;
	}

	public void setHostname(String hostname) {
		this.hostname = hostname;
	}

	public short getPort() {
		return port;
	}

	public void setPort(short port) {
		this.port = port;
	}
}

Note that type conversion is done by the container: The port value is converted to int automatically. This works for all base types and for a few other classes like URL.

Now let’s actually read the configuration file and obtain a ready to use Config object:

package de.mafr.demo;

import org.springframework.beans.factory.BeanFactory;
import org.springframework.beans.factory.xml.XmlBeanFactory;
import org.springframework.core.io.FileSystemResource;

public class Main {
    public static void main(String[] args) {
    	BeanFactory factory = new XmlBeanFactory(
    			new FileSystemResource("config.xml"));

    	Config config = (Config) factory.getBean("config", Config.class);

        // use config
    }
}

If there’s an error, you are notified immediately by an exception from the constructor. The configuration file can be read from disk directly like in the example above (using FileSystemResource), from the classpath (ClassPathResource), a servlet context (ServletContextResource), from a URL (UrlResource), or other sources.

To get the example up and running, a subset of Spring and a few dependencies are required. If you’re using Maven, all you have to do is to add the following dependency to your pom.xml:

    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-beans</artifactId>
      <version>2.0.7</version>
    </dependency>

A lot more is possible with Spring Beans. See the official reference manual for all the details.

Setting up a Java EE application is no trivial task. Dependency management, building, and deployment can get pretty complex and require a thorough understanding of deployment descriptors, jars, ears and other artifacts.

In this article, I’m going to present a truly minimal maven-based setup for the JBoss Application Server (tested with jboss-4.0.5.GA), providing a solid base for further work. The project is an EJB3 demo application which uses JPA to access a database and makes the data available via a remote session bean interface. The application will be deployed in an ear file which contains the required dependencies (unless they are already provided by JBoss).

First of all, download the application and extract it. In the root directory, there’s a pom.xml file (the POM) which defines the top-level project. Each of the subdirectories (ear, ejb, and client) is a maven project itself with a pom.xml on its own. Apart from some basic values, the top-level POM specifies its sub-modules and gives a list of dependencies for the project. The dependency list is just a declaration, its purpose is to let the other POMs reference dependencies without having to specify the version number.

The ejb module is the heart of the application. It contains the server side source code, which is the data source layer (DAOs), JPA Entities, and a remote facade (in this case: a stateless session bean). The ejb module creates two jar files: One containing all the code, and the other one with all the interfaces and data transfer objects relevant for remote clients (the client-jar).

The ear module contains no source code, its purpose is to configure the deployment descriptor (application.xml) and to package the ejb module’s jar file and all the dependencies into a single ear file, ready for deployment into the jboss application server. You’ll find the created file in ear/target.

The client module implements a simple client application which accesses the server via its remote facade. It can be run from the command line or from within eclipse.

To build the project, run mvn install from the command line. If you want to import the project into eclipse, execute mvn eclipse:eclipse to generate the required .project and .classpath files. From within the IDE, choose File/Import, General/Existing Projects into Workspace, browse to the project’s base directory and hit Finish.

Update: I updated the demo application to version 0.2. The new version removes an unused import script that caused confusion and the documentation notes where the JBoss dependencies are coming from.

When using an ordinary jar library, IDEs like Eclipse’s JDE don’t have enough information to display inline javadocs for referenced classes. You have to manually direct Eclipse to the library’s source code to take advantage of inline documentation. Maven’s source plugin provides a convenient solution to this problem. When building the library, the plugin creates a source jar containing the package’s source code. This source package can be installed or deployed into your repository. Maven’s eclipse plugin then adds the source package to the generated .classpath file.

This is how it works. Before installing or deploying your library, run the source plugin:

  mvn source:jar

Your next install or deploy call will take care of the rest. In the dependent project (the one using your library), run the eclipse plugin:

  mvn eclipse:eclipse

After refreshing the project in eclipse, you should now have access to inline documentation.

If you don’t want to run source:jar each time, you can attach the call to the build lifecycle’s package phase:

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-source-plugin</artifactId>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>jar</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

After running mvn package, there should be a source package in your target directory.