Gaggle Genome Browser

There's a certain windmill I've been tilting towards for a couple of years now. It's known as the Gaggle Genome Browser and we've published a paper on it called Integration and visualization of systems biology data in context of the genome.

The Gaggle Genome Browser is a cross-platform desktop program, based on Java and SQLite for interactively visualizing high-density genomic data, joining heterogeneous data by location on the genome to create information-rich visualizations of genome organization, transcription and its regulation. As always, a key feature is interoperability with other bioinformatics apps through the Gaggle framework.

Here it is displaying some tiling microarray data for Sulfolobus solfataricus. Click for a bigger graphic. The reference sample is shown in blue circles overlaid with segmentation in red. Eight time-points along a growth curve are plotted as a heat map - red indicating increased transcription relative to the reference; green indicating decreased transcription. We also show Pfam domains, predicted operons, and some previously observed non-coding RNAs, several of which we were able to confirm.

One of the features I'm most proud of is the integration with R, a tactic also being used by MeV. At this point it's only partially complete. There's quite a bit more that could be done with it, and I'm looking for time (or help!) to finish.

The past couple of years have seen a whole crop of new genome browsers. See the entry browsing genomes for a partial list. One reason is a new generation of lab hardware and techniques, including ChIP-chip, tiling arrays and high-throughput next-generation sequencing. Another is the ever changing landscape in computing.

It's lacking polish in some places. There's plenty yet to be done. Maybe later, I'll write up some lessons learned and mistakes made, but for now, I'm happy to have it published and out there.

Read more about the biology here:

• Prevalence of transcription promoters within archaeal operons and coding sequences.

Check out the screencast by OpenHelix here:

• Tip of the Week: Gaggle Genome Browser

Layered architecture for a Java application

This is a template for an application architecture that came out of my genome browser project. This Java/Swing desktop application aspires to be very flexible - implementing new visualizations and accommodating task-specific components when necessary. I'm not sure how well I did at this goal, so don't take this as here's-what-you-should-do, just here's-what-one-poor-schmuck-did.

The Application object holds application scope state, and binds components together, performing dependency injection where necessary. The entry point is myapp.Main, which configures Options using command-line arguments and starts the application. The application manages a concurrent event dispatch queue.

Components communicate with each other by events placed on the event queue and have access to an application API. Like in Swing, component code may run in the thread of the event dispatch, or, if it is long running, spin up its own thread. Components are responsible for managing their own threading and synchronization.

Layering, in this template, works something like this:

+--------+
|   UI   |
+--------+
+-----------------+ +----------------+
|   Application   | |   components   |
+-----------------+ +----------------+
+--------------+
|   services   |
+--------------+
+------------+ +----------+
|   domain   | |   util   |
+------------+ +----------+

Higher layers can depend on lower layers, while dependencies may not flow up. Utilities and domain model can have no internal dependencies and anything may depend on them. Services are things like file I/O, persistence, and maybe algorithms. The application exposes an API to components and will in turn implement that API in terms of components. I think this circular dependency doesn't bother me too much. If it did, the Application could have no dependencies on the components and the components could have some organized dependency structure among themselves. The UI sits on top where nothing should depend on it.

Autocompletion and Swing

I can remember being asked to implement cross-browser autocompletion in 2000 and telling my employers that it couldn't be done. We got a prototype working on Netscape (remember when that was the most advanced browser?) but it was buggy on Internet Explorer (remember when IE was the bane of every web developer's existence? Wait, some things never change...) and latency was way too high for most users. Anyway, I didn't last long in that gig, but I still think that for practical purposes at the time I was right. Of course, things are different now.

Type something into Google or Amazon's search box and you'll get a nice drop-down list of possible completions. For a biological example, check out NCBI's BLAST. Make sure the database chooser reads "Nucleotide collection" and start typing "Pyrococcus". Nice, huh?

Several ways to give poor abandoned Swing an autocompleting upgrade are documented in a java.net article. Sadly, they all seem to suffer from one deficiency or another.

The JIDE common layer an open source library that spun out of Jide's commercial offerings seems to be the most stable, but isn't nearly as convenient as the javascript versions. GlazedLists does a nice job, but it's currently (still) broken on OS X. Out of the solutions I found, GlazedLists seems most promising, especially if that bug gets fixed.

I also checkout out the Substance Java look & feel. It looks really sharp. The developer has done some really slick transitions -- highlights that fade in and out or components that expand like icons on OS X's dock. It's probably great on Windows, but unfortunately, it didn't seem very stable on the Mac.

There should be some good lessons to be learned from the failure of Swing. It seems apparent that several developers who are a lot smarter than me have tried to get Swing to cough up a decent UI. The results seem to be consistently limited. Not that some aren't impressive; they are. But the limits to the success of some very good developers speaks very loudly.

More attempts...

• AutoCompleteCombo by Exterminator13 (haha) -

  Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException: -1
   at java.util.ArrayList.get(ArrayList.java:323)
   at dzone.AutoCompleteCombo$Model.getElementAt(AutoCompleteCombo.java:476)

• An autocomplete popup by Pierre Le Lannic, which seems to work as long as you don't care about upper case.
• Java2sAutoTextField from Sun

User directory structures

Some OS's call them home or user home directories, document directories. Here's a quick guide to how different operating systems typically organize user directories.

Directory Type	Directory path returned for each operating system
User Home	Windows XP: C:\Documents and Settings\<user> Windows Vista: C:\Users\<user> Mac OSX: /Users/<user> Linux: /home/<user>
User Desktop	Windows XP: C:\Documents and Settings\<user>\Desktop Windows Vista: C:\Users\<user>\Desktop Mac OSX: /Users/<user>/Desktop Linux: /home/<user>/Desktop
User Documents	Windows XP: C:\Documents and Settings\<user>\My Documents Windows Vista: C:\Users\<user>\Documents Mac OSX: /Users/<user>/Documents Linux: /home/<user>
User Application Data	Windows XP: C:\Documents and Settings\<user>\Local Settings\Application Data Windows Vista: C:\Users\<user>\AppData\Local Mac OSX: /Users/<user>/Library/Application Support Linux: /home/<user>
User Preferences	Windows XP: C:\Documents and Settings\<user>\Local Settings\Application Data Windows Vista: C:\Users\<user>\AppData\Local Mac OSX: /Users/<user>/Library/Preferences Linux: /home/<user>
Public Documents	Windows XP: C:\Documents and Settings \All Users\Documents Windows Vista: C:\Users\Public\Documents Mac OSX: /Library/Application Support Linux: /usr/local/
Public Application Data	Windows XP: C:\Documents and Settings\ All Users\Application Data Windows Vista: C:\ProgramData Mac OSX: /Library/Application Support Linux: /usr/local/
Public Preferences	Windows XP: C:\Documents and Settings\ All Users\Application Data Windows Vista: C:\ProgramData Mac OSX: /Library/Preferences Linux: /etc
System Libraries	Windows XP: C:\Windows\System32 Windows Vista: C:\Windows\System32 Mac OSX: /Library/Frameworks Linux: /usr/lib
Application Files	Windows XP: C:\Program Files Windows Vista: C:\Program Files Mac OSX: /Applications Linux: /usr/local/
Volume Root	Windows XP: C:\ Windows Vista: C:\ Mac OSX: / Linux: /
Temp	Windows XP: C:\Documents and Settings \<user>\Local Settings\Temp Windows Vista: C:\Users\<user>\AppData \Local\Temp Mac OSX: /private/tmp/folders.501 /TemporaryItems Linux: /tmp

Thanks to National Instruments for the data.

See Wikipedia entries Home directory and My Documents. If you're in Java land, values of the os.name property are documented here and here.

Great example of what not to do

In case anyone was wondering, one of these web sites got it right. The other got it wrong. First, have a look at the query page from the bug database on java.net: Now, consider the front page of alltheweb.com: Some poor shlep put a lot of work into that query page for the bug database. I'm tempted to shake my head and say, "WTF?", but I know I've been guilty of this kind of thing. To be fair, java.net has a much easier search box, hidden off to the side where bozos like me won't find it. So, here's a reminder not to do shit like that.

Bug in GlazedLists' AutoCompleteSupport?

I found what looks to me like a couple bugs in the AutoCompleteSupport of GlazedLists. I'm not going to talk about how cool GlazedLists is - it's very cool. I just want to document the bugs. Either that, or document that I'm a bonehead and I'm doing something bass-ackwards with a perfectly good library.

Note: I am doing something boneheaded. I'm documenting a known bug (bug 458). I spammed GlazedLists' mailing list. The first issue, selecting items from the popup, I reported as bug 469.

Here's my repro. I'm working on OS X with Java 10.5.0_16, which I suspect is a factor.

package org.cbare.testglazedlists;

import javax.swing.*;

import ca.odell.glazedlists.BasicEventList;
import ca.odell.glazedlists.EventList;
import ca.odell.glazedlists.swing.AutoCompleteSupport;


public class SpeciesChooser {
 JFrame frame;
 AutoCompleteSupport autocomplete;
 private JComboBox chooser;

 public SpeciesChooser() {
  initGui();
 }

 private void initGui() {
  Box vbox, hbox;
  frame = new JFrame("Species Chooser");
  
  vbox = Box.createVerticalBox();
  vbox.setBorder(BorderFactory.createEmptyBorder(12, 12, 12, 12));
  hbox = Box.createHorizontalBox();
  
  hbox.add(new JLabel("Select Species:"));
  chooser = new JComboBox();
  chooser.setPrototypeDisplayValue("Marmoset");
  hbox.add(chooser);
  SwingUtilities.invokeLater(new Runnable() {
   public void run() {
    autocomplete = AutoCompleteSupport.install(chooser, getSpecies());
   }
  });

  vbox.add(hbox);
  frame.add(vbox);
  frame.pack();
  frame.setVisible(true);
 }

 private EventList getSpecies() {
  EventList result = new BasicEventList();
  result.add("Marmoset");
  result.add("Monkey");
  result.add("Moose");
  result.add("Mouse");
  result.add("Spaaa");
  result.add("Spider");
  result.add("Spidooo");
  return result;
 }

 public static void main(String[] args) {
  SpeciesChooser s = new SpeciesChooser();
 }
}

I used Jing to make a little screencast of my flailing about. Instructions for the repro follow:

First, if I type a prefix which matches several item in the list, and I use the down-arrow to scroll through the matches and pick one, it fails to update the context of the text box. In my example, I type "mo" which matches, Monkey, Moose, Mouse. I press down-arrow twice to select Moose then press Enter. "Monkey" is still in the text box. If I repeat that a couple of times, the popup then shows all choices, not just those matching my prefix. Maybe that's intentional.

Second, if I type something that matches nothing in the list, say "moz" and then backspace over the "z" I get an exception. The stack trace is shown here:

Exception in thread "AWT-EventQueue-0" java.lang.IndexOutOfBoundsException: Index: 1, Size: 0
    at java.util.ArrayList.RangeCheck(ArrayList.java:546)
    at java.util.ArrayList.get(ArrayList.java:321)
    at ca.odell.glazedlists.impl.gui.ThreadProxyEventList.applyChangeToCache(ThreadProxyEventList.java:175)
    at ca.odell.glazedlists.impl.gui.ThreadProxyEventList.access$600(ThreadProxyEventList.java:68)
    at ca.odell.glazedlists.impl.gui.ThreadProxyEventList$UpdateRunner.listChanged(ThreadProxyEventList.java:237)
    at ca.odell.glazedlists.event.ListEventAssembler$ListEventFormat.fire(ListEventAssembler.java:412)
    at ca.odell.glazedlists.event.ListEventAssembler$ListEventFormat.fire(ListEventAssembler.java:409)
    at ca.odell.glazedlists.event.SequenceDependenciesEventPublisher$SubjectAndListener.firePendingEvent(SequenceDependenciesEventPublisher.java:445)
    at ca.odell.glazedlists.event.SequenceDependenciesEventPublisher.fireEvent(SequenceDependenciesEventPublisher.java:344)
    at ca.odell.glazedlists.event.ListEventAssembler.commitEvent(ListEventAssembler.java:316)
    at ca.odell.glazedlists.impl.gui.ThreadProxyEventList$UpdateRunner.run(ThreadProxyEventList.java:225)
    at ca.odell.glazedlists.impl.swing.SwingThreadProxyEventList.schedule(SwingThreadProxyEventList.java:33)
    at ca.odell.glazedlists.impl.gui.ThreadProxyEventList.listChanged(ThreadProxyEventList.java:118)
    at ca.odell.glazedlists.event.ListEventAssembler$ListEventFormat.fire(ListEventAssembler.java:412)
    at ca.odell.glazedlists.event.ListEventAssembler$ListEventFormat.fire(ListEventAssembler.java:409)
    at ca.odell.glazedlists.event.SequenceDependenciesEventPublisher$SubjectAndListener.firePendingEvent(SequenceDependenciesEventPublisher.java:445)
    at ca.odell.glazedlists.event.SequenceDependenciesEventPublisher.fireEvent(SequenceDependenciesEventPublisher.java:344)
    at ca.odell.glazedlists.event.ListEventAssembler.commitEvent(ListEventAssembler.java:316)
    at ca.odell.glazedlists.FilterList.constrained(FilterList.java:389)
    at ca.odell.glazedlists.FilterList.changeMatcher(FilterList.java:286)
    at ca.odell.glazedlists.FilterList.changeMatcherWithLocks(FilterList.java:269)
    at ca.odell.glazedlists.FilterList.access$100(FilterList.java:51)
    at ca.odell.glazedlists.FilterList$PrivateMatcherEditorListener.changedMatcher(FilterList.java:443)
    at ca.odell.glazedlists.matchers.AbstractMatcherEditor.fireChangedMatcher(AbstractMatcherEditor.java:115)
    at ca.odell.glazedlists.matchers.AbstractMatcherEditor.fireConstrained(AbstractMatcherEditor.java:73)
    at ca.odell.glazedlists.matchers.TextMatcherEditor.setTextMatcher(TextMatcherEditor.java:321)
    at ca.odell.glazedlists.matchers.TextMatcherEditor.setFilterText(TextMatcherEditor.java:292)
    at ca.odell.glazedlists.swing.AutoCompleteSupport.applyFilter(AutoCompleteSupport.java:1271)
    at ca.odell.glazedlists.swing.AutoCompleteSupport.access$2300(AutoCompleteSupport.java:209)
    at ca.odell.glazedlists.swing.AutoCompleteSupport$AutoCompleteFilter.postProcessDocumentChange(AutoCompleteSupport.java:1497)
    at ca.odell.glazedlists.swing.AutoCompleteSupport$AutoCompleteFilter.remove(AutoCompleteSupport.java:1450)
    at javax.swing.text.AbstractDocument.remove(AbstractDocument.java:572)
    at javax.swing.text.DefaultEditorKit$DeletePrevCharAction.actionPerformed(DefaultEditorKit.java:1030)
    at javax.swing.SwingUtilities.notifyAction(SwingUtilities.java:1576)
    at javax.swing.JComponent.processKeyBinding(JComponent.java:2772)
    ...

In spite of my whining, GlazedLists is very well thought out and a nice piece of work. For later reference, I also reported bug 472.

Split split

Apparently, there's some disagreement about what it means to split a string into substrings. Biological data frequently comes in good old fashioned tab-delimited text files. That's OK 'cause they're easily parsed in the language and platform of your choice. Most languages with any pretention of string processing offer a split function. So, you read the files line-by-line and split each line on the tab character to get an array of fields.

The disagreement comes about when there are empty fields. Since we're talking text files, there's no saying, "NOT NULL", so it's my presumption that empty fields are possible. Consider the following JUnit test.

import org.apache.log4j.Logger;
import static org.junit.Assert.*;
import org.junit.Test;

public class TestSplit {
  private static final Logger log = Logger.getLogger("unit-test");

  @Test
  public void test1() {
    String[] fields = "foo\t\t\t\t\t\t\tbar".split("\t");
    log.info("fields.length = " + fields.length);
    assertEquals(fields.length, 8);
  }

  @Test
  public void test2() {
    // 7 tabs
    String[] fields = "\t\t\t\t\t\t\t".split("\t");
    log.info("fields.length = " + fields.length);
    assertEquals(fields.length, 8);
  }
}

The first test works. You end up with 8 fields, of which the middle 6 are empty. The second test fails. You get an empty array. I expected this to return an array of 8 empty strings. Java's mutant cousin, Javascript get's this right, as does Python.

Rhino 1.6 release 5 2006 11 18
js> a = "\t\t\t\t\t\t\t";
js> fields = a.split("\t")
,,,,,,,
js> fields.length
8

Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17)
>>> a = "\t\t\t\t\t\t\t"
>>> fields = a.split("\t")
['', '', '', '', '', '', '', '']
>>> len(fields)
8

Oddly enough, Ruby agrees with Java as does Perl.

>> str = "\t\t\t\t\t\t\t"
=> "\t\t\t\t\t\t\t"
>> fields = str.split("\t")
=> []
>> fields.length
=> 0

My perl is way rusty, so sue me. but, I think this is more or less it:

$str = "\t\t\t\t\t\t\t";
@fields = split(/\t/, $str);
print("fields = [@fields]\n");
$len = @fields;
print("length = $len\n");

Which yields:

fields = []
length = 0

How totally annoying!

Browsing genomes

I may as well come clean and admit that I'm developing a genome browser. What? Another genome browser? Why? You may well ask these questions. Well, it's a long story. But here is a completely non-exhaustive list of existing genome browsers.

• The classics: UCSC Genome Browser Home and paper and it's microbial sibling.
• Argo a Java rich-client genome browser built at the Broad Institute.
• Integrative Genomics Viewer also from Broad (see press release).
• Affymetrix spun out it's Integrated Genome Browser into an open source project, along with a library of re-usable components called GenoViz.
• GBrowse Lincoln Stein's Perl based genome browser, followed by the AJAXy javascripty JBrowse (with a paper, nicely written as usual from Lincoln Stein.).
• x:map an AJAX genome browser based on the Google Maps API.
• The biology of extemophiles lab at the University of Paris Sud have a nice little web based browser for Sulfolobus.
• NCBI has a new AJAX tool called Sequence Viewer
• Flash based OmicBrowse is apparently big in Japan.
• MochiView is really nice. Read the paper in BMC Biology.
• GenomeGraphs: integrated genomic data visualization with R
• GenomeView
• Visualization guru Ben Fry (of processing fame) wrote at least two:cd36 browser and a handheld genome browser
• A guy who calls himself Saaien Tist implemented a circular genome browser in ruby-processing
• Back in 2002, some canucks built BioViz an SVG based genome browser
• The Savant Genome Browser is a desktop visualization tool for genomic data. It was primarily developed for visualizing high throughput (aka next generation) sequencing data... Savant comes out of the Computational Biology Lab at the University of Toronto - also home of Cytoscape Web.

Note: updated in Sept. 2009 to reflect the fact that everyone and their uncle built a genome browser this past couple of years. See Brother, can you spare a genome browser?

Note: updated again in May of 2010 and again in Feb 2011 to add Savant.

Better color chooser for Java Swing

I came across a color chooser that kicks the Swing JColorChooser's booty. It looks great and supports transparency. It's available at colorchooser.dev.java.net and on the developer's blog. It will soon be used in my genome browser. Thanks, Jeremy!

Java in Firefox extension hosed again

A full-featured browser, an XUL front end, and the wealth of libraries available in Java makes for a powerful and flexible combination. The browser extension capability of Firefox, along with LiveConnect has been used by at least three extensions:

• Simile's Piggy Bank, a semantic web browser
• XQuseme, embeds a XQuery engine into the browser
• Firegoose, exchanges data between bioinformatics web sites

Most of what I've figured out about using Java from an extension came from Simile's David Huynh. Sadly, development of Piggy Bank has now "quiesced".

I don't know about the others, but Firegoose is hosed by the latest Java 6 update 10. Apparently, Java 6.10 introduces some significant changes into LiveConnect and the Java browser plugin. It's certainly good that Java in the browser is getting some attention, but I wish Java in a Firefox extension was a supported and regression tested use case (see whining here). The fact that it's such an arcane, unsupported and brittle hack is holding back what could otherwise be a nice technique.

Interest in Java in Firefox extensions appears to exist according to these posts in the MozillaZine Extension Development forum:

• getting java.security.PrivilegedActionException from JAR
• Java Based Extension Development
• Java
• Using java in an Extension, how ?
• Calling JAVA method. Chrome URL

First Problem: The error I get appears to happen when reflectively instantiating a Java array and looks like this:

Error calling method on NPObject!
[plugin exception: java.lang.IllegalArgumentException: No method
found matching name newInstance and arguments
[com.sun.java.browser.plugin2.liveconnect.v1.JavaNameSpace,
java.lang.Integer]].

Instantiating the array through reflection was, itself, a work-around for another LiveConnect issue with type conversion between Javascript arrays and Java arrays. It's barfing on line 03 below:

// from http://simile.mit.edu/repository/java-firefox-extension/firefox/chrome/content/scripts/browser-overlay.js
_toJavaUrlArray: function(a) {
 var urlArray = java.lang.reflect.Array.newInstance(java.net.URL, a.length);
 for (var i = 0; i < a.length; i++) {
  var url = a[i];
  java.lang.reflect.Array.set(
   urlArray,
   i,
   (typeof url == "string") ? new java.net.URL(url) : url
  );
 }
 return urlArray;
}

Update 1: First problem solved easily enough.

var dummyUrl = new java.net.URL("http://gaggle.systemsbiology.org");
var urlArray = java.lang.reflect.Array.newInstance(dummyUrl.getClass(), a.length);

Now, on to more hideousness:

Error calling method on NPObject!
[plugin exception: java.security.AccessControlException: access denied
(java.lang.RuntimePermission createClassLoader)].

...caused by trying to instantiate a URLClassLoader. The next-generation Java Plug-in, including in update 10, makes changes to the security policy such that calls from Javascript to Java are uniformly treated as untrusted.

Update 2: A Work-around!

A post on the java.net forum has a work-around. You can disable the "next-generation plug-in" through the Java control panel. Under the Advanced tab, open Java Plug-in, deselect Enable the next-generation Java Plug-in, then, restart Firefox. There is a bug filed whose comments seem to suggest that it will be addressed in a future release of the Java Plug-in.

Update 3: According to this thread on java.net, a fix is on the way in Java SE 6 update 12. Thanks, Sun!

More references:

• Java Firefox Extension by ancaluca
• mozilla.dev.tech.java
• java security policy reference
• Mozilla on Java in a Firefox Extension
• Simile mailing lists

SQLiteJDBC from Xerial

As noted earlier, I haven't been able to get native mode JDBC drivers for SQLite to work on Java 6 on OS X. Xerial has made what appears to be a fork of the Zentus SQLiteJDBC driver. They even have a discussion group, which apparently Zentus used to have but doesn't anymore.

Maybe ch-werner's javasqlite is another possibility.

Clean code, properties, and patterns

Robert C. Martin has a new book out called Clean Code that looks worthwhile, as if I'm not far enough behind in reading. I also came across his Principles and Patterns article. Completely unrelated to that, The Universal Design Pattern by Steve Yegge describes object system of Javascript as an instance of what he calls the Properties Pattern. Java's static types and even classes in Python and Ruby feel constrained after working with Javascript's bag-of-properties approach to objects. There's also some interesting stuff at a blog called Red and Sensual Java, if you can get past the name.

Notes on SqliteJDBC and Java Web Start

Java has a long standing limitation about loading native code from a jar, stemming from the unfortunate belief that native code was somehow impure. In the practical world, there was occasional need to do this, so people came up with the work-around of copying the native library out of the jar and into the local filesystem where it could then be loaded with System.load().

Another way do do this, within web start is to platform specific resource elements like these to your jnlp:

  
<resources os="Windows" arch="x86">
  <j2se href="http://java.sun.com/products/autodl/j2se" version="1.5+"/>
  <nativelib href="sqlitejdbc-v052-native-win.jar"/>
</resources>
<resources os="Mac OS">
  <j2se href="http://java.sun.com/products/autodl/j2se" version="1.5+"/>
  <nativelib href="sqlitejdbc-v052-native-mac.jar"/>
</resources>
<resources os="Linux">
  <j2se href="http://java.sun.com/products/autodl/j2se" version="1.5+"/>
  <nativelib href="sqlitejdbc-v052-native-lin.jar"/>
</resources>

This way, we get the native library for the right platform, if all goes according to plan. One wacky detail is that System.loadLibrary("foo") munges the library name in some platform specific and poorly documented way. So, you need a libfoo.jnilib for OS X, libfoo.so for linux, and a foo.dll for windoze. It's kind-of a hassle to set up all these separate jars. And System.load(...) just tries to do what it's told without munging the name, so people still frequently use the trick described above.

SqliteJDBC does just that to load the native Sqlite library. SqliteJDBC also includes "pure Java" drivers, which are considerably slower for some operations, but act as a nice fallback when the proper native libraries aren't handy.

This all works well enough. But, on OS X 10.5, Java 6, it ends up falling back on the non-native drivers. After digging a bit, I found that the call to System.load(..) was failing with this exception:

/private/tmp/libsqlitejdbc-19214.lib: 
java.lang.UnsatisfiedLinkError: /private/tmp/libsqlitejdbc-19214.lib: 
 at java.lang.ClassLoader$NativeLibrary.load(Native Method)
 at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1822)
 at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1702)
 at java.lang.Runtime.load0(Runtime.java:770)
 at java.lang.System.load(System.java:1005)
        ...

The same thing works find when using Java 5 on the same machine and on Java 6 on Windoze, so I'm guessing the native library needs to be recompiled to work correctly with OS X's 64-bit Java 6.

A little spelunking into SqliteJDBC's code reveals that the author has kindly provided hooks where you can insert the path and name of your own driver using these system properties:

• org.sqlite.lib.path
• org.sqlite.lib.name

The driver will then attempt this:

System.load(new File(libpath, libname).getAbsolutePath());

More Resources:

• Unofficial JNLP FAQ
• How to package a native library in a jar

Java.next

Stuart Halloway says in his Java.next series on the Relevance Blog:

Many of our best ideas on how to use a Java Virtual Machine (JVM) are now being baked into more advanced languages for the JVM. These languages tend to provide two significant advantages: essence over ceremony and functional programming.

Eclipse templates and import statements

I was encouraged by this guy to make better use of Eclipse's code templates. I've used templates for a while for boilerplate code like adding a log4j logger to a class, like this:

class Foo {
   private static final Logger log = Logger.getLogger(Foo.class);
    ...
}

Using templates, I just have to type 'l4j' and cntl-space and there I go. One thing that had been plaguing me is that I still had to cnt-space again to add the imports, and thanks to Sun's needless duplication of effort, I had to select the log4j Logger over the lame java.util.Logger.

Little did I know about the ${import} and ${importStatic} variables. The following template gives me the required imports for free.

${imp:import(org.apache.log4j.Logger)}
private static final Logger log = Logger.getLogger(${enclosing_type}.class);

And how about this for JUnit 4 imports:

${imp:import(org.junit.Test)}
${impst:importStatic('org.junit.Assert.*')}

With enough Eclipse templates, Java might not drive me completely batshit after all. I love it when I file a bug and the feature already exists. Thanks, Eclipse.

Generics hurt my head some more

As previously mentioned, generics hurt my head. Sun's Java generics tutorial calls Collections.max() one of the subtlest examples of the use of generics.

public static <T extends Object & Comparable<? super T>> T max(Collection<? extends T> coll)

Subtle is one word for it. Anyway, to my walnut-sized brain, generics are confusing; maybe too confusing to be worth it. Was ClassCastException really a major problem for anybody?

Generics can be used in two places. Generic classes, like List<T>, and generic methods, like Collections.max().

As I learned in Programming Languages 505, return types are covariant while method parameters are contravariant (see the wikipedia entry). Generics, because they can be used to parameterize both return types and method parameters, are invariant. Keeping this in mind explains a lot. Thus, List<String> is not a subtype of List<Object>. This leads to a lot of confusion, particularly because Arrays in Java took the opposite choice. String[] is a subtype of Object[]. Aaarg!

Wildcards attempt to mitigate this problem, with extends and super.

List<? extends Flapdoodle>
TreeSet(Comparator<? super E> c)

I pronounce List<?> as "list of WTF". You're really supposed to say it's an unbounded wildcard type. Seems about as annoying as @SuppressWarnings("unchecked").

Effective Java 2nd Ed., Generics

For more confusing information, see the Generics chapter out of Effective Java, 2nd Ed. (Thanks to Joe Bowbeer for the pointer!):

• Produce-extends, consumer-super

  In other words, if a parameterized type represents a T producer, use <? extends T>; if it represents a T consumer, use <? super T>.

• Private helper method for wildcard capture
• Typesafe heterogeneous containers

Source Control and Continuous integration

Version Control:

• svn subversion
• hg mercurial
• bzr bazaar
• git apparently, git is better than x

Continuous integration:

• Hudson
• CruiseControl

All the cool kids are on github. ...and I still thought SVN was new and hip. Guess I'm behind the times.

Generics solution

Previously, generics were hurting my head. The generics tutorial has this to say about the problem that is at issue here:

Wildcards are designed to support flexible subtyping [...]. Generic methods allow type parameters to be used to express dependencies among the types of one or more arguments to a method and/or its return type.

The solution here (thanks to Eric Jain) uses a generic method to express the dependency between the type Features in the Container passed into renderMap's getRendererFor(...) function and the type of Renderer returned. The renderer must be able to render that type of feature.

package generics.solution;


// We have a hierarchy of specialized subtypes of feature.
class Feature {
}

class SpecialFeature extends Feature {
}

// Features of a given kind are accessed through containers
interface Container<F extends Feature> {
 public Iterable<F> features();
}

// We want to draw features in various specialized ways.
interface Renderer<F extends Feature> {
 public void render(Iterable<? extends F> features);
}

class SpecialRenderer implements Renderer<SpecialFeature> {
 public void render(Iterable<? extends SpecialFeature> features) {
  // render special features
 }
}

// Which renderer we use for a given container of features gets
// configured at runtime. This holds the mapping between renderers
// and containers.
interface RendererMap {
 public <F extends Feature> Renderer<? super F> getRendererFor(Container<F> container);
}

class RenderingScheduler {
 RendererMap rendererMap;

 public void render(Iterable<Container<? extends Feature>> containers) {
  for (Container<? extends Feature> container : containers) {
   render(container);
  }
 }

 public <F extends Feature> void render(Container<F> container) {
  Renderer<? super F> renderer = rendererMap.getRendererFor(container);
  renderer.render(container.features());
 }
}

Update: Generics hurt my head some more.

Generics hurt my head

Here's a little generics problem, in the form of some code:

package generics.question;


// We have a hierarchy of specialized subtypes of feature.
class Feature {
}

class SpecialFeature extends Feature {
}

// Features of a given kind are accessed through containers
interface Container<T extends Feature> {
 public Iterable<T> foos();
}

class SpecialContainer implements Container<SpecialFeature> {
 public Iterable<SpecialFeature> foos() {
  return null;
 }
}

// We want to draw features in various specialized ways.
interface Renderer<V extends Feature> {
 public void render(Iterable<V> foos);
}

class SpecialRenderer implements Renderer<SpecialFeature> {
 public void render(Iterable<SpecialFeature> foos) {
  // render Subfoo objects
 }
}

// Which renderer we use for a given container of features gets
// configured at runtime. This holds the mapping between renderers
// and containers.
interface RendererMap {
 Renderer<? extends Feature> getRendererFor(Container<? extends Feature> container);
}

class RenderingScheduler {
 RendererMap rendererMap;

 public void render(Iterable<Container<? extends Feature>> containers) {
  for (Container<? extends Feature> container : containers) {
   Renderer<? extends Feature> renderer = rendererMap.getRendererFor(container);

   // The following line produces a compile error:
   //   The method render(Iterable<capture#3-of ? extends Feature>) in
   //   the type Renderer<capture#3-of ? extends Feature> is not
   //   applicable for the arguments (Iterable<capture#4-of ? extends
   //   Feature>)
   renderer.render(container.foos());

   // this works, but... ug.
   // How are these type parameters helping me again?
   Renderer<Feature> r = (Renderer<Feature>)renderer;
   Iterable<Feature> foos = (Iterable<Feature>)container.foos();
   r.render(foos);
  }
 }
}

Update: Solution! ...and generics hurt my head some more.

Back to RMI Basics

RMI forms the basis for Java's own remoting capabilities powering the platform's domination on servers everywhere. This article attempts to see through all the frameworks, programming models and technology built on top of it, and tries to get to the lower level basics of RMI. RMI makes remoting transparent by making remote objects look like local objects.