JPA Enhancement done right

JPA Enhancement done right

When it comes to handling databases in a highlevel language there are basically 2 different approaches:

a.) Systems which give you access to all the gory details and therefor have a pretty high entry barrier.
b.) Systems which are perfectly easy to use for samples and simple apps – but you need all the (well hidden) gory details if you use that stuff in real world projects.

Without a doubt JPA is more the category b.) type of project.

Using JPA

If you look at a classical JPA entity, you will find something similar to our following example:

import javax.persistence.*;

@Entity
public class Customer {
  @Id
  @GeneratedValue
  private Long id;

  @Column(length = 50)
  private String firstName;

  public Long getId() {
    return id;
  }

  public String getFirstName() {
    return firstName;
  }

  public void setFirstName(String firstName) {
    this.firstName = firstName; 
  }
}

Usually the entities get created with the ‘new’ keyword and then stored by calling EntityManager#persist()

Customer c = new Customer();
c.setFirstName("John");
entityManager.persist(c);

Looks easy, isn’t?

Enhancement

A lot of the ‘easyness’ of JPA actually comes by the feature of ‘enhancing’ classes which are annotated with @Entity. And this enhancement step is usually well hidden deep inside the code.

So let’s first look what this ‘enhancement’ does for you!

Whenever you touch a JPA entity, it will track the following information for each and every non-transient attribute of the entity:

* _dirty: if you change firstName from ‘John’ to ‘Barry’, this field becomes ‘dirty’. JPA uses this information to decide which fields/entities needs to get written to the database.

* _loaded: JPA entities might contain fields which have FetchType.LAZY. They will only get loaded from the database once they are accessed the first time. This is pretty neat if you have a fat @ElementCollection which takes some time to load. Some older JPA impls did rely on if(field == null) but what about c.setFirstName(null); to clear out this info in the database? How to distinguish between a field which didn’t yet get laoded (and is null) from a field which got loaded and afterwards set to null?
_loaded will be used to track this info.

But where is this _dirty and _loaded info in my entities? Well, this is exactly what the ‘enhancement’ step is doing. It adds this information to the entity class file for you!

Basically the setFirstName in the ‘enhanced’ class will finally look somehow like the following (completely transparent for the user!):

public void setFirstName(String firstName) {
  if (safeEquals(this.firstName, firstName) {
    return;
  }

  //EntityManager must not get accessed in different threads
  assertEntityManagerLocking();

  setDirty(FIRSTNAME_FIELD);
  this.firstName = firstName;
}

Technically there are 3 different ways to achieve this

1.) Using Proxies/Subclassing at runtime

A Proxy is basically a dynamically created Subclass. This is the default mode for Hibernate and has quite a few drawbacks.

1.a.) you can only proxy methods. It’s not possible to intercept the field access inside the class level. This makes it much harder to correctly write EntityListeners and other methods which juggle around with fields.

1.b.) Only property access for classes loaded via a query or EntityManager#find() can get tracked. It’s not possible to do the same for Entities just created via ‘new’ because there is no proxy in this case.

1.c.) Lazy loading is not possible in this case. All fields (including expensive @ElementCollection reads) will be performed eagerly.

1.d.) Hibernate seems to have a mode where it dynamically replaces Lists stored in an entity with their own ProxyLists for managed entities. That way they can at least do some LazyLoading. But doing the dirty-check requires to compare each entry in those Lists with the database – that’s damn slow!

2.) Using Compilation Enhancement

You can use either ant or maven to ‘enhance’ the class files of your entities to contain all the additional code which is needed to work properly. This is the prefered way as it provides the full set of features and also the best performance at runtime. Actually this option comes in 2 different flavours:

2.a.) Build Time Enhancement: The class file get’s modified even before it will get packaged into the JAR. I will come back to this again a bit later and explain it in detail.

2.b.) Deploy Time Enhancement: The class file get’s enhanced by the ‘deployer’ in a JavaEE container. If you upload your WAR/EAR to your server, it will get unpackaged, all classes with an @Entity will automatically get enhanced immediately afterwards by the container.

3.) Runtime Enhancement (via ClassTransformer)

At the first glance the JavaAgent [1] looked like a smart way to perform the enhancement at runtime, but it turned out that there are fundamental problems with this approach. Again there are 2 different flavours for this way of enhancement:
3.a.) using the Java5 -javaagent option.
3.b.) using the Java6 Instrumentation#retransformClasses. This allows to drop out loaded classes and replace them with their ‘transformed’ version.

The major problem with both of the aforementioned mechanism is that they use the SystemClassLoader to register the ClassTransformer and subsequently also to load the classes needed to perform the transformation!

This has the bad side effect that the SystemClassloader gets polluted with some impl classes of your persistence provider. But not all, and if the transformation is finished, the jar containing the Transformer will get detached from the JVM again. This will leave you with 30% of Hibernate/OpenJPA/EclipseLink lying around in your SystemClassPath and the rest is not available.

Not that worse, one might say, because all those classes are still available in your web application. Too bad that a nice little security mechanism kicks in and destroys all our dreams. The Servlet specification defines that any container must prevent redefining ‘system classes’ – that is all classes which get loaded via the SystemClassLoader. Thus in any spec conform servlet container (e.g. tomcat [2]) you are now bound to the few classes of your jpa provider which are available in your SystemClassLoader. This will result in tons of NoClassDefFound errors and many tears…

The Winner: Build Time Enhancement

Technically only Built Time Enhancement and Deploy Time Enhancement do work sufficiently at all. The reason I don’t use Deploy Time Enhancement is that you are not able to test and debug locally what you have running in your server. All the unit tests are not worth much if you don’t run the code which gets executed in production. Thus Build Time Enhancement is the clear winner to me.

The openjpa-maven-plugin

Quite some time ago we wrote the openjpa-maven-plugin [3] which allows to do all the enhancement steps while building your project with maven. This plugin got moved over to the OpenJPA project itself and will first be available with the upcoming OpenJPA-2.2.0 version (expect this to be out this month). A similar tool is available as ant-task. There is also a maven plugin for doing the same with hibernate [4].

IDE pitfalls

Most IDEs nowadays compile the classes on the fly. If you don’t have a special IDE plugin installed which ‘fixes’ the entity classes afterwards, you might get errors reporting that you ‘try to run unenhanced classes’. Usually this happens only the first time or if you change an Entity class.

In this case simply enhance them again on the commandline.
$> mvn clean process-classes

Afterwards you can work in your IDE without any further problems.

Glossar:

[1] javaagent: http://docs.oracle.com/javase/6/docs/api/java/lang/instrument/package-summary.html

[2] tomcat WebAppClassLoader: http://svn.apache.org/repos/asf/tomcat/trunk/java/org/apache/catalina/loader/WebappClassLoader.java
see public synchronized Class loadClass(String name, boolean resolve)

// (0.2) Try loading the class with the system class loader, to prevent
// the webapp from overriding J2SE classes

[3] http://mojo.codehaus.org/openjpa-maven-plugin/usage.html

[4] http://mojo.codehaus.org/maven-hibernate3/hibernate3-maven-plugin/

OpenJPA: http://openjpa.apache.org/builds/latest/docs/manual/manual.html#ref_guide_pc_enhance

Hibernate: http://docs.jboss.org/hibernate/core/3.3/reference/en/htm/performance.html#performance-fetching-lazyproperties

solving the browser Tab problem

Web Browser tabs and windows have been an issue for a long time now. They cause broken ViewMaps (used in @ViewScoped), FlashScope and lots other issues [1].

There is also a very ugly negative effect with using multiple browser tabs when using server side state saving in JSF: the last viewStates of a client is only stored per session and not per browser tab. Thus if a person clicks long enough in 1 browser tab (the default value is 20 for both Mojarra and MyFaces) then all your other open tabs will loose the view state. And this will cause a ViewExpiredException once you click around in one of those other browser tabs. This can only be solved if JSF natively supports multiple browser tabs and stores the view states per browser tab.

The problem is that providing a really working windowId handling is not exactly easy (as outlined in [1]).

Apache MyFaces CODI ClientSideWindowHandler

A solution which works perfectly fine on a technical level has been implemented by Jakob, Gerhard and me in Apache MyFaces CODI [2]. But it had the bad user effect that rendering the intermediate page (which checks for a valid site) and only then requests the real page did cause some ‘flickering’. It seems we have solved this problem now!

While I explained Dan Allen how we do the windowId handling in CODI currently, he came up with the idea of ‘taking a screenshot somehow and show it on the intermediate page’. One idea hit the other and today I tried to use html5 localStorage to store the head and body elements and dynamically apply them on the indermediate windowhandler.html page. Voila, works perfect so far. Of course, we need to test this with complex pages and pages which heavily uses javascript, but so far it works fine.

How it works

The site which contains a GET link needs to use some JavaScript which changes all <a href../> and add/delegate a
onclick="storeWindowTree(); return true;"

The storeWindowTree is pretty easy:

<script type="text/javascript">
// store the current body in the html5 localstorage
function storeWindowTree() {
localStorage.setItem(window.name, document.body.innerHTML);
}
</script>


So after we click on a link, we have the body of the current page in the html5 localStorage. (We should store the head too in the future).

On the windowhandler.html intermediate page we just do the following:


<script type="text/javascript" >
function getOldBody() {
var oldBody = null;
if (window.name.length != null) {
oldBody = localStorage.getItem(window.name);
}
return oldBody;
}

window.onload = function() {
var oldBody = getOldBody();

if (oldBody != null) {
document.body.innerHTML = oldBody;
localStorage.removeItem(window.name);
}…
do all the windowId stuff (works fine anyway)
window.location.reload();
}

easy, isn’t?

[1] http://myfaces.apache.org/orchestra/myfaces-orchestra-core/multiwindow.html
[2] https://cwiki.apache.org/confluence/display/EXTCDI/JSF+WindowHandler
// ]]>

Flexible Configuration for Modular Systems

The Problem

Properties are cool, but you can only pickup the configuration of one property file at a time. If you have a base configuration in your framework jar and need to allow further tweaking of that configuration in a higher level, then a well defined chain of responsibilities often helps.
Usually this is done by

• having the base configuration in yourframework.jar
• overriding the configuration if the same config.properties file exists in WEB-INF/classes

But this solution is pretty restrictive and wont help you if you have a plugin architecture utilising the java.util.ServiceLoader mechanism. Speaking of OpenWebBeans, we have most of our functionality provided as SPI (Service Provider Interface) and the implementation which should get used is being picked via a config.properties. The problem we face is that we have lots of droppable plugins which need a different configuration. If you use the openwebbeans-resource.jar alone, then you’ll get a basic support for JPA via a SPI-implementation. If you additionally use openwebbeans-openejb, then we need to use another SPI implementation to get EntityManagers from OpenEJB instead.
But the config.properties are all contained in JARs which have the same layer (all in WEB-INF/lib, your EAR lib or in the EE-Server classpath) – which means the algorithm mentioned above isn’t enough anymore. This is even more true in OSGi environments where you don’t have any control over the classloader hierarchy anymore.

The Solution

Instead of maintaining the importance of an application implicitly via it’s location, we just explicitely give each config file a configuration.ordinal property. Higher values mean higher importance. And here is how it works:

• First we scan for all config.properties files with ClassLoader.getResources()
• Scan all found properties files for it’s configuration.ordinal value. If no such property is found, we assume a default value of 100
• Sort all found config files by their configuration.ordinal values in ascending order
• Start overlaying all properties by applying them in ascending configuration.ordinal order. All properties defined with a higher ordinal will just override the values from any less important config file.

The Usage

Just type your properties files as usual.

myconfig.properties has a configuration.ordinal of 15

configuration.ordinal=15
my.value = hello user
my.othervalue = not overwritten


Another jar contains a myconfig.properties with a configuration.ordinal of 20

configuration.ordinal=20
my.value = and now something completely different


The resulting property values are:

my.value = and now something completely different
my.othervalue = not overwritten


The Source

The whole source can be found in our Apache OpenWebBeans SVN repository:
PropertyLoader.java

HowTo ChangeLog with JPA

The problem
In almost all projects which have some kind of administrative functionality, there is a need to keep track of changes done to the data.

If we take a car rental store as example, it is good to know who reserved a car and who changed a reservation.

Our entity looks like the following (only the important parts of classes are shown here)

@Entity
public class CarReservation {
@Id, @Version ..
private Car car;
private @Temporal.. Date reservedFrom;
private @Temporal.. Date reservedUntil;
private Customer reservedFor;
.. getters and setters
}


Now let’s consider that the customer calls our shop and likes to move his reservation to a slightly different date. The clerk officer opens the reservation and changes it accordingly. Too bad that he edited the wrong reservation and saved it. Even if he recognises his error quickly, he has no chance to revert the wrong reservation back to the correct values, because they didn’t got stored in the system.

The solution
JPA provides a nice way to receive lifecycle callbacks JPA2 JavaDoc.
We need to somehow get the old values at the time we store the changes in our entity so we can create a ‘diff’. Since there always exists exactly 1 instance of a DB entry per EntityManager, we don’t have a chance to get those old values at save time. There exists a dirty trick with opening a new EntityManager inside the save operation, but that doesn’t perform well. The way I get my hands on the old values is to store them in a Map at load time by utilising @PostLoad via a EntityListener.

The single parts involved in the magic:

• The TrackChangesListener which listens to @PostLoad and @PreUpdate JPA events
• A new @TrackChanges annotation which marks entity fields as being subject to be tracked
• A new @ChangeLog annotation which mark the one String @Lob field to stores the changeLog
• An Editor component which ThreadLocally stores the oldValues and knows which entities are currently being edited (for performance reasons). It also knows the current user so we can also store who edited the entry.

How our Car entity looks after we applied the changelog functionality:

@Entity
@EntityListeners({TrackChangesListener.class})
public class CarReservation {
  @Id, @Version ..

  private @TrackChanges Car car;
  private @TrackChanges @Temporal.. Date reservedFrom;
  private @TrackChanges @Temporal.. Date reservedUntil;
  private @TrackChanges Customer reservedFor;
  private @ChangeLog @Lob String changeLog;
  .. getters and setters
}


And here comes the code:

First the annotations:

@Target(ElementType.FIELD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface TrackChanges {
}

and

@Target(ElementType.FIELD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface ChangeLog {
}


And here comes the most important part, the EntityListener:

public class TrackChangesListener
{
    @PostLoad
    public void loadOldData(Object he) {
        if (!Editor.isEdited(he)) {
            return;
        }

        Map oldData = new HashMap();

        collectOldValues(oldData, "", he);

        Editor.setOldData(he, oldData);
    }

    private void collectOldValues(Map oldData, String prefix, Object he)
    {
        List historisedFields = getHistorisedFields(he);

        for (Field field : historisedFields) {
            field.setAccessible(true);
            try
            {
                if (field.getAnnotation(Embedded.class) != null) {
                    Object value = field.get(he);
                    if (value != null) {
                        // recurse into the Embedded field
                        collectOldValues(oldData, prefix + field.getName() + ".", value);
                    }
                }
                else {
                    oldData.put(prefix + field.getName(), field.get(he));
                }
            }
            catch (IllegalAccessException e)
            {
                throw new RuntimeException(e);
            }
        }
    }

    @PreUpdate
    public void updateChangeLog(Object he) {
        if (!Editor.isEdited(he)) {
            return;
        }

        try
        {
            Field changeLogField = getChangeLogField(he);
            Validate.notNull(changeLogField, "could not find @ChangeLog field in Class " + he.getClass().getName());

            StringBuilder changeLog = new StringBuilder();
    
            String changeLogValue = (String) changeLogField.get(he);

            if (changeLogValue != null) {
                changeLog.append(changeLogValue).append("\n\n");
            }

            changeLog.append("changedBy " ).append(Editor.getUser());
            changeLog.append(" on ").append((new Date()).toGMTString()).append('\n');

            boolean changed = false;

            changed = collectChanges(changeLog, Editor.getOldData(he), "", he);

            // update the changeLog field if a trackable change was detected
            if (changed) {
                changeLogField.set(he, changeLog.toString());
            }
        }
        catch (IllegalAccessException e)
        {
            throw new RuntimeException(e);
        }
    }

    private Field getChangeLogField(Object he)
    {
        Field[] fields = he.getClass().getDeclaredFields();

        for (Field field : fields) {
            if (field.getAnnotation(ChangeLog.class) != null) {
                field.setAccessible(true);
                return field;
            }
        }
        return null;
    }

    private boolean collectChanges(StringBuilder changeLog, Map oldData, String prefix, Object he)
    {
        boolean changed = false;
        List historisedFields = getHistorisedFields(he);
        for (Field field : historisedFields) {
            field.setAccessible(true);
            String fieldName = field.getName();
            try
            {
                if (field.getAnnotation(Embedded.class) != null) {
                    Object value = field.get(he);
                    if (value != null) {
                        // recurse into the Embedded field
                        changed |= collectChanges(changeLog, oldData, prefix + field.getName() + ".", value);
                    }
                }
                else {
                    Object newValue = field.get(he);
                    Object oldValue = oldData.get(prefix + fieldName);
                    changed |= addChangeLine(changeLog, prefix + fieldName, oldValue, newValue);
                }
            }
            catch (IllegalAccessException e)
            {
                throw new RuntimeException(e);
            }
        }
        return changed;
    }

    private boolean addChangeLine(StringBuilder changeLog, String fieldName, Object oldValue, Object newValue)
    {
        if (oldValue == null && newValue == null) {
            return false;
        }
        if (oldValue != null && oldValue.equals(newValue)) {
            // no change
            return false;
        }

        String changeLine = fieldName + " old: [" + (oldValue != null ? oldValue : "") + "] new: [" + (newValue != null ? newValue : "") + "]\n";
        changeLog.append(changeLine);
        return true;
    }


    private List getHistorisedFields(Object he)
    {
        Field[] fields = he.getClass().getDeclaredFields();
        List histFields = new ArrayList();

        for (Field field : fields) {
            if (field.getAnnotation(TrackChanges.class) != null) {
                histFields.add(field);
            }
        }

        return histFields;
    }

}