Jonathan Hedley

On Adding HTTP/2 Support to jsoup

2025-03-18T00:43:49Z

Mar 8, 2025

I recently released jsoup 1.19.1, which includes support for making HTTP/2 requests through jsoup’s Connection client. HTTP/2 support had been on my mind for a while, and although implementing it proved relatively straightforward, it involved some interesting design and integration work. The resulting pattern will be reusable for adding other future features, so I thought I’d share some notes on the process.

As background, jsoup handles HTTP requests using Java’s built-in HttpUrlConnection, which has existed since time immemorial. When Java introduced HTTP/2 in Java 11, rather than extending the existing HttpUrlConnection API, it created a completely new HttpClient interface. This meant existing code couldn’t benefit from HTTP/2 simply by upgrading the JDK. But, the new API did clean up some old warts, and bring a path for future version upgrades.

Android NIO desugaring is a bytecode transformation process that updates syntax and adds to the classpath.

Unfortunately, Android chose not to implement Java’s new HttpClient interface, despite otherwise robust support for Java 8 and Java 11 language features through their “desugaring” implementation.

A key goal in adding HTTP/2 support was to make it a seamless, drop-in upgrade for jsoup users. At the same time, I wanted to maintain backward compatibility, keep support for Java 8 (the minimum required by jsoup), and continue full Android compatibility.

So, how to square this circle?

Multi-release JAR files are a feature of the JAR file format that allow for a single JAR file to support multiple major versions of Java platform releases.

Java offers multi-release JARs (“MRJARs”), which allow code partitioning for different Java versions. The JVM runtime just ignores classes targeted for newer Java versions than the one it’s running on.

To leverage this, I refactored jsoup’s HttpConnection implementation by abstracting out the HttpUrlConnection logic into a delegate RequestExecutor abstract class. Then, I provided two implementations: one for HttpUrlConnection, and another for HttpClient placed exclusively in the Java 11 version directory. This structure ensures that earlier Java versions continue to compile and run jsoup without issues.

To maintain Android compatibility, I created a RequestDispatch provider that attempts to instantiate the HttpClient implementation and falls back to HttpUrlConnection if necessary.

You can see it come together in the Pull Request: Support HTTP/2 requests via HttpClient.

java/
java11/
- HttpClientExecutor

One significant advantage of HTTP/2 for clients is faster repeated requests to the same host. In Java’s HttpClient, this is implemented by maintaining a connection pool in a HttpClient instance. I integrated this capability into jsoup’s existing Connection.newSession() method, as seen in this commit.

Testing the Implementation

jsoup already has comprehensive unit and integration tests for its connection code, and I wanted to ensure that the new HttpClient implementation had at least the same coverage. This proved very simple to implement; just adding test classes (in the 11 version) that extend the existing tests classes (which then execute all the existing tests in a new context), and use a setup method to toggle-on the HttpClient. And other than a few implementation nuances, all the existing paths pass correctly.

You can see the testing approach here:

I found using jenv invaluable to flip between different Java environments and verify everything worked OK.

Given the significance of this change, the HTTP/2 feature in jsoup 1.19.1 is currently gated behind an opt-in flag (jsoup.useHttpClient). Users can enable it to test and uncover potential integration issues. I’ve been running it in production (e.g., on Try jsoup) and have found it stable and performant. Once it’s had more time to bake, I’ll enable it by default in future releases.

See the Wikipedia article for more details on HTTP/3.

A good backgrounder on the improvements between 1.1, 2, and 3 is: HTTP/3 is everywhere but nowhere.

There’s also a Java Enhancement Proposal (JEP) underway to introduce HTTP/3 support to Java’s HttpClient. This will enable jsoup to easily upgrade to HTTP/3 once it’s released.

Try it out!

To enable HTTP/2 requests, make sure you upgrade to jsoup 1.19.1, and add this line to your request code:

static {
    System.setProperty("jsoup.useHttpClient", "true");
}

This will enable HTTP/2 support for all requests made through jsoup.

If you find any issues, please report them on the jsoup issue tracker.

jsoup SAX + DOM StreamParser

2025-03-02T04:11:30Z

Mar 2, 2025

I’ve added a cookbook example for jsoup’s StreamParser, that demonstrates how to use this hybrid Java SAX + DOM parser. Based on examples from the original PR.

Problem

You need to parse an HTML or XML document that is too large to fit entirely into memory, or you want to process elements progressively as they are encountered. A typical use case is extracting specific elements from a large document, or handling streamed HTML from a network source efficiently.

Traditional Java SAX parsers offer efficient streaming parsing for XML and HTML, but they lack an ergonomic way to traverse or manipulate elements like a DOM parser. Meanwhile, standard DOM parsers, such as Jsoup.parse(), require loading the entire document into memory, which may be inefficient for large files.

Solution

Use the StreamParser, which allows you to parse parsing an HTML or XML document in an event driven hybrid DOM + SAX style. Elements are emitted as they are completed, enabling efficient memory use and incremental processing. This hybrid approach allows you to process elements as they arrive, including their children and ancestors, while still leveraging jsoup’s intuitive API.

This makes StreamParser a viable alternative to traditional SAX parsers while providing a more ergonomic and familiar API. And jsoup’s robust handling of malformed HTML and XML ensures that real-world documents can be processed effectively.

Link

Convert Microsoft Word to Plain Text

2025-03-02T04:09:49Z

May 4, 2004

This is a repost of an entry from 2004(!). This Word-cleaning functionality is showing up in more and more web editors, and most sites actually support UTF-8, but people might still find this useful.

Most of the time when I’m writing content for the web (for this blog, or a forum comment, or whatever), I’ll write in Microsoft Word for the spell check and other features that aren’t in a standard textarea widget, and then I’ll cut and paste into the form on the site.

The problem is that this carries all the high characters (“smart-quotes” and the like) that MS Word makes straight through to the site — and unfortunately many sites still aren’t set up to handle them. They expect plain (“Latin”) text.

A solution: this script converts text copied from MS Word into plain text. Paste your input into the top box, press clean, and the input will be scrubbed and sent to the lower box. (If you want to clean up Word HTML, rather than just create plain text, I suggest that you use HTML Tidy with the “clean” and “Word 2000” boxes checked.)

Web Developers: feel free to use this code on your own forms to clean your user’s input (although you’d probably be better off doing it server-side). Just change the onClick() method to convert the text inplace.