Reproducing a Tricky Bug in Minutes With a Custom Linux Scheduler Written in Java

Posted on by
Reply

Ever had a tricky bug caused by a race condition or rare concurrency condition that was really hard to reproduce? It’s great when you have a fix that should work in theory, but without a reproducer, only time will tell whether your fix really worked. In this blog post, we’ll revisit my old blog post Hello eBPF: Concurrency Testing using Custom Linux Schedulers (19), and try to use the concurrency-fuzz-scheduler to reproduce a bug I fixed a while ago in the OpenJDK.

The scheduler aims to be as chaotic as possible; hence, Jake Hillion’s Rust version is called scx_chaos. But we’ll focus on the Java version, the concurrency-fuzz-scheduler, because it’s not only implemented in Java on top of my hello-ebpf library, but it’s also optimized for fuzzing Java applications, inserting random sleeps at the scheduler level with a focus on non-VM threads.

TL;DR: The concurrency scheduler is a nice tool to provoke rare parallelism conditions and create reproducers.

Image

The bug in question is JDK-8366486, reported by David Holmes in August 2025: A test case that checks that we can run multiple recordings with the CPU-time sampler in direct succession does work. The only problem: The test should not work, but it still worked most of the time. If you’re only interested in the actual bug, skip ahead to the end of the blog post for an explanation.

You’ll find the fixed version here and the broken version here (because the old JDK with the actual bug had compilation issues on my current system, I had to reintroduce the bug in a separate branch).

Let’s start with running the test case with the standard Linux scheduler on a large machine, so that everything can run nicely in parallel:

Continue reading →

Reading and Writing JFR Files Programmatically

Posted on by
Reply

Last week, I showed you the Fastest Way to get the Version of a Java Installation. This week, I’ll show you something completely different: how to interact with JFR data programmatically, showcasing a new library called basic-jfr-processor in the process.

While JFR is a great tool for profiling your application and gaining insights, the file format is, on purpose, not well documented or specified. One of the best sources of information is Gunnar Morling’s blog post on the topic, and of course, the OpenJDK source code.

But of course, there are ready-made APIs for reading JFR files and OpenJDK-adjacent libraries to write them. In this overview blog post, I’ll showcase the built-in Java JFR API, Jaroslav Bachorik’s jafar API, and the JMC JFR writer API, as well as my own basic-jfr-processor library based on the latter.

We start with the built-in API:

Continue reading →

The Fastest Way to get the Version of a Java Installation

Posted on by
Reply

Last week, I demonstrated that OpenJDK is faster than GraalVM Java, at least for obtaining the Java version. This even prompted the mighty Thomas Wuerthinger (creator of GraalVM) to react. But the measured ~20ms for the OpenJDK is still too slow for applications like execjar, where it could significantly increase the runtime of short-running CLI tools. In this week’s brief blog post, I’ll show you the fastest way to access the Java version.

The main performance issue is that calling java -version creates a process with a fairly large (around 38MB) maximum resident set size, and using a proper command line parser. But do we actually need to call the java binary to get the version?

TL;DR: I created the java-version tool, which can obtain the Java version in under a millisecond.

Basic Idea

No, we can just realize that most Java installations have a release file that contains the relevant information in a machine-readable format. You can find this file in the main folder of the installation (./release when java is in ./bin).

Continue reading →

OpenJDK is faster than GraalVM Java*

Posted on by
Reply

Well, we all know that the most crucial feature of the JVM runtime is the -version output. So how does the OpenJDK (in the form of SapMachine) compare with GraalVM? It’s significantly faster. Using hyperfine, we can see that GraalVM 25 CE takes almost twice as long to emit the version number as a regular SapMachine 25 on my MacBook Pro M5:

Image

The slowness of java -version was actually one of the performance issues of the tool I showcased in How to Build an Executable from a JAR using ExecJAR, as it originally used java -version a lot to check the Java version constraint.

Is this relevant? Not really. However, so are most microbenchmarks and benchmarks in general that are taken out of context. You should not generalize small benchmarks, and modern systems are complex.

You can find some bigger, non-version-related benchmarks comparing the different JVMs, for example, at https://ionutbalosin.com/2024/02/jvm-performance-comparison-for-jdk-21/.

Join me next week for a blog post on something different and learn how to check the version of a Java installation even faster in under one millisecond:

Image

P.S.: I just ran some more benchmarks: OpenJDK 25 is 18% faster than OpenJDK 17 and 21 and a whopping 84% faster than OpenJDK 11. Upgrade now!

Image

P.P.S.: As many people (Thomas Wuerthinger, Fabio Niebhaus, Volker Simonis, and multiple of my SapMachine colleagues) pointed out, the differences between OpenJDK and GraalVM are due to the GraalVM initializing the JVM Compiler Interface (JVMCI). The difference between the two becomes negligible when running OpenJDK with enabled JVMCI (initialize the JIT at the beginning):

Image

How to Build an Executable from a JAR using ExecJAR

Posted on by
Reply

In my last blog post, I covered a new tool called jstall, which enables you to quickly check on a Java application. Because it was tiresome to always call the tool via java -jar jstall, I looked for a way to create executables directly from JARs, inspired by async-profiler’s build system. And I, of course, went down a rabbit hole. In this blog post, I’ll show you how use execjar to easily create your own executable JARs that you can execute directly on the command line while still being valid JARs.

TL;DR: execjar is a CLI and Maven plugin that enables you to create executables from JARs by just adding a few lines to your Maven file:

<plugin>
  <groupId>me.bechberger</groupId>
  <artifactId>execjar</artifactId>
  <version>0.1.1</version>
  <executions>
    <execution>
      <goals>
        <goal>execjar</goal>
      </goals>
    </execution>
  </executions>
</plugin>

When your project is called jstall, this creates an executable with the same name that you can execute directly via ./jstall.

Important: The resulting executable is compatible only with UNIX (Linux and macOS) environments.

However, before I delve into the in-depth configuration options of my new tool, I’d like to provide some background on its implementation.

Continue reading →

Quickly Inspect your Java Application with JStall

Posted on by
Reply
Image

Welcome to the last blog post of the year. Last week, I discussed the limitations of custom JFR events. This week, I’ll also be covering a profiling-related topic and showcasing a tiny tool called JStall.

I hope I’m not the only one who sometimes wonders: “What is my Java application doing right now?” When you don’t see any output. Yes, you could perform a simple thread dump via jstack, but it is hard to understand which threads are actually consuming CPU and making any sort of progress. This is where my tiny tool called JStall comes in:

JStall is a small command-line tool for one-shot inspection of running JVMs using thread dumps and short, on-demand profiling. The tool essentially takes multiple thread dumps of your application and uses the per-thread cpu-time information to find the most CPU-time-consuming Java threads.

First, download the JStall executable from the GitHub releases page. Let us then start by finding the currently running JVMs:

> ./jstall
Usage: jstall <command> <pid|files> [options]

Available commands:
  status     - Show overall status (deadlocks + most active threads)
  deadlock   - Check for deadlocks
  most-work  - Show threads doing the most work
  flame      - Generate flame graph
  threads    - List all threads

Available JVMs:
  7153 ./jstall
  1223 <unknown>
  8136 ./renaissance-gpl-0.16.0.jar
  6138 org.jetbrains.idea.maven.server.RemoteMavenServer36
  5597 DeadlockDemo
  49294 com.intellij.idea.Main

This provides us with a list of options for the main status command, as well as a list of JVM processes and their corresponding main classes. Let’s start checking for deadlocking:

Continue reading →

Don’t use Arrays or other Complex Types in Custom JFR Events

Posted on by
Reply

JDK Flight Recorder (JFR) provides support for custom events as a profiler. Around two years ago, I wrote a blog post on this very topic: Custom JFR Events: A Short Introduction. These custom events are beneficial because they enable us to record additional project-specific information alongside the standard JFR events, all in the same file. We can then view and process this information with the JFR tools. You can freely specify these events in Java.

There is only one tiny problem nobody talks about: Array support (and, in more general, the support of complex types).

Take the following event:

class ArrayEvent extends Event {
    
    @Label("String Array")
    String[] stringArray;
    
    @Label("Int Array")
    int[] intArray;
    
    @Label("Long Array")
    long[] longArray;
    
    @Label("Non array field")
    String nonArrayField = "default";
}

What would you expect to happen when we create the event using the following code?

ArrayEvent event = new ArrayEvent();
event.stringArray = new String[]{"one", "two", "three"};
event.intArray = new int[]{1, 2, 3, 4, 5};
event.longArray = new long[]{100L, 200L, 300L};
event.commit();
Continue reading →

Creating a Gridfinity Chocolate Advent Calendar

Posted on by
Reply

This week is a bit different as I’m working on a fun year-end blog and doing my regular work, so I’ll share with you how to create an advent calendar:

Normal Advent calendars are boring: so let’s make our own! We’ll combine all our favorite technologies: Gridfinity (for the grid system), 3D printing (for the grid), vacuum molding (for the chocolate), laser cutting (for the frame), and automated paper cutting to create advent calendars that are both beautiful and functional.

Along the way, we’ll cover practical food safety considerations and show how these techniques come together to produce something tasty, nerdy, and gift-worthy:

You can mind the positive for the chocolate mold on MakerWorld:

Image

I also created a 2×2 Gridfinity grid that fits RitterSport:

Image

You can learn how to create QR codes out of chocolate in another video of mine:

I usually post computer/Java-related blog posts, but I hope you also liked this blog post, in which I shared my hobbies.

Who instruments the native instrumenters?

Posted on by

Hot-patching the JVM to hook native Java agents

Over a year ago, I wrote a blog post called Who instruments the instrumenters? together with Mikaël Francoeur on how we debugged the Java instrumentation code. In the meantime, I gave a more detailed talk on this topic at VoxxedDays Amsterdam. The meta-agent that I developed for this worked well for Java agents/instrumenters, but what about native agents? Marco Sussitz found my agent and asked exactly this question. Native agents are agents that utilize the JVMTI API to, for example, modify class bytecode; however, they are not written in Java. With this blog post, I’m proud to announce that the meta-agent now supports instrumenting native agents.

TL;DR: Meta-agent allows you to see how an agent, native or Java, transforms bytecode.

There are many examples of native agents, like DynaTrace‘s monitoring agent or async-profiler‘s method tracer. I’m using the latter in my example here, as it’s open-source and readily available. The method tracer instruments the Java bytecode to trace the execution time of specific methods. You can find more about it in the async-profiler forum.

As a sample program, we use Loop.java:

public class Loop {
    public static void main(String[] args) 
      throws InterruptedException {
        while (true) Thread.sleep(1000);
    }
}

Let’s trace the Thrread.sleep method and use the meta-agent to see what async-profiler does with the bytecode:

java -agentpath:native/libnative_agent.dylib \
     -javaagent:target/meta-agent.jar=server \
     -agentpath:libasyncProfiler.dylib=start,trace=java.lang.Thread.sleep,file=duration.html \
     Loop.java

This opens a server at localhost:7071 and we check how async-profiler modified the Thread class:

Code transformation by the native Java method tracer of async-profiler

So we can now instrument native agents like any other Java agent. And the part: As all Java agents are built on top of the libinstrument native agent, we can also see what any Java agent is doing. For example, we can see that the Java instrumentation agent instruments itself:

Image

So I finally built an instrumenter that can essentially instrument my instrumentation agent, which in turn instruments other instrumentation agents. Another benefit is that the instrumenter can find every modification of any Java agent.

Continue reading →

Running an LLM on an Android Phone

Posted on by

In my last blog post, I showed you how to work with JFR files using DuckDB, which started a blog series that I surely will continue. Just not this week. Instead, I want to showcase a tiny app to run AI models using the MediaPipe API directly on your phone. I created the app for another purpose (perhaps described in a future blog post) earlier this year, but never wrote anything about it. So here we are.

TL;DR: I built an Android app that offers AI models via a server

The app is open-source and available on GitHub; it’s experimental, but maybe it can help you build your own apps. You can download the releases page of the repo and install it.

Image
The LLM API endpoint, writing a poem on a backyard scene
Continue reading →

Making JFR Quack: Importing JFR files into DuckDB

Posted on by
Image

In my previous post, I showed you how tricky it is to compare objects from the JFR Java API. You probably wondered why I wrote about this topic. Here is the reason: In this blog post, I’ll cover how to load JFR files into a DuckDB database to allow querying profiling data with simple SQL queries, all JFR views included.

This blog post will start a small series on making JFR quack.

TL;DR

You can now use a query tool (via GitHub) to transform JFR files into similarly sized DuckDB files:

> java -jar target/query.jar duckdb import jfr_files/recording.jfr duckdb.db
> duckdb duckdb.db "SELECT * FROM Events";
┌───────────────────────────────┬───────┐
│             name              │ count │
│            varchar            │ int32 │
├───────────────────────────────┼───────┤
│ GCPhaseParallel               │ 69426 │
│ ObjectAllocationSample        │  6273 │

Or run the queries directly, with the database file being cached (if you don’t pass --no-cache), directly supporting all built-in JFR views:

> java -jar target/query.jar query jfr_files/metal.jfr "hot-methods" 
Method                                                                                                   Samples Percent
-------------------------------------------------------------------------------------------------------- ------- -------
java.util.concurrent.ForkJoinPool.deactivate(ForkJoinPool.WorkQueue, int)                                   1066   8.09%
scala.collection.immutable.RedBlackTree$.lookup(RedBlackTree.Tree, Object, Ordering)                         695   5.27%
akka.actor.dungeon.Children.initChild(ActorRef)                                                              678   5.14%

This view is implemented as:

CREATE VIEW "hot-methods" AS
SELECT
  (c.javaName || '.' || m.name || m.descriptor) AS "Method",
  COUNT(*) AS "Samples",
  format_percentage(COUNT(*) / (SELECT COUNT(*) FROM ExecutionSample)) AS "Percent"
FROM ExecutionSample es
JOIN Method m ON es.stackTrace$topMethod = m._id
JOIN Class c ON m.type = c._id
GROUP BY es.stackTrace$topApplicationMethod, c.javaName, m.name, m.descriptor
ORDER BY COUNT(*) DESC
LIMIT 25
Continue reading →

JFR and Equality: A tale of many objects

Posted on by

In the last blog post, I showed you how to silence JFR’s startup messages. This week’s blog post is also related to JFR, and no, it’s not about the JFR Events website, which got a simple search bar. It’s a short blog post on comparing objects from JFR recordings in Java and why this is slightly trickier than you might have expected.

Example

Getting a JFR recording is simple; just use the RecordingStream API. We do this in the following to record an execution trace of a tight loop using JFR and store it in a list:

List<RecordedEvent> events = new ArrayList<>();
// Know when to stop the loop
AtomicBoolean running = new AtomicBoolean(true);
// We obtain one hundred execution samples 
// that have all the same stack trace
final long currentThreadId = Thread.currentThread().threadId();
try (RecordingStream rs = new RecordingStream()) {
    rs.enable("jdk.ExecutionSample").with("period", "1ms");
    rs.onEvent("jdk.ExecutionSample", event -> {
        if (event.getThread("sampledThread")
                 .getJavaThreadId() != currentThreadId) {
            return; // don't record other threads
        }
        events.add(event);
        if (events.size() >= 100) {
            // we can signal to stop
            running.set(false);
        }
    });
    rs.startAsync();
    int i = 0;
    while (running.get()) { // some busy loop to produce sample
        for (int j = 0; j < 100000; j++) {
            i += j;
        }
    }
    rs.stop();
}
Continue reading →

Silencing JFR’s Startup Message

Posted on by

TD;DR: -Xlog:jfr+startup=error is your friend.

Ever wondered why JFR emits something like

[0.172s][info][jfr,startup] Started recording 1. No limit specified, using maxsize=250MB as default.
[0.172s][info][jfr,startup] 
[0.172s][info][jfr,startup] Use jcmd 29448 JFR.dump name=1 to copy recording data to file.

when starting the Flight Recorder with -XX:StartFlightRecorder? Even though the default logging level is warning, not info?

This is what this week’s blog post is all about. After I showed you last week how to waste CPU like a Professional, this week I’ll show you how to silence JFR. Back to the problem:

Continue reading →

How to waste CPU like a Professional

Posted on by

Or: Hey, keeping the CPU busy for a given amount of time should be easy?

Welcome back to my blog. Last week, I showed you how to profile your Cloudfoundry application, and the week before, how I made the CPU-time profiler a tiny bit better by removing redundant synchronization. This week’s blog post will be closer to the latter, trying to properly waste CPU.

As a short backstory, my profiler needed a test to check that the queue size of the sampler really increased dynamically (see Java 25’s new CPU-Time Profiler: Queue Sizing (3)), so I needed a way to let a thread spend a pre-defined number of seconds running natively on the CPU. You can find the test case in its hopefully final form here, but be aware that writing such cases is more complicated than it looks.

So here we are: In need to essentially properly waste CPU-time, preferably in user-land, for a fixed amount of time. The problem: There are only a few scant resources online, so I decided to create my own. I’ll show you seven different ways to implement a simple

void my_wait(int seconds);

method, and you’ll learn far more about this topic than you ever wanted to. That works both on Mac OS and Linux. All the code is MIT licensed; you can find it on GitHub in my waste-cpu-experiments, alongside some profiling results.

As another tangent: Apparently, my Java 25’s new CPU-Time Profiler (1) blog post blew up on Hacker News. Fun times.

Continue reading →

Profiling with the Cloud Foundry CLI Java plugin

Posted on by

Welcome back to my blog, this time for a blog post on profiling your Java applications in Cloud Foundry and the tool I helped to develop to make it easier.

Cloud Foundry “is an open source, multi-cloud application platform as a service (PaaS) governed by the Cloud Foundry Foundation, a 501(c)(6) organization” (Wikipedia). It allows you to run your workloads easily in the cloud, including your Java applications. You just need to define a manifest.yml, like for example:

---
applications:
- name: sapmachine21
  random-route: true
  path: test.jar
  memory: 512M
  buildpacks: 
  - sap_java_buildpack
  env:
    TARGET_RUNTIME: tomcat
    JBP_CONFIG_COMPONENTS: "jres: ['com.sap.xs.java.buildpack.jdk.SAPMachineJDK']"
    JBP_CONFIG_SAP_MACHINE_JDK : "{ version: 21.+ }"
    JBP_CONFIG_JAVA_OPTS: "[java_opts: '-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints']"

But how would you profile this application? This and more is the topic of this blog post.

I will not discuss why you might want to use Cloud Foundry or how you can deploy your own applications. I assume you came this far in the blog post because you already have basic Cloud Foundry knowledge and want to learn how to profile your applications easily.

The Java Plugin

Cloud Foundry has a cf CLI with a proper plugin system with lots of plugins. A team at SAP, which included Tim Gerrlach, started to develop the Java plugin many years ago at SAP. It’s a plugin offering utilities to gain insights into JVMs running in your Cloud Foundry app.

Continue reading →

Java 25’s new CPU-Time Profiler: Removing Redundant Synchronization (4)

Posted on by

The changes I described in this blog post led to segfaults in tests, so I backtracked on them for now. Maybe I made a mistake implementing the changes, or my reasoning in the blog post is incorrect. I don’t know yet.

In the last blog post, I wrote about how to size the request queue properly and proposed the sampler queue’s dynamic sizing. But what I didn’t talk about in this or the previous blog post are two topics; one rather funny and one rather serious:

  1. Is the sampler queue really a queue?
  2. Should the queue implementation use Atomics and acquire-release semantics?

This is what we cover in this short blog post. First, to the rather fun topic:

Is it a Queue?

I always called the primary data structure a queue, but recently, I wondered whether this term is correct. But what is a queue?

Definition: A collection of items in which only the earliest added item may be accessed. Basic operations are add (to the tail) or enqueue and delete (from the head) or dequeue. Delete returns the item removed. Also known as “first-in, first-out” or FIFO.

Dictionary of Algorithms and Data Structures by Paul E. Black

But how does my sampler use the sampler queue?

Continue reading →

Java 25’s new CPU-Time Profiler: Queue Sizing (3)

Posted on by

Welcome back to my series on the new CPU-time profiler in Java 25. In the previous blog post, I covered the implementation of the new profiler. In this week’s blog post, I’ll dive deep into the central request queue, focusing on deciding its proper size.

The JfrCPUTimeTraceQueue allows the signal handler to record sample requests that the out-of-thread sampler and the safepoint handler process. So it’s the central data structure of the profiler:

Image

This queue is thread-local and pre-allocated, as it’s used in the signal handler, so the correct sizing is critical:

  • If the size is too small, you’ll lose many samples because the signal handler can’t record sample requests.
  • If you size it too large, you waste lots of memory. A sampling request is 48 bytes, so a queue with 500 elements (currently the default) requires 24kB. This adds up fast if you have more than a few threads.

So, in this blog post, we’re mainly concerned about setting the correct default size and discussing a potential solution to the whole problem.

Continue reading →

Java 25’s new CPU-Time Profiler: The Implementation (2)

Posted on by

I developed, together with others, the new CPU-time profiler for Java, which is now included in JDK 25. A few weeks ago, I covered the profiler’s user-facing aspects, including the event types, configuration, and rationale, alongside the foundations of safepoint-based stack walking in JFR (see Taming the Bias: Unbiased Safepoint-Based Stack Walking). If you haven’t read those yet, I recommend starting there. In this week’s blog post, I’ll dive into the implementation of the new CPU-time profiler.

It was a remarkable coincidence that safepoint-based stack walking made it into JDK 25. Thanks to that, I could build on top of it without needing to re-implement:

  • The actual stack walking given a sampling request
  • Integration with the safepoint handler

Of course, I worked on this before, as described in Taming the Bias: Unbiased Safepoint-Based Stack Walking. But Erik’s solution for JDK 25 was much more complete and profited from his decades of experience with JFR. In March 2025, whether the new stack walker would get into JDK 25 was still unclear. So I came up with other ideas (which I’m glad I didn’t need). You can find that early brain-dump in Profiling idea (unsorted from March 2025).

In this post, I’ll focus on the core components of the new profiler, excluding the stack walking and safepoint handler. Hopefully, this won’t be the last article in the series; I’m already researching the next one.

Main Components

There are a few main components of the implementation that come together to form the profiler:

Image
Continue reading →

Profiling idea (unsorted from March 2025)

Posted on by

This is my actual collection of ideas from March 2025, when it was unclear whether the updated JFR sampling at safepoints made it into JDK 25. It eventually did, so I scrapped the ideas. But it offers the reader an interesting, unfiltered look into my ideas and thoughts at the time, probably only useful for people who are really into profiling and the OpenJDK. Just be aware that it is therefore a document of its time (March 2025) and doesn’t reflect the actual current implementation. Also, don’t expect any deeper explanations.

Well, I warned you…

An Experimental Front-End for JFR Queries

Posted on by

Ever wondered how the views of the jfr tool are implemented? There are views like hot-methods which gives the most used methods, or cpu-load-samples that gives you the system load over time that you can directly use on the command line:

> jfr view cpu-load-samples recording.jfr

                                     CPU Load

Time                         JVM User           JVM System           Machine Total
------------------ ------------------ -------------------- -----------------------
14:33:29                        8,25%                0,08%                  29,65%
14:33:30                        8,25%                0,00%                  29,69%
14:33:31                        8,33%                0,08%                  25,42%
14:33:32                        8,25%                0,08%                  27,71%
14:33:33                        8,25%                0,08%                  24,64%
14:33:34                        8,33%                0,00%                  30,67%
...

This is helpful when glancing at JFR files and trying to roughly understand their contents, without loading the files directly into more powerful, but also more resource-hungry, JFR viewers.

In this short blog post, I’ll show you how the views work under the hood using JFR queries and how to use the queries with my new experimental JFR query tool.

I didn’t forget the promised blog post on implementing the new CPU-time profiler in JDK 25; it’ll come soon.

Under the hood, JFR views use a built-in query language to define all views in the view.ini file. The above is, for example, defined as:

[environment.cpu-load-samples]
label = "CPU Load"
table = "SELECT startTime, jvmUser, jvmSystem, machineTotal FROM CPULoad"

With my new query tool (GitHub), we can plot this as:

Image
Continue reading →