0.24.0 delivers SASL Termination and a significant evolution of the Filter/Router APIs.

The API change is a breaking one: if you’ve written your own filter or router, you’ll need to make small/easy code changes before adopting this release — we’ve shipped tooling to help. As maintainers, we are genuinely disappointed to have to introduce breaking changes; we never take them lightly. You can read more about why we needed to do this below.

Release Highlights at a Glance:

• New Feature: SASL Termination (OAUTHBEARER, SCRAM-SHA-256/512) directly at the proxy layer.
• Breaking API Change: Relocated core API types from org.apache.kafka.* to io.kroxylicious.kafka.*.
• Automated Upgrade: OpenRewrite recipes available to migrate your codebase automatically.
• Community Feedback: Active discussion open on upcoming design proposals.

Introducing SASL Termination

Kroxylicious 0.24.0 adds built-in SASL Termination, allowing the proxy to authenticate incoming client connections directly rather than forwarding authentication handshakes to upstream Kafka brokers.

Why SASL Termination?

Filters that rely on client identity (such as custom authorization or dynamic routing) need to know who is making a request. While SASL Inspection handles direct downstream-to-upstream connections easily enough, it finds itself out of depth across multiple upstreams. PLAIN and OAUTHBEARER require identical credentials across all upstreams, while SCRAM fails entirely because each upstream expects its own unique handshake token.

SASL Termination resolves this by authenticating requests downstream at the proxy. This opens up support for mixed authentication: with this release, you can accept SCRAM from a downstream client while using mTLS upstream (where the proxy authenticates using its own certificate). While dynamically selecting an identity and how to present it upstream isn’t supported yet, termination provides the baseline needed to build that flexibility.

What it does in 0.24.0:

• Authenticates clients directly — terminates SASL at the proxy for OAUTHBEARER, SCRAM-SHA-256, and SCRAM-SHA-512. The broker never sees the client’s authentication exchange.
• Exposes a verified principal to downstream filters — after authentication, the identity is available to any filter further down the chain, such as the new Authorization filter.
• Works without a broker connection — authentication completes before any upstream connection is made, which is a prerequisite for routing decisions that depend on who the client is.
• Credential isolation — the proxy holds only PBKDF2-derived keys, not plaintext passwords. Client credentials never reach the broker.
• KIP-368 reauthentication — enforces session lifetimes and handles periodic reauthentication transparently.

Getting started

For SCRAM mechanisms, credentials are managed using the bundled scram-credential-tool, which ships in the Kroxylicious distribution at bin/scram-credential-tool.sh. The tool creates and manages a proxy SCRAM credential file—a PKCS#12 file containing only derived keys, never plaintext passwords. Use it to create the credential file and provision users before starting the proxy.

The filter is also hardened against timing side-channel attacks by default: a fixedAuthDelay (200 ms by default) ensures authentication responses take consistent time regardless of whether a username exists or a password is correct. Phantom SCRAM challenges are generated for unknown usernames so that an attacker cannot distinguish a missing user from a failed authentication by counting protocol round-trips.

The filter works in both standalone and Kubernetes deployments. In Kubernetes, credentials are stored in a Secret and referenced using ${secret:...} interpolation in the KafkaProtocolFilter resource—the operator mounts the secret entries automatically.

For full configuration details, see the Authentication Guide. To explore the architecture and design behind this feature, see Design Proposal 124.

The API Shift: Decoupling the Kroxylicious Public API from Kafka Internal APIs.

Until now, the Kroxylicious filter API has depended directly on classes from Apache Kafka’s kafka-clients JAR — types like *Data message classes, protocol infrastructure, and record classes that appear directly in filter method signatures. In Kafka 4.3, the Kafka maintainers did something entirely reasonable: they moved classes they consider implementation details into internal packages to make that boundary explicit. That’s good API hygiene on their part. The uncomfortable truth it forced us to confront is that Kroxylicious was depending on things Kafka never intended as public API — and those classes lived in Kafka’s codebase, under Kafka’s package naming, on Kafka’s timeline. That’s not a stable foundation for a 1.0.

So in 0.24.0 we took source ownership of every type that appears in the Kroxylicious public API. The classes previously imported from org.apache.kafka.* now live under io.kroxylicious.kafka.*, mechanically translated so the sub-package hierarchy is preserved. The original Apache Software Foundation copyright headers are retained, as is appropriate for code redistributed under the Apache 2.0 licence. We now own the source, which means we can evolve it where appropriate for Kroxylicious going forward.

The practical upshot for filter developers: kafka-clients is no longer a compile or runtime dependency of kroxylicious-api. If your filter depends on kafka-clients directly, that’s still fine — it’s your dependency to manage. But the proxy core no longer drags it in transitively. The separation is real at the artifact level, not just cosmetic.

One break, then stable

We made a deliberate choice to do this in a single, clean migration rather than absorb the changes piecemeal. Drip-feeding breaking changes across releases is worse than one honest “here’s what moved, here’s how to update.” Our goal for 1.0 is a stable, predictable API that we control — this is the last time we expect to move these types.

Wire compatibility

Owning the source raises an obvious question: how do you know the classes you copied still produce byte-identical wire output to Kafka’s originals? The answer is that we prove it on every build. We ship a suite of byte-level round-trip fidelity tests that serialise a message with our generated classes, deserialise with Kafka’s, and assert equality — then run the same test in reverse. Every protocol message type, every supported protocol version. If something drifts, CI breaks before it ships.

The wire format is governed by KIPs and is genuinely stable. The Java classes representing it were not. We’ve separated those two concerns.

If you encounter any gaps or missing classes in your custom filters after upgrading, please open an issue on GitHub — we moved what was required, but we may have missed something in the long tail.

Automated Migration with OpenRewrite

To make updating your codebase as smooth as possible, we are providing automated migration recipes powered by OpenRewrite. While we make it sound like a simple x -> y transition, a bash one-liner isn’t going to cut it (no matter how good your Perl is—yes, I’m looking at you, Claude).

Maven

mvn org.openrewrite.maven:rewrite-maven-plugin:run \
  -Drewrite.recipeArtifactCoordinates=io.kroxylicious:kroxylicious-migrations:0.24.0 \
  -Drewrite.activeRecipes=io.kroxylicious.migrations.rewrite.v0_24.MigrateTo0_24

Gradle

plugins {
    id("org.openrewrite.rewrite") version "6.x.x"
}

dependencies {
    rewrite("io.kroxylicious:kroxylicious-migrations:0.24.0")
}

./gradlew rewriteRun -Drewrite.activeRecipe=io.kroxylicious.migrations.rewrite.v0_24.MigrateTo0_24

Further details around running these migrations can be found on GitHub.

Design Proposals & Shaping the Future

We explored the trade-offs and options for this migration in Design Proposal 116.

If you read Proposal 116, you will notice it originally suggested a simple sed script for migration. Design proposals are point-in-time snapshots of our thinking, not unchangeable commandments. As we got into implementation, OpenRewrite offered a far superior, AST-aware refactoring experience.

This shift highlights the importance of our proposal process. If API stability or new proxy features impact how you use Kroxylicious, we strongly encourage you to review and participate in active design proposals on the Kroxylicious Design Repository. Your feedback helps shape the project as we head toward 1.0.

Community Contributions

This release included commits from:

• AdityaThakur1998
• DragonFSKY
• jjj-n
• k-wall
• lntutor
• lukman48
• matheusandre1
• oozan
• piotrpdev
• RafaelReia
• robobario
• SamBarker
• tamikikoz
• TheDarkniteFalls
• tombentley

Thank you all, your hard work is massively appreciated by the PMC!

Artefacts

Download binary distributions and container images from the download page.

Feedback

Drop by and say hello on Slack, GitHub, or bsky. You can also join us in person at a community call.

Highlights of this release:

• Raised the project’s Java baseline to 21, up from 17. Make sure your build targets Java 21 before you upgrade.
• Bumped the Kafka dependency to 4.3.1.
• Relaxed @BrokerConfig so you can now override the replication factor of internal topics (offsets, transaction state log, share coordinator state). This helps when testing coordinator failover across multi-broker clusters.

Feedback

Please let us know, through Slack or GitHub, if you find the extension interesting or helpful.

This release adds PEM key material support to the KMS integrations, and adds support for multi-tenancy in the Thales Cipher Trust Manager integration. Router API integration work continues in incubation.

Check out the full Changelog for everything including deprecations, changes, and removals.

Here are the highlights:

PEM Key Material for KMS Integrations

A long-standing request (this one goes all the way back to #933!) is finally hatched: the KMS integrations now support PEM format key material for TLS trust and client identity. PKCS#1 and PKCS#8 private key formats are both supported. You can configure TLS for your KMS provider directly from PEM certificates, without converting to PKCS#12 keystores first.

CipherTrust Manager Multi-Tenant Support

The CipherTrust Manager KMS plugin now supports an optional domain field in userCredentials. When set, the password-grant token request is scoped to that domain, enabling multi-tenant CipherTrust deployments where different virtual clusters authenticate against different CipherTrust domains:

kms: CipherTrustKmsService
kmsConfig:
  endpointUrl: https://ctm.example.com
  userCredentials:
    username: myuser
    password:
      passwordFile: /path/to/password
    domain: my-tenant-domain

If you use client certificate based authentication, domains are supported too.

Cross-Namespace Strimzi References

Rafael Reia contributed support for cross-namespace Strimzi Kafka references. KafkaService.spec.strimziKafkaRef.namespace can now point to a Strimzi Kafka cluster in any watched namespace, not just the namespace of the KafkaService itself. This is particularly useful for organisations that separate their Kafka infrastructure and proxy configurations into different namespaces.

Router API Integration

We continued integrating the Router API into the heart of the proxy. A RouterFactory can now create a Router that fans out requests (RouterContext.sendRequest()) from a single client connection to multiple upstream clusters based on request content.

This is still an incubating feature and not ready for production, but if you’re curious, unlock the feature flag KROXYLICIOUS_UNLOCK_ROUTING=true (environment variable) and get routing! We’d love to hear your feedback.

Community Contributions

This release included commits from:

AdityaThakur1998, Dahyun Woo, DeCluttered, Geonhyeon Kim, Keith Wall, M Clavo, Rafael Reia, Robert Young, Sam Barker, Tom Bentley, Urjit Patel

Thank you all!

Artefacts

Download binary distributions and container images from the download page.

Feedback

Drop by and say hello on Slack, GitHub, or bsky. You can also join us in person at a community call.

Here are the highlights:

Thales CipherTrust Manager KMS

Keith Wall added a new KMS provider for Record Encryption backed by Thales CipherTrust Manager. Supported authentication mechanisms are username/password and client certificate authentication.

kms: CipherTrustKmsService
kmsConfig:
  endpointUrl: https://ctm.example.com
  userCredentials:
    username: myuser
    password:
      passwordFile: /path/to/password

Named Cluster Definitions (clusterDefinitions)

Previously each virtual cluster had its own inline targetCluster, duplicating connection details across virtual clusters that share the same upstream. Now you define the target cluster once under the top-level clusterDefinitions list and reference it with target: { cluster: "<name>" } from any virtual cluster.

Before:

virtualClusters:
  - name: dev
    targetCluster:
      bootstrapServers: broker1:9092,broker2:9092
  - name: test
    targetCluster:
      bootstrapServers: broker1:9092,broker2:9092

After:

clusterDefinitions:
  - name: my-cluster
    bootstrapServers: broker1:9092,broker2:9092

virtualClusters:
  - name: dev
    target:
      cluster: my-cluster
  - name: test
    target:
      cluster: my-cluster

The targetCluster field is deprecated and will be removed in a future release, but continues to work unchanged for now.

KafkaProxyIngress Infrastructure Annotations

We added KafkaProxyIngress.spec.infrastructure.annotations to the KafkaProxyIngress custom resource. The operator now propagates these custom annotations to the Services and Routes it manages. For example, on AWS you can request a Network Load Balancer instead of the default Classic Load Balancer:

spec:
  infrastructure:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

Java 21 Now Required

Java 17 support has been removed. Java 21 is now the minimum runtime required.

Operator Resilience

Sam Barker fixed a class of operator bugs where KafkaProxy, KafkaService, and VirtualKafkaCluster resources could get stuck under API server load or transient unavailability. The operator watches related resources (such as Secrets and ConfigMaps) and reconciles the owning primary resource when they change. Previously, each such event triggered a live API server lookup to find the primary resource; under pressure this could fail and leave the resource stuck. The operator now reads from its local cache instead.

Foundations for What’s Next

Hot Reload

Urjit Patel completed the hot-reload engine! What this means is users that embed Kroxylicious can implement their own mechanism for dynamically reloading individual Virtual Clusters without restarting the whole proxy process. Note that the standalone binary distribution and operator do not yet take advantage of this engine, we are currently shaping this in design proposal #117. Embedders today can use:

• The KafkaProxy.reconfigure() API to push a new configuration while the proxy is running.
• Add, remove, or replace the filter chains of individual virtual clusters.
• New metrics tracking lifecycle state: kroxylicious_virtual_cluster_state, kroxylicious_virtual_cluster_transitions_total, kroxylicious_reconfigure_total and kroxylicious_reconfigure_duration_seconds.

Big thank you to Urjit for driving this implementation.

Routing API

We have taken our first steps towards implementing the Routing API! We recently accepted this proposal which adds powerful capabilities to Kroxylicious. Currently when a client connects to Kroxylicious, the proxy establishes a single connection to an upstream node. The Routing API decouples things so that messages received on a single client connection can be routed to multiple upstream nodes. The new features are not yet user facing, we have published the routing interfaces, enabling developers to start building Router implementations, and have begun implementing the Routing engine.

Community Contributions

This release included commits from:

Dahyun Woo, DeCluttered, Devendra Reddy Pennabadi, DragonFSKY, Francisco Vila, Keith Wall, mapan1984, PaulRMellor, Piotr Płaczek, polachandu, Robert Young, Roshni R, Sam Barker, Tom Bentley, Urjit Patel

Thank you all!

Artefacts

Binary distributions and container images are available on the download page.

Feedback

We’d love to hear from you! Whether you’re kicking the tyres, running Kroxylicious in production, or just find the project interesting — drop by and say hello. You can reach us through Slack, GitHub or even bsky, or tell us in person on one of our upcoming community calls.

Harder than expected. More interesting too.

We gave everyone the numbers in a bland, but slide worthy way, already. This one is the engineering story: how we built the harness, what the flamegraphs actually show, the workload design choices that changed the answers, and the bugs we found in our own tooling.

Why not Kafka’s own tools?

Kafka ships with kafka-producer-perf-test and kafka-consumer-perf-test. We’d used them before. The problems:

• Too noisy: individual runs produced widely varying results depending on JVM warm-up, scheduling jitter, and GC behaviour. Results were hard to trust and harder to compare across scenarios.
• Producer-only view: kafka-producer-perf-test gives you publish latency, but nothing about the consumer side. You can’t see end-to-end latency — which is something operators actually care about.
• Awkward to sweep: running parametric rate sweeps requires scripting around these tools, and comparing results across scenarios requires manual work.
• Coordinated omission: under load, kafka-producer-perf-test only measures requests it actually sends! So when things start loading up and applying back pressure the send rate drops and the latency stays looking nice and healthy. Only it’s not healthy in reality, things are queuing up in your producer.

And critically, it’s never heard of Kroxylicious… You have though, you’re here!

OpenMessaging Benchmark (OMB) is a better fit. It’s an industry-standard tool used by Confluent, the Pulsar team, and others for their published performance comparisons — so who am I to argue? OMB coordinates producers and consumers across separate worker pods, runs a configurable warmup phase before taking measurements, takes latency tracking seriously — correcting for coordinated omission, and outputs structured JSON that’s straightforward to process programmatically. What’s not to like?

Using OMB also means our methodology is directly comparable to other published Kafka benchmarks. The numbers aren’t comparable, of course — it’s not the same hardware, network conditions or phase of the moon.

What we built on top of OMB

So we just fire up OMB and get some numbers, right? Errr no. OMB just does the measurement part. I work really hard at being lazy, I hate clicking things with a mouse and I knew these tests needed to be repeatable. So we scripted deployment (of all the things) teardown (for isolation), diagnostic collection (WHAT BROKE NOW??), and last but not least result processing (what does this wall of JSON mean?)

So now all of that lives in kroxylicious-openmessaging-benchmarks in the main tree (mono repo FTW).

So we have a tool and we think Kroxylicious is fast — but how do we turn that into something we can actually show management? “Fast” is shorthand for “low impact”, and the impact of a proxy shows up along two dimensions:

• Latency: how much extra time does this additional hop add?
• Throughput: how much does routing traffic through the proxy cost my topic throughput?

Two dimensions, two questions — and it turns out they need quite different experimental approaches to answer.

Rate sweep — where does latency start to bite? scripts/rate-sweep.sh holds the connection count fixed and steps the producer rate up in fixed increments, letting the cluster stabilise at each step. We defined saturation as the sustained throughput dropping more than 5% below the target rate. The rate sweep tells you where the cliff edge is and what latency looks like as you approach it.

Connection sweep — is the ceiling per-connection or per-pod? scripts/connection-sweep.sh holds the per-producer rate fixed and steps up the number of producers (1, 2, 4, 8, 16 by default) — consumers scale to match. This tells you the aggregate throughput ceiling of a single proxy pod (need more? help out!): the point where adding more connections stops increasing total throughput.

Both sweeps use scripts/run-benchmark.sh under the hood, which:

1. Deploys the Helm chart for the requested scenario
2. Waits for the OMB Job to complete
3. Collects results: OMB JSON, a JFR recording, an async-profiler flamegraph, and a Prometheus metrics snapshot
4. Tears down

The --skip-deploy flag lets you re-run a probe against an already-deployed cluster — both sweep scripts deploy once and probe many times.

Banishing click-ops

Coming from Red Hat, my instinct is to reach for an operator — but operators are great at managing cohesive things. The stack we needed to deploy is anything but cohesive: an OMB coordinator, worker pods, a Strimzi-managed Kafka cluster, the Kroxylicious operator, the proxy itself, and HashiCorp Vault for the KMS. It’s less “managed application” and more all your base CRs belong to us.

We could have dumped some YAML in a directory and used kustomize apply. But I am lazy, and that’s a lot of typing. Helm handles this beautifully — one chart, scenario-specific overrides, and a single command to deploy the whole thing. Scenario-specific configuration lives in helm/kroxylicious-benchmark/scenarios/ as YAML overrides — the base chart stays stable and each scenario adds only what it needs:

If you have your own KMS — and you will run this on your own infrastructure, right?! — you can swap Vault out without touching the base chart.

JSON always comes in megabytes

Each benchmark run produces a blob of structured JSON. Useful in principle; a wall of noise in practice. Three JBang-runnable Java programs (I’m a dyed in the wool java dev, sue me) pull out the signal:

• RunMetadata: captures the run context — git commit, timestamp, cluster node specs (architecture, CPU, RAM), and on OpenShift, NIC speed read from the host via the MachineConfigDaemon pod. Generates run-metadata.json alongside each result so you can always tell what conditions produced a number. This is what makes run-to-run comparisons meaningful — and when a run takes 12 hours, trust me, you don’t want to re-run it without good reason.
• ResultComparator: answers “did this change hurt?” — reads two scenario result directories and produces a markdown comparison table. Baseline vs encryption is the obvious use, but the tool is generic. Already running a proxy? proxy-no-filters vs encryption tells you the cost of the filter itself, not the proxy hop. Building your own filter? That’s your comparison — measure the chain with and without it.
• ResultSummariser: answers “where does it fall over?” — reads a rate-sweep result directory and prints a summary table: target rate, achieved rate, p99, and whether the probe saturated. Where ResultComparator compares two scenarios at a fixed rate, ResultSummariser tracks one scenario across a range of rates.

Getting NIC speed from a Kubernetes node turned out to be non-trivial — you need host filesystem access to read /sys/class/net/<iface>/speed. On OpenShift, the MachineConfigDaemon pods mount the host at /rootfs, so we kubectl exec into the MCD pod and chroot /rootfs to read the speed file without creating any new privileged resources. Fiddly, but worth it — knowing your NIC speed is the difference between “the ceiling was the NIC” and “the ceiling wasn’t the NIC”.

Workload design

Benchmarks are artificial constructs. Your traffic patterns are never stable — message sizes vary, topic counts grow, producers burst — so there’s always a tension between numbers that are representative and numbers that are actually repeatable. We leaned towards repeatable.

The primary workload makes Kafka experts wince (I had to squirm to type it) — 1 topic, 1 partition, 1 KB messages. Concentrating everything onto a single TopicPartition means we hit the limits earlier, at lower absolute volumes, which makes the proxy’s contribution easier to isolate. Isolating the proxy is, after all, the goal.

But Kafka is often described as a distributed append-only log, and we can’t ignore the word “distributed” when it comes to latency. With RF=1, the proxy doubles the sequential hops in the critical path: one becomes two. That’s not wrong, but it’s not a fair picture either — nobody runs RF=1 in production. With RF=3, the leader waits for ISR acknowledgements before confirming the produce, so there’s already replication latency in the critical path. The proxy adds a real, sequential hop — we’re not trying to bury that — but it lands alongside a cost that’s already there. One extra hop on top of a multi-hop round trip is a different picture from doubling a single-hop one. Three brokers, hot partition replicated across all of them.

But we didn’t abandon representative entirely. The multi-topic runs (10 and 100 topics) are the reconnection point: load spread across more topics, closer to what production actually looks like, at rates well below any saturation point. You’re measuring the proxy’s baseline tax — the cost you always pay, not just the cost when you’re pushing hard. It holds.

That covers the first dimension — the proxy’s latency tax at normal load. For the second, throughput, the question is: how much does routing through the proxy reduce your maximum sustainable rate? That needs a different approach. We used rate sweeps: hold the connection count fixed, step the rate up incrementally, and watch what happens. Below the ceiling, achieved throughput tracks the target — the system keeps up. Above it, it can’t, and falls behind. The point where achieved throughput diverges from the target rate — where we defined that as dropping below 95% — is the saturation point. That’s the knee of the curve, and that’s what we were hunting.

False summit

The rate-sweep result was in: the encryption scenario hit a ceiling on our original cluster at around 37k msg/s. Summit reached.

Except — the proxy had spare CPU cycles. Not a little: meaningful headroom. If the proxy isn’t CPU-saturated, whatever we hit isn’t the proxy’s ceiling.

Was it the NIC? At 37k msg/s and 1 KB messages, produce traffic alone is 37 MB/s. Add RF=3 replication: the leader ships two copies outbound, ~74 MB/s more. 111 MB/s total — fine for 10 GbE, obviously broken for 1 GbE. If the NICs had been gigabit, replication traffic would have saturated them long before we got to 37k. Network eliminated.

Was it the proxy pod, or just one connection? The rate sweep runs with a single producer. We ran four at the same per-producer rate. Aggregate throughput climbed higher than one producer alone could push — the pod had headroom the single connection wasn’t using. We checked proxy metrics: back pressure was minimal. The proxy wasn’t the constraint. Whatever was limiting one connection, it wasn’t us.

We tried anti-affinity

Then a curveball: could it be node saturation? The original cluster had three worker nodes — and three Kafka brokers. Strimzi, being sensible, spreads brokers evenly: one per node. If the proxy had landed on the same node as a busy broker, that node could be the bottleneck rather than the proxy pod itself.

We added a hard anti-affinity rule to keep the proxy off broker nodes. It wouldn’t schedule.

The penny drops: three worker nodes, three brokers, one per node — there is nowhere for the proxy to go that isn’t already co-located with a broker. Obvious in hindsight. We needed a bigger cluster.

We provisioned one: eight workers, three masters, 16 vCPU per node.

The baseline shock

Baseline first. Direct Kafka, no proxy.

~19,400 msg/s. The original cluster had been sustaining ~50,000.

The proxy wasn’t in the picture. We checked the obvious suspects: disk I/O — fine, local and unsaturated. OMB worker scaling — correct. Broker CPU: ~1.2 vCPU. Nothing was at a limit.

The answer was in the pipeline arithmetic. A Kafka producer has a maximum number of in-flight requests — batches sent but not yet acknowledged. With real round-trip times between nodes, that in-flight window bounds throughput. We measured: 0.87 ms between worker nodes, with three replication hops before the leader can confirm a produce at RF=3 — roughly 3–4 ms total. Five in-flight requests across that round trip gives a theoretical ceiling in the right ballpark, and the measured ~19,400 msg/s baseline confirmed it.

On the original cluster, those nodes were almost certainly co-located on the same physical host. Inter-node RTTs at that scale are sub-millisecond — effectively free. The original cluster’s 50k baseline wasn’t what a 3-broker Kafka cluster does. It was what a 3-broker Kafka cluster does when the network is a memcpy.

The new cluster was genuinely distributed. Real latency, real pipeline limits, real Kafka — and the cluster we used for everything from here.

(The ~37k ceiling is the only figure in this post from the original cluster. Everything that follows — the coefficient, the CPU sweep, the prediction — was measured on the new cluster. The physics are part of what makes those numbers honest.)

Another penny dropped. We’d had the same scheduling problem with OMB all along. The producer and consumer worker pods were landing on broker nodes — and when pods share a node, the SDN detects that traffic doesn’t need to leave the node and bypasses the NIC entirely. The producers and consumers weren’t paying for network transit at all.

The proxy pod was on a different node, but on a 3-node cluster where every node already had a broker, the odds of those nodes sharing a physical host on Fyre were high. Almost certainly getting the same benefit, just one layer down.

Now push harder

The new cluster had an honest baseline — but RF=3 pipeline limits meant we couldn’t push a single topic past ~17k msg/s. There was no room to find the proxy’s CPU ceiling when Kafka’s pipeline hits the wall first.

RF=1, 10 topics. With no replication hops, the round-trip drops to producer→leader only: 0.87 ms. Spread across 10 partitions, no single one becomes the bottleneck before the proxy does. We validated the workload with the passthrough proxy: throughput scaled well past anything encryption constrains. The ceiling we were now measuring was proxy CPU.

How much more?

The RF=1 10-topic workload spread load across partitions. At 1000m, the run tells us: comfortable at ~82 MB/s (publish p99: 246 ms, E2E p99: 344 ms), with E2E latency blowing up at ~164 MB/s — the proxy’s CPU budget exhausted. The coefficient comes from JFR CPU data across the non-saturated probes:

Measured: 10.0 mc per MB/s of total proxy traffic (±7.8 stdev, n=6 non-saturated probes)
→ operator formula: 10 mc per MB/s of total proxy traffic
→ for 1:1 produce:consume at 1 KB: 20 mc per MB/s of produce throughput

I was proudly showing off some early numbers — baseline vs proxy, looking good — when one of the computer science PhDs on the team asked, “is the difference real?” Best answer I could come up with at the time: “Good question.” So I went and added statistical significance testing.

check-significance.sh runs Mann-Whitney U at p < 0.05, comparing per-window p99 latency samples between baseline and candidate at each rate step. OMB slices the test phase into time windows and records a p99 per window — ~30 samples per 5-minute run — so MWU has enough data to distinguish real signal from noise. It’s not perfect: those per-window samples aren’t entirely uncorrelated — a GC pause can drag multiple adjacent windows — but it gives a principled answer to “is this overhead real, or am I chasing noise?”

The coefficient is a different matter. It’s derived from JFR CPU data across n=6 non-saturated probes; the ±7.8 stdev reflects measurement noise, not a tested confidence interval. It holds at 1, 2, and 4 cores — the linear scaling claim is consistent — but its validity across message sizes or workload shapes is untested.

The mechanism: cpu: 1000m → availableProcessors()=1 → one Netty event loop thread. At 4000m that’s four threads, each handling its share of connections in parallel. If the ceiling scales linearly with thread count, a 4-core pod should handle roughly four times as much. We ran it.

At 4000m: comfortable at 164 MB/s, E2E catastrophic at 329 MB/s — ceiling reached. The proxy isn’t hitting a fixed architectural wall — it’s hitting a CPU budget wall, and that wall moves when you give it more CPU.

The prediction

One validated scaling point isn’t a sizing model. The coefficient predicts that 2-core should sustain well past 82 MB/s and not saturate until well above 164 MB/s. We ran 2-core next.

At 164 MB/s across 10 partitions, each partition carries 16 MB/s — within the budget of a single Netty thread. The elevated E2E latency reflects the proxy running near but not at its ceiling. The 2-core saturation point sits above 164 MB/s; the model is consistent.

The full picture — coefficient measured across all three core counts and three topic counts:

Sizing coefficient (mc per MB/s of total bidirectional proxy traffic). Post 1 uses k=25 mc/MB/s (4-core 10-topic) as the conservative upper bound.

† 4-core 1-topic: at high producer counts, the Kafka partition limit caps single-partition throughput before the proxy saturates — the high stdev (±18.6) reflects this. Use the 10-topic or 100-topic rows for sizing.

The coefficient rises with core count, which might seem at odds with the linear scaling claim. The ceiling table above — 4-core sustaining double the throughput of 1-core — is the cleaner test of linearity. The coefficient is measured from a connection-count sweep where each data point adds both more connections and more throughput simultaneously; decomposing those two effects requires a separate rate sweep at fixed connection count, which we have not yet run. Preliminary analysis suggests the per-byte encryption cost is roughly stable across core counts, and the rising coefficient reflects a per-thread baseline overhead that scales with the number of Netty event loop threads.

Setting requests equal to limits makes this practical: a pod that can burst above its CPU limit introduces headroom uncertainty that breaks the model. Fix the CPU budget; fix the ceiling.

The flamegraph: where the CPU actually goes

I care deeply that the proxy does as little work as possible on the hot path. Optimization is often less about swapping algorithms — if you only ever have five items, who cares how you sort them — and more about realising what work not to do, or finding a better time to do it. Amdahl’s law governs this: the maximum speedup you can get from optimizing a component is bounded by how much of total execution time that component actually owns. If the proxy accounts for 2% of CPU, you can’t optimize your way to a 10% win — not there.

That framing is exactly why flamegraphs matter to me. Not as a debugging tool, but as a way of seeing the shape of the work. I was also hoping to tell a fuller story here — profiles across the full rate sweep, watching the mix shift as the proxy approaches saturation. Getting stable, reproducible numbers turned out to be harder than expected, and the bugs described in the next section cost us more runs than I’d like. So these are two snapshots at a single rate, not the sweep-correlated picture I had in mind. Still enough to see where the CPU goes. I hope to revisit this properly in the future — but right now the proxy’s performance is good enough that I’m focused on functionality, and the benchmarking harness itself still has room to mature.

We captured CPU profiles using async-profiler attached to the proxy JVM via jcmd JVMTI.agent_load, during the steady-state measurement phase. These are self-time percentages — where the CPU is actually spending cycles, not inclusive call-tree time.

The flamegraphs below are fully interactive: hover over a frame to see its name and percentage, click to zoom in, Ctrl+F to search. Scroll within the frame to explore the full stack depth.

No-filter proxy

The proxy is overwhelmingly I/O-bound. 63% of CPU is in send/recv syscalls — the inherent cost of maintaining two TCP connections (client→proxy, proxy→Kafka) with data flowing through the JVM. The proxy itself accounts for 1.7% — and understanding why that number is so small is the interesting part.

Kroxylicious decodes Kafka RPCs selectively: each filter declares which API keys it cares about, and the proxy only deserialises messages that at least one filter needs. Even in the no-filter scenario, the default infrastructure filters are doing genuine L7 work — broker address rewriting, API version negotiation, topic name caching — which means metadata, FindCoordinator, and API version exchanges are fully decoded. But the high-volume produce and consume traffic? The decode predicate skips full deserialisation for those entirely, passing them through at close to L4 speed.

The 1.7% is the cost of a proxy that is selectively L7: doing real Kafka protocol work where it matters, and treating the hot path like a TCP relay where it doesn’t. That’s not a side-effect — it’s what the decode predicate design is for, and this flamegraph validates it.

Encryption proxy (same 13,600 msg/s)

* Send/recv and Netty I/O appear to shrink as a percentage share because encryption adds CPU work that grows the total pie. The absolute I/O cost is similar in both scenarios.

The direct crypto cost is 9.3%: 6.5% in the AES-GCM cipher itself, with the additional 2.8% in Kroxylicious proxy logic representing the encryption filter’s dispatch and record handling on top of the baseline proxy work. But the more striking result is the indirect costs — encryption’s second-order effects dwarf its first-order cost:

• GC pressure (+10.3%): encryption creates a stream of short-lived byte buffers — encrypted ciphertext, plaintext copies, re-encoded Kafka record wrappers — most of them dead before the next record arrives. The JVM’s young-gen collector is working hard to keep up.
• Buffer management (+8.5%): each record must be read into a buffer, decrypted or encrypted into a new buffer, then re-packed into Kafka protocol format — three buffer lifetimes per record instead of one.
• JDK security infrastructure (+5.9%): cipher instance creation, security provider dispatch, key spec handling.
• Kafka protocol re-encoding (+2.3%): encrypted records are a different size to their plaintext originals; they must be re-serialised into the Kafka batch format before forwarding.

If you wanted to optimise this, the highest-impact areas would be: reducing buffer copies (encrypt in-place or use composite buffers), pooling encryption buffers to reduce GC pressure, and caching Cipher instances to reduce per-record JDK security overhead.

There are wins inside the proxy we haven’t chased yet — serialisation and deserialisation we could avoid, buffer copies imposed by how memory records are structured. Some would be straightforward; others would require rethinking how Kafka records are modelled in memory. We haven’t gone after them. But to put it plainly: we can optimise all we like inside the proxy, and we’re still not going to make AES faster.

Bugs we found in our own tooling

During the 4-producer rate sweep, we noticed that JFR recordings and flamegraphs from probes 2 onwards all looked identical to probe 1. They were stale copies. Three bugs.

Bug 1 — wrong JFR settings: When restarting JFR for a subsequent probe in --skip-deploy mode, the script was using settings=default instead of settings=profile. The default profile omits I/O events including jdk.NetworkUtilization — the event we were using to read network throughput from JFR. Fixed to always use settings=profile.

Bug 2 — async-profiler not restarted: The restart block restarted JFR but never restarted async-profiler. All probes after the first had a flamegraph from probe 1 only.

Bug 3 — wrong guard variable: The async-profiler restart was guarded by checking AGENT_LIB (the path to the native library). AGENT_LIB is always set when the library exists on the image — even when profiling was intentionally skipped on clusters where the Unconfined seccomp profile couldn’t be applied. The correct guard is ASYNC_PROFILER_FLAGS, which is only set when the seccomp patch was successfully applied.

Spotting these required noticing that two different probe flamegraphs were pixel-for-pixel identical, then working back through the restart logic. The lesson: when reusing a deployed cluster across multiple probes, validate that diagnostic collection is actually running fresh for each one.

Run it yourself

We’re an open source project — we share our workings. The raw OMB result JSON, JFR recordings, and flamegraph files that back this post are available for download. If you want to verify the numbers, reproduce the analysis, or compare against your own runs, everything you need is there.

If you want to run it against your own cluster, everything is in kroxylicious-openmessaging-benchmarks/ in the main Kroxylicious repository. See QUICKSTART.md for step-by-step instructions. You’ll need a Kubernetes or OpenShift cluster, the Kroxylicious operator installed, and Helm 3. Minikube works for local runs — the quickstart covers recommended CPU and memory settings.

I got so bored re-evaluating everything as I explored anti-affinity that I even scripted the whole exercise for this post — but brace yourself, it has about a 18 hour runtime. tmux and a control node or jump host are your friends here. The full blog post script is available as a gist if you want to reproduce the exact run.

# Run a baseline vs encryption comparison
./scripts/run-benchmark.sh --scenario baseline
./scripts/run-benchmark.sh --scenario encryption

# Compare results
jbang src/main/java/io/kroxylicious/benchmarks/results/ResultComparator.java \
  results/baseline results/encryption

Wait! Just… one… more… run…

The coefficient is validated at 1, 2, and 4 cores for 1 KB messages. There are more runs I’d like to do:

• io_uring: 63% of the passthrough proxy’s CPU goes to send/recv syscalls — each a kernel entry and exit. io_uring batches I/O through a shared ring buffer, eliminating per-operation syscall overhead entirely. Netty has a native io_uring transport. If it halves that 63%, the passthrough proxy becomes genuinely unmeasurable. The flamegraph made this obvious; the run hasn’t happened yet.
• GC tuning: Netty naturally produces a clean generational profile, and the proxy leans into this rather than fighting it. Per-message allocations don’t outlive their request-response cycle; connection-scoped state lives for the lifetime of the connection. Encryption magnifies the GC cost because it requires more per-message allocations than passthrough, but the memory pressure is expected. That makes it a natural fit for tuning: the collector doesn’t have to guess what’s garbage. Generational ZGC is one well-motivated experiment: designed for high-allocation workloads where most objects die young, which matches exactly. But it runs concurrently, trading stop-the-world pauses for continuous background CPU cost. Whether that helps or hurts a proxy that’s already CPU-bound at saturation — here be dragons.
• Message size variation: larger messages should show lower overhead as a percentage (fixed per-record costs spread over more bytes); smaller messages the opposite. 1 KB is a reasonable middle ground but not the whole story.
• Horizontal scaling: multiple proxy pods haven’t been measured; linear scaling is expected but unconfirmed.
• Multi-pass sweeps: each rate point was measured once. Running each probe three times and taking the median would tighten the bounds in the saturation transition zone.

The operator-facing sizing reference and all the key tables are in SIZING-GUIDE.md in the benchmarks directory.

There’s a practical question underneath the hunch too. The most common thing operators ask us is some variation of: “How many cores does the proxy need?” Which translates, from polite engineering into plain English, as: “is this thing going to slow down my Kafka?” We’d been giving the classic answer: “it depends on your workload and traffic patterns, so you’ll need to test in your environment.” Which is true. And also deeply unsatisfying for everyone involved, including us.

So we stopped saying “it depends” — we built something you can run yourselves on your own infrastructure with your own workload, and measured it. Here are some representative numbers from ours.

TL;DR:

• A passthrough proxy adds negligible overhead: publish latency impact is below measurement noise, E2E adds ~2 ms at moderate topic rates, throughput unaffected
• Add record encryption and expect a ~25% throughput reduction; at comfortable rates, E2E latency stays within measurement noise and publish latency adds up to ~10 ms
• The throughput ceiling scales linearly with CPU: budget ~25 mc per MB/s of total proxy traffic (conservative; the companion post has the full coefficient grid)
• The full benchmark harness is open source — run it on your own cluster for numbers that reflect your workload

What we measured

We ran three scenarios against the same Apache Kafka® cluster on the same hardware:

• Baseline — producers and consumers talking directly to Kafka, no proxy in the path
• Passthrough proxy — traffic routed through Kroxylicious with no filter chain configured
• Record encryption — traffic through Kroxylicious with AES-256-GCM record encryption enabled, using HashiCorp Vault as the KMS

We used OpenMessaging Benchmark (OMB) rather than Kafka’s own kafka-producer-perf-test. OMB is an industry-standard tool that coordinates producers and consumers together, measures end-to-end latency (not just publish latency), and produces structured JSON that makes comparison straightforward. More on why we built a whole harness around it in the companion engineering post.

Test environment

No, we didn’t run this on a laptop — it’s a realistic deployment: an 11-node OpenShift cluster on Fyre (8 workers, 3 masters), IBM’s internal cloud platform — a controlled environment. Kroxylicious ran as a single proxy pod with a 1000m CPU limit. The cluster is sized so that the Kafka brokers, the proxy, and the benchmark workers each run on separate nodes, ensuring traffic crosses real network links rather than looping back on the same host.

The primary workload used 1 topic, 1 partition, 1 KB messages. We chose single-partition deliberately: it concentrates all traffic on one broker, so you hit ceilings quickly and any proxy overhead is easy to isolate. We also ran 10-topic and 100-topic workloads to make sure the results hold when load is spread more realistically across brokers.

One important caveat: this Kafka cluster is deliberately untuned. We’re not trying to squeeze every message-per-second out of Kafka — we’re using it as a fixed baseline to measure what the proxy adds on top. Kafka experts will find obvious headroom to improve on our baseline numbers; that’s fine and expected. The deltas are what matter here, not the absolutes.

The passthrough proxy: negligible overhead

Good news first. The proxy itself — with no filter chain, just routing traffic — adds almost nothing. The tables below show all three scenarios side by side.

A quick note on percentiles for anyone not steeped in performance benchmarking: p99 latency is the value that 99% of requests complete within — meaning 1 in 100 requests takes longer. Averages flatter; the p99 is what your slowest clients actually experience, and it’s usually the number that matters.

Two latency metrics appear in the tables. Publish latency is measured from the record’s intended send time — as dictated by the target producer rate — to when the producer receives the broker’s acknowledgement. That means it captures any producer-side delay (backpressure, client queuing, batch accumulation) alongside the network round-trip and ISR replication (we run with acks=all). End-to-end (E2E) latency is measured from that same intended send time to when the consumer receives the record, adding consumer-side fetch batching on top of everything publish latency already covers.

10 topics, 1 partition each, 1 KB messages — 50,000 msg/s (50 MB/s)

At moderate topic counts, traffic is concentrated enough that proxy overhead is more visible.

Negative deltas for publish latency are within measurement noise — they indicate the proxy is indistinguishable from baseline, not that it improves latency.

The passthrough proxy is not adding measurable per-record overhead at this rate. E2E average overhead is +2.1 ms (p<0.001), but practically negligible for any sizing decision.

100 topics, 1 partition each, 1 KB messages — 50,000 msg/s (50 MB/s)

At higher topic counts, the same total load is spread across more partitions and brokers. The proxy does identical work per record regardless — there is no cross-partition coordination. The point of this table is simply to confirm the pattern holds when load is distributed more broadly.

Publish latency overhead is statistically significant at 100 topics (proxy-no-filters p99 +27%, p<0.001). But publish latency at 500 msg/s per topic is a small fraction of E2E, and the E2E picture is what operators care about: differences are within measurement noise.

1 topic, 1 partition, 1 KB messages — 10,100 msg/s (10 MB/s)

With all traffic on a single topic and partition, Kafka is under the most concentrated load — every record contends for the same broker, the same partition, and the same ISR replication round-trip. The proxy still doesn’t register.

The headline: negligible passthrough overhead — throughput unaffected.

What did I take away from this? We replaced a hunch with data. The remarkable part: the proxy is doing this at Layer 7. Most proxies operate on Kafka at Layer 4 — they shuffle bytes without ever understanding what those bytes mean. Kroxylicious works at Layer 7, parsing every Kafka message, yet still adds only a few milliseconds at the E2E average. That’s the design working.

The overhead staying flat across 1, 10, and 100 topics makes sense for the same reason: the proxy doesn’t contend between topics. Think of the proxy as independent circuits on a distribution board — switching the breaker for lights doesn’t cut power to the fridge. A Kafka broker is more like the mains supply itself — every circuit draws from the same source, so heavy load anywhere reduces what’s available everywhere. In the proxy, topics don’t contend for shared resources: proxy overhead scales linearly across them, and this data validates it.

Record encryption: now we’re doing real work

Ok, so let’s make the proxy smarter — make it do something people actually care about! Record encryption uses AES-256-GCM to encrypt each record passing through the proxy. AES-256-GCM is going to ask the CPU to work relatively hard on its own, but it’s also going to push the proxy to parse each record it receives, unpack it, copy it, encrypt it, and re-pack it before sending it on to the broker. With all that work going on we expect some impact to latency and throughput. To answer our original question we need to identify two things: the latency when everything is going smoothly, and the reduction in throughput all this work causes. Monitoring latency once we go past the throughput inflection point isn’t very helpful — it’s dominated by the throughput limits and their erratic impacts on the latency of individual requests (a big hello to batching and buffering effects).

Latency at sub-saturation rates

So we know encryption is doing a lot of work, but to find out the real impact we need to compare it to a plain Kafka cluster (and yes, people do run Kroxylicious without filters — TLS termination, stable client endpoints, virtual clusters — but that’s a different post). The table below tells us that above a certain inflection point the numbers get really, really noisy — especially in the p99 range.

1 topic, 1 KB messages — baseline vs encryption (selected rates from rate sweep):

The table shows encryption’s p99 spiking sharply at 18,500 msg/s — but that ~18k figure is roughly where the forwarding proxy itself saturates (close to the bare Kafka baseline of ~19,400). Encryption gives out earlier. The rate sweep finds exactly where.

Throughput ceiling

A rate-sweep is exactly what it sounds like: pick a starting rate, let OMB run long enough to get a stable measurement, then step up by a fixed increment and repeat until the system can’t keep up. We defined “can’t keep up” as the sustained throughput dropping by more than 5% below the target rate — at that point, something has saturated.

We stepped up from 8k to 22k msg/s in 700 msg/s increments, looking for where throughput drops more than 5% below target. The results:

• Baseline: sustained up to ~19,400 msg/s (the ceiling at RF=3 on our test cluster)
• Encryption: sustained up to ~14,600 msg/s, then started intermittently saturating
• Cost: approximately 25% fewer messages per second per partition

The transition wasn’t a clean cliff edge — the proxy alternated between sustaining and saturating in a narrow band just above the ceiling. That pattern is characteristic of running right at a limit: it’s not that it suddenly falls over, it’s that small fluctuations (GC pauses, scheduling jitter) are enough to tip it either way. Stay below 14k and you’re fine. Creep above it and you’ll notice. The numbers are not absolute — they are just what we measured on our cluster; your mileage will vary.

The ceiling scales with CPU budget

The fact the proxy is low latency didn’t surprise me, but this did — and it matters when we think about scaling. We maxed out a single connection, but that didn’t mean we’d maxed out the proxy.

The single-producer ceiling at RF=3 is Kafka-limited, not proxy-limited — the ISR replication round-trip caps single-partition throughput regardless of how much CPU the proxy has. The proxy still had meaningful headroom: we ran four producers and aggregate throughput climbed higher, while proxy CPU sat at 570m/1000m. The proxy wasn’t the constraint.

To find the proxy’s real ceiling, you need a workload that doesn’t hit the Kafka partition limit first: RF=1, spread across multiple topics. With that workload, the ceiling is squarely in the proxy — and it scales linearly with CPU. The mechanism: CPU limit controls availableProcessors(), which controls how many Netty event loop threads the proxy creates. More threads, more concurrent connections handled in parallel, higher aggregate ceiling.

The practical implication: the throughput ceiling is not a fixed number — it’s a function of the CPU you allocate. Set requests equal to limits in your pod spec; this makes the CPU budget deterministic and the ceiling predictable. The companion engineering post has the full story of how we found this, including the workload design choices needed to isolate proxy CPU from Kafka’s own limits.

Sizing guidance

Numbers without guidance aren’t very useful, so here’s how to translate these results into pod specs.

Passthrough proxy: size your Kafka cluster as you normally would. The proxy won’t be the bottleneck — but if you want to verify that on your own hardware, the rate sweep — which steps the producer rate up incrementally until the system can’t keep up — is exactly the tool for it. Run the baseline and passthrough scenarios back-to-back and you’ll have your own numbers.

With filters (record encryption is the representative example here):

1. Throughput budget: record encryption — among the most CPU-intensive filters we can imagine — imposes a CPU-driven throughput ceiling. As a planning formula:

   CPU (mc) = k × (P + N × C)

   where mc = millicores (the Kubernetes CPU scheduling unit; 1,000 mc = 1 core per second), k = sizing coefficient (mc/MB/s), P = produce throughput (MB/s), N = number of consumer groups, C = consume throughput per group (MB/s)

   On our hardware (AMD EPYC-Rome 2 GHz with AES-NI), we measured k = 25 mc/MB/s on a 10-topic workload with record encryption — a conservative estimate: more realistic deployments with 100+ topics show k = 4–8 mc/MB/s, roughly 3× lower. Simpler filters will be cheaper still. k is measured from real workloads, so measure your throughput and validate on your own hardware. The companion post has the full coefficient grid across topic counts and core allocations.

   1:1 (100k msg/s at 1 KB, 1 consumer group): k=25, P=100, N=1, C=100 → 25 × (100 + 1 × 100) = 5,000m (~5 cores)

   Fan-out (same rate, 3 consumer groups): k=25, P=100, N=3, C=100 → 25 × (100 + 3 × 100) = 10,000m (~10 cores)

   Not running on Kubernetes? Divide the result by 1,000 to get the number of cores to allocate to the proxy process.

2. Latency budget: well below saturation, expect 2–3 ms additional average publish latency and up to ~15 ms additional p99. The overhead scales with how hard you’re pushing — give yourself headroom and you’ll barely notice it.

3. Scaling: set requests equal to limits in your pod spec — this makes the CPU budget deterministic, which makes the throughput ceiling predictable. To increase throughput, raise the CPU limit. For redundancy, add proxy pods.

4. KMS overhead: DEK caching means Vault isn’t on the hot path for every record. Our tests triggered only 5–19 DEK generation calls per benchmark run. The KMS is not the thing to worry about.

Caveats and next steps

These are real results from real hardware, but they don’t tell a story for your workload. A few things worth knowing before you put these numbers in a slide deck:

• Sub-saturation assumed: all results assume the system is operating below its throughput ceiling — both the proxy’s and Kafka’s own replication limits. Above either, queueing and batching effects dominate and the numbers in this post no longer apply. The companion post explains how to identify where those ceilings are.
• Message size: all results use 1 KB messages. The coefficient is message-size-dependent — encryption overhead as a percentage is likely lower for larger messages.
• Horizontal scaling: linear scaling has been validated across CPU allocations on a single pod; multi-pod horizontal scaling hasn’t been measured but is expected to follow the same coefficient.
• Memory: the workloads tested here are CPU-bound before they become memory-bound — we kept container memory settings consistent across all runs (2 Gi request / 4 Gi limit at the pod level) and it was never the constraint. If you’re running larger messages or larger batches, revisit this assumption.

For the engineering story — why we built a custom harness on top of OMB, what the CPU flamegraphs actually show, and the bugs we found in our own tooling along the way — that’s in the companion post.

The full benchmark suite, quickstart guide, and sizing reference are in kroxylicious-openmessaging-benchmarks/ in the main Kroxylicious repository.

Here’s a demo:

But feel free to run this for yourself by checking out the routing POC branch on GitHub. In instructions are in demo/topic-router/README.md.

What’s the backstory?

Nearly a year ago, I had an idea which I wrote up in what’s become known as The Routing Proposal PR. (You know something might be important if it gets initial caps). While a few people commented and it felt like a good concept, it was also a long way from the reality of the proxy codebase at that time. And anyway, it’s not as if this is the only good idea that we had, or the only thing we had to work on. So it got put on The Back Burner, where many an idea sits until its time comes, or it gets forgotten about.

However, for this particular idea it was hard to forget about it for long. Engineers from various companies showed up on our slack channel asking about it. They had use cases similar to some of the use cases described in the proposal. They wanted to know: Was it being worked on? Some were even uttering those words that are music to an open source project’s ears: “How can we help?”

Some of our amazing contributors started the process with some initial refactorings. Now, nearly a year later, we’ve found the time (and, I admit the tokens) to throw at building a proof of concept.

Right now the proof of concept branch is an interleaved mishmash of commits in each of these three areas. However, the take-away is that we believe we have an API which is capable of supporting a non-trivial router.

The router we’ve implemented allows clients to talk to multiple clusters.

How does it work?

At its core, the router maintains a map of which topics belong to which cluster. These mappings are validated to be non-overlapping, so there’s never any ambiguity about where a request should go.

When a producer sends a PRODUCE request, the router groups the topic partitions by their target cluster, fans the request out to each cluster in parallel, then merges the responses back into a single reply. FETCH requests are handled similarly for consumers. This all happens transparently to the client.

To support incremental FETCH (where the broker remembers which topic partitions a consumer cares about, saving the consumer from re-sending that list on every poll), the router maintains per-connection session state for both the client-proxy and proxy-broker sides of each connection. This is more complex under the hood, but invisible to the consumer.

Transactions and consumer groups work differently. Both rely on a broker to act as a “coordinator” for multiple participants (brokers or clients) to work together. At the moment the router supports these Kafka features in a more limited way, by only allowing these client interactions when all the topics involved reside in the same target cluster. The target cluster is determined from the client’s authenticated identity - a constraint imposed by how the protocol works. We have a good idea about how this could be solved in a more general way, but for now our focus is on getting the POC into something which could actually run in production.

Why is this useful?

There are two dimensions to this:

• Building a topic router is useful in itself for helping address a number of real-life headaches for Kafka users. It means that clients don’t need to know where their topics actually live, and it means Kafka service providers can move topics between clusters without bothering users. The holy grail would be to do this completely transparently. Our POC router has limitations which make it somewhat apparent to users, but it’s a big step forward from where we were.

• Building a topic router is a great way to validate the Router API itself: That the runtime is providing the right services to the Router implementation, through appropriate abstractions.

What happens next?

This is at a POC stage. This blog post is announcing that we’re in the middle phase of the transition from Vapourware to Software. What’s that called? Condensationware?

The big picture of what we’ve done breaks down like this:

• The work to add the Router API, and implement the runtime support for it.

• The work to implement the Topic Router plugin using that API.

This is where the Kroxylicious community comes in - turning those three big pieces into something that’s actually usable and supportable.

At this stage none of this is set in stone and we have no idea how long it’s likely to take for us to turn this from a POC into released software that you can actually use. Hopefully it will be less than the year it’s taken to go from idea to POC though, but the truth is no one can say.

Nor can we guarantee that we’ll eventually be able to handle the full gamut of the Kafka protocol. We need to investigate some of the newer functionality (e.g. Two phase commit, Share Groups, and so on). And the whole time the Apache Kafka community are powering onwards adding more functionality.

So why all the hullabaloo?

We think this is pretty cool, and we wanted to let people know that we’re actively working on this. And being open source, we wanted to remind people that they can join us and get involved in any of the following ways:

• Tell us how you’d use the Router API: We have some ideas for other routers we want to build, but maybe you have an idea that only makes sense in your company. That’s OK — not every router needs to be general-purpose. Kroxylicious is built for exactly that kind of bespoke use case.
• Tell us how you’d use a topic router. What Kafka features does it need to support? What functionality does it need for operators?
• Engage with our process for ironing out all the kinks in the API.
• Help us test this thing! We’re especially interested to hear from people who can test at scale on real workloads.

Here are the highlights:

Alpha: Kubernetes Admission Webhook for Sidecar Injection

The headline feature is our new Kubernetes admission webhook for automatic sidecar injection. This alpha release enables transparent Kafka protocol proxying without any application code changes. Define your sidecar configuration with the KroxyliciousSidecarConfig CRD, and the webhook automatically injects the proxy sidecar into matching pods when they are created (note that once created the proxy will not be updated, and must be recreated to reflect changes in the KroxyliciousSidecarConfig). Perfect for adding encryption, validation, or multi-tenancy capabilities to existing Kafka applications.

See the admission webhook guide in the documentation for installation and usage.

Graceful Connection Draining

Virtual clusters now support graceful connection draining during shutdown. Configure drainTimeout on your virtual cluster, and the proxy will stop accepting new connections while waiting for in-flight requests to complete before shutting down. New metrics track whether disconnections completed gracefully or hit the timeout. Essential for zero-downtime deployments and rolling updates in Kubernetes. Sidebar: while we are talking about restarts and deployments, there is also a Connection Expiration Filter which will help rebalance your connections over time.

HAProxy PROXY Protocol Support

Hrishabh Gupta added HAProxy PROXY protocol support. Configure proxy.proxyProtocol.mode to enabled, and the proxy expects the PROXY protocol header before the TLS handshake (or first Kafka RPC if kroxylicious is not terminating TLS). This enables deployment behind HAProxy or other load balancers that use PROXY protocol, and in future could enable new topologies. For example, you could terminate TLS at the load-balancer and pass the SNI hostname information to kroxylicious via the PROXY Protocol. The load balancer would handle TLS computation and certificate rotation, while the Proxy could then offer a single port for all traffic and not terminate TLS.

Strimzi Integration Enhancements

Shubham Rawat enhanced the Kubernetes operator with automatic TLS trust discovery for Strimzi-managed Kafka clusters. Set trustStrimziCaCertificate in your KafkaService, and the operator automatically configures the proxy to trust the Strimzi-signed cluster certificates. One less manual step when integrating with Strimzi. (Note that the Strimzi CA secret must be in the same namespace as the KafkaService)

This release also upgrades Strimzi support to version 1.0.0. If you’re using the Strimzi integration feature (spec.strimziKafkaRef in KafkaService CR), Strimzi 0.49.0 or later is now required.

AWS KMS Improvements

Oleksiy Pylypenko has extended the Record Encryption AWS KMS, added native support for IRSA (IAM Roles for Service Accounts) and EKS Pod Identity credential providers. The credential configuration has been restructured under a unified credentials node, with new credentials.webIdentity and credentials.podIdentity options for EKS workloads. Existing configurations using top-level longTermCredentials or ec2MetadataCredentials continue to work unchanged. See the Proxy Guide for details.

Dynamic TLS Credential Selection

Paco Viramontes implemented a new plugin API enabling dynamic TLS credential selection for upstream connections. Implement ServerTlsCredentialSupplier to select different client certificates for the connection from kroxylicious to a target cluster, based on the TLS certificates sent from the client to kroxylicious. This dynamic selection allows Implementors to build their own complex mutual TLS client certificate selection logic.

Schema Validation Enhancements

Carles Arnal updated the record validation filter to support Avro and Protobuf schema validation alongside the existing JSON schema support. Validate your records against schemas in Apicurio Registry regardless of serialization format. See the docs for schemaType here.

Container Image Rename

The primary proxy container image has been renamed from quay.io/kroxylicious/kroxylicious to quay.io/kroxylicious/proxy. The operator automatically uses the new image name. If you’re deploying the proxy image directly (without the operator), update your deployment configurations. We will continue publishing new public images to quay.io/kroxylicious/kroxylicious, but it is deprecated and will be removed in a future release.

Community Contributions

This release saw exceptional contributions from the community, with commits landed from:

Carles Arnal, Dahyun Woo, Dan Vulpe, Francisco Vila, Hrishabh Gupta, Keith Wall, Ken Huang, Liberty-Swine, m1a2st, Mario Salinas, Matt Van Horn, Mirtunjay Singh, msalinas-se, Oleksiy Pylypenko, Paco Viramontes, PaulRMellor, Piotr Płaczek, Robert Young, Sam Barker, Shubham Rawat, Tanner Smith, Tom Bentley, Trevin Chow, Urjit Patel, ZhangDT

Thank you to everyone who contributed!

Artefacts

Binary distributions and container images are available on the download page.

Feedback

We’d love to hear from you! Whether you’re kicking the tyres, running Kroxylicious in production, or just find the project interesting — drop by and say hello. You can reach us through Slack, GitHub or even bsky, or tell us in person on one of our upcoming community calls.

Here are the highlights:

Connection Expiration Filter

Thanks to the work of May-Abo, a new filter joins the family! The Connection Expiration filter closes client connections after a configurable maximum age. This is perfect for dynamic environments like Kubernetes where you want to rebalance connections across proxy instances as pods scale up or down.

Entity Isolation Filter

Multi-tenancy gets even better with the Entity Isolation filter. Initially supporting groupId and transactionalId entity types, this filter helps you enforce isolation boundaries between tenants sharing a cluster.

Record Validation Filter - Apicurio v3 Upgrade

• Record validation just got a major upgrade with Apicurio v3. The default schema identification has changed from globalId to contentId for better interoperability with Confluent-based Kafka clients. To support migrations, you can still use the old behavior by setting wireFormatVersion to V2, though this mode is deprecated and will be removed in a future release.
• The schema validation filter can now connect to a schema registry protected by internally signed TLS certificates. One less obstacle for air-gapped or enterprise environments.

Kubernetes Operator Enhancements

Two improvements for Kubernetes users:

• OpenShift Route Support: Enable external access to Virtual Clusters via OpenShift Routes using KafkaProxyIngress.spec.openShiftRoute. Off-cluster client access is now a breeze on OpenShift.
• Server-Side Apply: The operator now uses Server-Side Apply for all dependent resources. Existing deployments are unaffected, and externally-applied patches (like annotations from observability tooling) will now survive operator reconciles.

Javadocs on the Website

Public API Javadocs are now published alongside version-specific documentation on kroxylicious.io, making it easier to explore our APIs without leaving your browser.

Configuration Improvements

• Configurable Netty shutdown: New shutdownQuietPeriod and shutdownTimeout fields give you fine-grained control over Netty shutdown behavior with Go-style durations.
• Duration serialization: Filter Config classes can now use Duration types that automatically serialize to/from Go-style strings (e.g., “1h”) without any annotations.

Artefacts

Binary distributions and container images are available on the download page.

Feedback

We’d love to hear from you! Whether you’re kicking the tyres, running Kroxylicious in production, or just find the project interesting — drop by and say hello. You can reach us through Slack, GitHub or even bsky), or tell us in person on one of our upcoming community calls.

Key Takeaways:

• Zero Client Changes: Kroxylicious is wire-protocol compatible. Your applications keep using their existing Kafka clients without knowing a proxy is in the middle.
• Centralized Control: Instead of chasing down policy updates across different languages and teams, you enforce them at the proxy layer.
• Plug-and-Play Filters: Use the built-in Filter mechanism to intercept and transform traffic. For example, our Record Encryption Filter can automatically handle PII encryption before data ever touches the broker.

Ready to see it in action? Head over to kroxylicious.io or jump straight into our quickstarts. We’ve also conveniently recorded a video demonstration of the Proxy Quickstart.