Compaction & Retention: Edge Cases That Make Your Kafka Lag Metrics Inaccurate

Mon, 09 Mar 2026 14:00:00 +0000
Apache Kafka, Lag Monitoring, Time Lag

Your dashboard shows 5 seconds of lag for the inventory-updates consumer. Everything looks fine. But users keep complaining about stale data – they're seeing 10-minute-old inventory counts.

You double-check the metrics. Still 5 seconds. The numbers say you're nearly caught up.

The numbers are lying.

After the last article where I explained why time lag matters more than offset lag, I need to clarify it: even an accurate time lag can mislead you. Two Kafka features – log compaction and retention – can delete the very messages your metrics depend on, causing reported lag to be significantly lower than reality.

Let me show you how this happens and what you can do about it.

Quick Recap: How Time Lag Works

Time lag calculation is straightforward in theory:

time_lag = current_time - timestamp_of_message_at_committed_offset

To get that timestamp, we seek the consumer's committed offset and read the message there. Simple.

But what if that message no longer exists?

Kafka doesn't throw an error. It silently returns the next available message instead. That message has a later timestamp, which means our calculated lag looks smaller than it actually is.

Think of it like measuring how far behind a runner is by checking their last checkpoint marker. If someone erased markers 1 through 10, and the next visible marker is 11, you'd record them at checkpoint 11 – even if they're really back at checkpoint 1.

This is exactly what happens with compaction and retention.

Log Compaction Problem

Log compaction is a Kafka feature for topics configured with cleanup.policy=compact. Instead of keeping all messages, Kafka retains only the latest message for each key, deleting older versions.

This is perfect for use cases like:

Change Data Capture (CDC) streams

KTable backing topics

Configuration or state stores

Here's how compaction changes your partition:

Before compaction:

  offset 100: key=user-1, value={"status":"active"},   ts=10:00:00

  offset 101: key=user-2, value={"status":"pending"},  ts=10:00:15

  offset 102: key=user-1, value={"status":"inactive"}, ts=10:00:30  ← newer user-1

  offset 103: key=user-3, value={"status":"active"},   ts=10:00:45

After compaction:

  offset 101: key=user-2, value={"status":"pending"},  ts=10:00:15

  offset 102: key=user-1, value={"status":"inactive"}, ts=10:00:30

  offset 103: key=user-3, value={"status":"active"},   ts=10:00:45

  (offset 100 is gone - compacted away)

The Lag Problem

Now imagine your consumer committed offset 100 before the compaction ran. You want to calculate time lag:

1. Seek to offset 100

2. Kafka can't find offset 100 (it's been compacted)

3. Kafka returns offset 101 instead

4. You read the timestamp 10:00:15

5. If the current time is 10:01:00, you report 45 seconds of lag

But the consumer is actually at offset 100, which was produced at 10:00:00. Real lag is 60 seconds – 25% higher than reported.

In this simple example, the error is small. In production, it can be massive.

Real-World Impact

Compacted topics often have slow-moving consumers. A classic example: rebuilding a KTable from the beginning. Your consumer might be at offset 0, slowly processing through millions of messages while compaction continuously deletes old data behind the latest offset.

Possible scenario:

Reported time lag: 30 seconds

Actual time lag: 30 minutes

The consumer was rebuilding the state from scratch on a heavily...