Log-first or Table-first? Apache Kafka, Fluss, and Streaming Tables

Fri, 28 Aug 2026 12:16:00 +0000
Apache Kafka, Stream Processing, datalake

Apache Kafka is a natural starting point for a streaming architecture. Applications produce events to Kafka, other applications consume them, and data that needs to be retained for longer eventually lands in object storage. A fairly typical version uses Kafka Connect with an S3 sink connector – for example, Confluent's Amazon S3 Sink Connector – to move older data into Amazon S3:

Kafka handles the real-time path, S3 provides cheap and durable storage for historical data, and for an archive this is often enough. The problems start when the data in S3 is expected to behave less like an archive and more like a database table.

Suppose a Kafka topic contains customer updates:

customer-123 -> { name: "Alice", status: "active" }
customer-456 -> { name: "Bob",   status: "active" }
customer-123 -> { name: "Alice", status: "blocked" }

Writing those records to Parquet is easy. Answering a different set of questions is harder:

What is the current state of every customer?

How should updates and deletes be represented?

Which files form a consistent version of the dataset?

What happens when the schema changes?

How do we avoid producing thousands of small files?

How can multiple query engines safely read the same dataset while it is being updated?

S3 gives us files. It does not give us a table.

From files to tables

Open table formats such as Apache Iceberg and Apache Paimon fill this gap. They put metadata around the files: schema, snapshots, partitions, data files, and changes between versions. The architecture becomes something closer to:

The difference is more than file layout. We are no longer simply writing files – we are maintaining a table. Queries can operate on a consistent snapshot, schema changes can be tracked, and data files can be compacted or replaced without exposing partially updated state to readers.

But somebody still needs to translate the Kafka stream into that table. In practice this means running another piece of infrastructure: Kafka Connect, Apache Flink, Spark Structured Streaming, AWS Glue, or a custom ingestion service. And that service has considerably more work to do than copying bytes from Kafka to S3. It might need to understand schemas, turn CDC records into inserts, updates, and deletes, create table commits, manage file sizes, compact small files, and publish metadata to a catalog.

There are two interesting ways to simplify this pipeline. Confluent's Tableflow makes Kafka-to-lakehouse materialization a managed Confluent Cloud capability; we will come back to it later. Apache Fluss takes a different route and questions whether the streaming layer should be a log in the first place. Before getting to Fluss, it is worth being precise about what Kafka gives us.

Kafka is still a log

A Kafka partition is an ordered log. Records have offsets, producers append records, and consumers move through the log and can rewind to replay old data.

This model is deliberately simple. Kafka does not need to know whether a record represents a customer, an invoice, a database row, or an HTTP...