feat(plugins): add Kafka driver plugin

[datlechin]

Adds a Kafka driver plugin: topics in the sidebar, messages in the data grid, consumer group
lag, and producing a message. Registry-only, pure Swift, no new static library.

Fixes #2419.

What it does

A topic is a table and a message is a row, with columns partition, offset, timestamp,
key, value, headers, key_size, value_size. Opening a topic shows its most recent
hundred messages, which is what a Kafka debugging session usually wants and what every other
Kafka UI defaults to.

The query editor takes a small command language:

CONSUME "orders" FROM NEWEST LIMIT 100
CONSUME "orders" FROM OFFSET 1000 LIMIT 100
CONSUME "orders" FROM TIME "2026-08-01T00:00:00Z" PARTITION (0,2) LIMIT 50
PRODUCE INTO "orders" KEY "k" VALUE "{}" HEADER "source" "tablepro"
SHOW TOPICS / SHOW BROKERS / SHOW GROUPS / SHOW CLUSTER
DESCRIBE TOPIC "orders" / DESCRIBE GROUP "order-processor"

DESCRIBE GROUP is the lag report: committed offset, end offset and the gap, per partition.

MCP needed no work. Every tool in TablePro/Core/MCP/ goes through the generic driver
abstraction (a grep for DatabaseType. there returns nothing), so list_tables,
browse_table, describe_table and execute_query all reach Kafka as soon as the driver
exists. That was the other half of the request.

Why pure Swift rather than librdkafka

Both vendoring routes were checked rather than assumed:

• swift-kafka-client declares platforms: [.macOS(.v15)] and TablePro targets macOS 14, so
  it cannot be linked at all. It also needs a .systemLibrary OpenSSL and linkedLibrary("sasl2"),
  which is not shippable in a notarized universal .tableplugin.
• A prebuilt librdkafka.a in Libs/ needs scripts/publish-libs.sh, a gated operation.

Kafka's protocol is length-prefixed binary over TCP and needs no C library. Transport is
SwiftNIO + NIOSSL, already in the dependency graph through Packages/TableProOracle, and
NIOSSL reads the PEM paths SSLConfiguration already carries. zstd comes from
facebook/zstd's own Package.swift (BSD, macOS 10.10+, builds from source), because macOS
supplies no zstd at any layer.

Verified against a real broker, not against the spec

The wire format was measured against Apache Kafka 4.3.1 in Docker before any of it was
written down. Four encoding decisions are not derivable from the surrounding format, and each
fails the same way when wrong: the broker consumes what it can and closes the socket with no
error response and nothing in any log.

Trap	What is actually true
ApiVersions response header	Carries no tagged fields even at v3, though the v3 request header does (KIP-511; Kafka's generator hard-codes apiKey == 18). Parsing one shifted a real broker's reply from error 0 to error 76.
Compact null array	0x00, not the legacy 0xff. 0xff sets the varint continuation bit and the broker eats the bytes that follow.
Metadata v11	Removed include_cluster_authorized_operations, so v12 has two trailing booleans, not three.
Fetch v13+	Replaced the topic name with a 16-byte UUID (KIP-516), so v12 is the version ceiling.

scripts/check-kafka-protocol.sh is committed so a Kafka version bump re-checks all of this
instead of trusting a transcription, the same shape as scripts/check-redis-command-routing.sh.
It runs 16 checks against a live broker, including producing a batch whose CRC-32C the broker
accepts, which is what proves the hand-rolled Castagnoli table.

Compression was measured too: the broker returns exactly what the producer stored and never
transcodes. Magic bytes confirmed gzip 1f8b, xerial-framed snappy \x82SNAPPY, LZ4
frame 04224d18, and zstd 28b52ffd, so all four are decoded client-side.

What the grid deliberately will not do

Kafka has no update primitive, no per-message delete and no server-side sort, so those are
disabled rather than faked. Topics are typed external table, whose allowsRowEditing is
false, so the grid refuses a cell edit up front with a real message instead of accepting one
and failing at save. generateStatements also returns nil, because returning nil alone makes
Save silently do nothing.

Paging is anchored: buildBrowseQuery bakes the resolved per-partition offsets into the
statement, so page two continues page one instead of re-deriving "newest" against a tail that
has moved. Fetches use read_committed and drop control batches, or a transactional topic
would show commit markers as rows.

Behavioural change outside the plugin

DatabaseManager+Tunnel.swift gains a Kafka arm. Kafka's Metadata reply names brokers by
their advertised address and clients are expected to dial those directly, but behind a
tunnel there is one forwarded port, so broker-2.internal:9092 is unreachable and the fetch
hangs. This is the same pin MongoDB (directConnection) and Redis (standalone) already get
there, for the same reason. It was found by hitting it against a real broker in Docker.

DatabaseCategory gains a streaming case. Kafka is an event streaming platform and the
nearest existing category was coordination, which it is not.

What review caught

Two independent reviews ran over this (a security-focused pass and a general one). Both found
real defects I had written, and both are fixed here rather than deferred.

Authentication bypass. SCRAM's server-signature check was skipped entirely when the
server-final message was not valid UTF-8: that branch returned, and a normal return meant
"authenticated". A hostile broker could answer any challenge with a single 0xFF byte and be
reported as connected, having never known the password. It now throws like every other path.

Attacker-chosen key-derivation cost. The SCRAM iteration count was accepted down to 1, so
a broker chose how cheaply an intercepted proof could be cracked offline. RFC 7677's floor of
4096 is now enforced.

Passwords containing , or = could never authenticate. RFC 5802's saslname escaping
was being applied to the password as well as the username. It belongs to the username alone;
the password is a raw PBKDF2 input. A Confluent-style API secret ending in = would have
failed with correct credentials.

A hang on the default browse path. bounds issued its two ListOffsets calls with
async let, and both landed on the same connection. Actors are re-entrant, so the second
request overwrote the first's continuation: the first never resumed and the tab spun forever.
The calls are sequential now, and expect refuses a second registration instead of dropping
one, so an overlap is a named error rather than a hang.

Paging drifted. buildBrowseQuery never emitted the ANCHOR clause the design called
for, so page two re-derived NEWEST against a moved tail and repeated rows from page one.
The resolved anchor is now recorded and named explicitly on every later page, which is what
the ANCHOR grammar existed for.

A crash from the query editor. SKIP was an unbounded Int: SKIP -1 trapped in
dropFirst and a huge value overflowed the skip + limit addition. Both ends are bounded now.

Also fixed: ListOffsets read uncommitted while Fetch read committed, so an open
transaction made every page escalate its byte budget through four round trips and report a
false truncation; aborted-transaction filtering keyed on offset alone, so one producer's
commit marker released another producer's aborted run and leaked rolled-back rows; the batch
decoder re-sliced the whole remaining blob per batch, making it quadratic in the fetch size;
reserveCapacity trusted a wire-supplied record count; the zstd path trusted a declared size
for both its Int conversion and its allocation; LZ4 sized its destination at 255x the block
instead of the frame's own declared maximum, zero-filling 16 MB per 64 KB block; and the
event loop group was created in init but released only by disconnect(), which a failed
connect never reaches, so every rejected credential leaked a thread.

Security

Security Protocol now decides whether the connection is encrypted and the SSL mode decides
how strictly the certificate is checked. Without that, choosing SASL_SSL while the SSL mode
sat at its default of Disabled opened a plaintext socket and sent the SASL password over it.
A TLS protocol with no mode chosen now verifies fully, and a non-TLS protocol never quietly
encrypts because a stale mode was left behind.

The TLS mode mapping matches OracleTLSMapper exactly, and .preferred/.required mapping
to no verification is the convention every other driver here follows. The driver no longer
retains the DriverConnectionConfig after init, because it carries the password and nothing
reads it again.

Testing

Unit tests across 8 suites: golden-byte codec tests for every framing decision above, record
batch decoding (multi-batch blobs, partial trailing batches, control batches, per-producer
aborted runs, null vs empty payloads, an encode/decode round trip with a verified CRC), all
four compression codecs, the KafkaQL parser including the bounded SKIP and the quote round
trip, the flattener's column typing and its arithmetic timestamp formatter cross-checked
against Foundation, merge ordering, the SSL/SASL resolution, and SCRAM against RFC 7677's
published vectors.

One of those tests caught a real bug before either reviewer saw it: the gzip loop treated
"input exhausted" as success, so a truncated stream returned partial output that looked whole.
Fixed in the source, not the test.

No UI automation. The plugin adds no new UI: it renders through the existing grid, sidebar and
query editor, and the flows that would be worth driving all need a live broker, which CI has
no way to provide.

Checks

Step	Result
verify.sh build	PASS
verify.sh test (8 Kafka suites)	PASS
verify.sh docs	PASS
swiftlint --strict (plugin, app and test files)	clean
KafkaDriverPlugin scheme	BUILD SUCCEEDED
check-plugin-manifest.py	24 plugins agree
check-kafka-protocol.sh against Kafka 4.3.1	16/16
verify.sh plugins (AllPlugins)	FAIL, pre-existing and unrelated

The AllPlugins failure is the Oracle plugin's @TaskLocal macro on this local toolchain
(unknown attribute 'usableFromInlinenonisolated'), which reproduces without this branch.
There are zero Kafka errors in that log and the KafkaDriverPlugin scheme builds on its own.

One thing worth flagging for review rather than fixing here: with SASL_PLAINTEXT selected,
the SSL pane still shows whatever mode was last chosen while the connection is deliberately
unencrypted. The behaviour is right (Kafka's listener protocol is the authority) but the pane
is misleading, and ConnectionSSLView already has the warning-label pattern for it.

Not in this change

Schema Registry (Avro and Protobuf decoding), ACL management, Kafka Connect, ksqlDB, topic
creation and deletion, topic config editing, consumer group offset reset, DeleteRecords,
tombstone delete, live tail, AWS MSK IAM, and OAUTHBEARER.

Schema Registry is the one worth calling out: on a cluster whose values are Avro, the value
column shows raw bytes including the five-byte Confluent prefix. The design is small (a
registry URL on the connection, a codec keyed off that prefix) and it is the obvious
fast-follow rather than a peer of the others on that list.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
TablePro	🟢 Ready	View Preview	Aug 26, 2026, 3:07 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.