Kafka and AI walk into a bar...

Tue, 11 Aug 2026 11:16:00 +0000
Kafka, Apache Kafka, AI, confluent, MCP

A big part of my everyday duties is working with Kafka clusters as an administrator. There are pretty nice UI tools, like for example AKHQ or Kafkdrop but let’s be honest – the CLI tools work best in many situations, including working directly on the brokers/controllers.

CLI tools are pretty stable, you know how to use them, but who REALLY remembers all this --very-long-switches?

Sometimes it’s against common sense, a few examples:

1. I can pass a list of bootstrap servers (comma separated), so the option is --bootstrap-servers? Of course not, it’s --bootstrap-server.

2. You want to pass the path of client configuration to kafka-topics, so it’s for sure --client-config? Of course not, it’s --command-config.

3. Does the console producer and consumer have the --command-config as well? Of course not. It’s --producer.config and --consumer.config, respectively. Is it only me who thinks that changing the --dash-convention to --dot.convention is not logical at all?

It’s August 2026, and AI is literally everywhere. The natural question is: do I have to struggle with all these things manually? Is there any tool that can do all these things for me? Moreover, I’m an old-school admin, which means I prefer the terminal over fancy and shiny UIs.

Let’s play with the mcp-confluent, which is supposed to address exactly this problem – managing Kafka clusters speaking to computers just like other human beings.

What is the MCP, actually?

To answer this question, let’s first answer what LLM is. LLM (Large Language Model) is the set of static knowledge (to all AI experts: sorry for this simplification). LLMs work great if you ask questions like “How..” or “What is..”. But what about “Do something”?

MCP (Model Context Protocol) is an open standard that, in short, allows LLMs to use external systems.

Using MCP servers, you can:

access external data,

call tools,

perform actions on behalf of the user,

work with live systems rather than static knowledge alone.

MCP typically exposes three kinds of capabilities:

1. tools, which are actions the assistant can perform;

2. resources, which are data the assistant can access;

3. prompts, which are predefined task templates or workflows.

Mcp-confluent overview

I decided to play with a tool created by Confluent (it’s worth noting that there are multiple MCP servers related to Kafka). mcp-confluent can work obviously with Kafka, but covers the whole Confluent ecosystem – it can work with Schema Registry, Flink, and Kafka Connect, etc. (depending on the Kafka deployment type, more on this later).

Configuration

According to the documentation, configuring it is pretty easy – run init:

npx @confluentinc/mcp-confluent --init-config

And edit the file.

But to be honest, I found it similar to the instruction on how to draw the owl:

The issue – mcp-confluent is very Confluent-Cloud-centric and it took me a while to configure it properly to work with the generic Apache Kafka cluster.

Here is the example configuration allowing connecting to the two Kafka clusters with typical authentication options: mtls and SCRAM:

server:
  transports: [stdio]
  log_level: "${LOG_LEVEL:-info}"...