[{"content":"Most habit trackers want three things from you before you check a single box: an account, a subscription, and permission to ping you all day.\nI wanted the opposite, a grid I could open, tap, and close. No login, no sync drama. Just Day 12 of 30 with a row of filled circles when I actually did the thing.\nThat became Aesthetic Habit Tracker: a challenge-based grid with a black-and-white, monospace look, local-only storage, and exports good enough to print or post. It\u0026rsquo;s a simple, minimalistic habit tracker that lives in your browser.\nTry it: joel-hanson.github.io/habit-tracker\nSource: github.com/Joel-hanson/habit-tracker\nWhy a Challenge Grid, Not a Calendar Calendar-month trackers are fine until your goal is \u0026ldquo;75 days of X\u0026rdquo; or \u0026ldquo;21 days to build the habit.\u0026rdquo; Then you\u0026rsquo;re mentally doing date math every time you open the app.\nThis tracker is built around a fixed challenge window:\nPick a start date (today, or whenever you actually begin) Pick a length — presets like 7, 14, 21, 30, 60, 75, 90, or 100 days, or any custom count up to 365 The grid shows Day 1 … Day N with real calendar labels underneath (Jun 7, Jun 8, …) You always know where you are: Day 18 of 30, 12 days left. Future days are disabled so you can\u0026rsquo;t accidentally \u0026ldquo;cheat\u0026rdquo; tomorrow\u0026rsquo;s checkbox today.\nThat framing sounds small, but it changes how the tool feels. You\u0026rsquo;re not managing a calendar — you\u0026rsquo;re finishing a challenge.\nWhat It Does Multiple habits — add as many rows as you need; tap a circle to mark complete Responsive grid — the day columns reflow to fit your screen (phone, tablet, desktop) instead of forcing horizontal scroll Touch-friendly — swipe across cells to check off days quickly; long-press a habit name on mobile to delete Share \u0026amp; export — download a screenshot, a print-ready A4 PNG/PDF, or use the native share sheet with an image attached Local backup — export/import JSON so you can move data between browsers or devices without a server Privacy by default — everything stays in localStorage; nothing is uploaded anywhere The visual language is intentionally plain: white card, black borders, uppercase labels, filled black dots for completed days. It reads like a paper tracker that happens to live in the browser.\nA 75-day workout challenge — sidebar shows progress, grid shows each day as a circle\nOn mobile — challenge settings up top, habits below; swipe to check days, long-press a habit name to delete\nShipping on GitHub Pages (Next.js Edition) I recently wrote about shipping a side project to GitHub Pages with plain HTML and JavaScript (Nursing Tracker). This app takes the same free subpath hosting idea but uses Next.js static export:\nhttps://joel-hanson.github.io/habit-tracker/ The build sets output: 'export', a basePath matching the repo name, and deploys via GitHub Actions on every push to main. One wrinkle worth noting: PWA service workers are disabled on GitHub Pages for this repo so a cached worker doesn\u0026rsquo;t interfere with other project sites under the same profile domain. Full PWA install works when you deploy elsewhere (e.g. Vercel).\nLocal preview of the Pages build:\nGITHUB_PAGES=true npm run build:pages npx serve out # Open http://localhost:3000/habit-tracker/ What I\u0026rsquo;d Use It For A 30-day morning routine block A 75 Hard-style multi-habit stretch (multiple rows, one grid) A 21-day \u0026ldquo;just show up\u0026rdquo; streak before deciding if a habit sticks When a challenge ends, export the PDF, start a new start date, and go again. Or import your JSON backup on a new phone and pick up where you left off.\nNo accounts. No upsell. Just circles that fill in when you did the work.\nLinks Live app: joel-hanson.github.io/habit-tracker GitHub repo: github.com/Joel-hanson/habit-tracker Related: From Prompt to Home Screen: Nursing Tracker on GitHub Pages If you build something similar — static frontend, local storage, no backend — GitHub Pages is still one of the fastest ways to put it in someone\u0026rsquo;s pocket.\nFor more side projects and tooling posts, browse the blog archive.\n","permalink":"https://joel-hanson.github.io/posts/28-aesthetic-habit-tracker-challenge-grid-on-github-pages/","summary":"Most habit apps want an account, a subscription, and a notification every hour. I built a minimalist challenge tracker that stores everything locally, exports to PDF, and ships free on GitHub Pages.","title":"Aesthetic Habit Tracker: A Challenge Grid That Lives in Your Browser"},{"content":"My wife was having a hard time keeping track of when she last fed our child, how long each feed took, and how many feeds happened in the morning. She was used to writing it down in a notebook, but that meant finding a book and a pen, and those sometimes got forgotten.\nA phone was always within reach. So I thought: why not build something so simple that a single tap starts recording, based on which side she’s feeding?\nThat became Nursing Tracker, a lightweight browser app with no database, no server, and no app store. Just a repo, GitHub Pages, and the ability to add it to the home screen.\nTry it: joel-hanson.github.io/booby-tracker\nSource: github.com/Joel-hanson/booby-tracker (repo name is an inside joke; the app shows up as “Nursing” on your home screen)\nWhy I Built It (and How It Grew) The first version did one thing: record which side was feeding and when, then show how long it had been since the last feed. That alone solved the “wait, did I already feed him?” problem.\nAs time went on, she had to start going back to work. We started pumping and storing milk so I could feed him during her working hours. I had the option to work from home, which made that possible. Once the app had been running for a few days, the logs told us something useful: we could see roughly how much he drank in a typical nursing session (based on pumping logs and feeding logs). From there it was easier to estimate how much pumped milk we’d need for a day (a rough number, but grounded in her actual averages rather than guesswork).\nSo I added pumped volume logging. Then, when she was at work and I was on feeding duty, I needed the same clarity she had: how much did I last feed him, and when was the last feed? The app became our shared log on two phones. I would have looked into syncing the logs to a server, but I didn\u0026rsquo;t want to pay for a database or hosting or need a over-engineered solution. This solution was simple, privacy-focused, and just worked.\nAfter a few weeks of real use, the patterns became obvious:\n~60–80 ml per feed (when bottle-feeding pumped milk) ~1 hour 45 minutes - 2 hours 15 minutes between feeds, on average (changes based on how much he drinks and how long he nurses for) That’s the whole point of a tool like this: start with the smallest problem, ship it, let real life tell you what to add next.\nDisclaimer: I am a parent and not a healthcare professional or lactation consultant. This is not medical advice. The app is not a replacement for professional advice.\nWhat It Does Nursing Tracker is a single-page app for timing nursing sessions and tracking pumped volume:\nOne-tap timers for left and right feeds Pumped milk logging in milliliters Log view with filters by side or date range Stats tabs for today, this week, and this month Local-only storage via localStorage - nothing leaves the browser It also behaves like a tiny PWA: a web manifest, a service worker for offline use, and home-screen icons so it feels like a native app after you install it.\nThe main screen—one tap to start a left or right feed timer\nThe Log tab is where everything lands: past nursing sessions, pumped volumes, and filters by side or date range. This is what made handoffs between us workable—I could open the log and see exactly when he last ate and how much.\nFeed history at a glance—when, which side, duration, and pumped volume\nWhere It Lives (and How I Published It) The app is hosted as a GitHub Pages project site under my personal profile:\nApp name Nursing Tracker Live URL https://joel-hanson.github.io/booby-tracker/ Repo Joel-hanson/booby-tracker Hosting GitHub Pages (free) Stack Static HTML, CSS, and JavaScript That URL pattern—https://\u0026lt;username\u0026gt;.github.io/\u0026lt;repo-name\u0026gt;/—is the key. You keep your main site at username.github.io (your profile repo) and publish side projects as subpaths, one repo per app.\nInstall on your phone iPhone: Open the link in Safari → Share → Add to Home Screen Android: Open in Chrome → menu → Add to Home Screen (Chrome may prompt automatically) After that it opens full-screen, like any other app icon.\nHow You Can Ship Yours the Same Way If you’re building small tools with an LLM (e.g. calculators, trackers, dashboards, demos), this is one of the fastest paths from “it works on my laptop” to “here’s the link.”\n1. Keep it static GitHub Pages serves files. No Node server, no Python process, no Docker. A folder with index.html (plus CSS, JS, icons) is enough.\nFor personal tools, browser storage (localStorage or IndexedDB) often replaces a backend entirely.\n2. Push to a new GitHub repo git init git add . git commit -m \u0026#34;Initial commit\u0026#34; git remote add origin git@github.com:YOUR_USERNAME/your-app-name.git git push -u origin main Name the repo something short (e.g. \u0026ldquo;nursing-tracker\u0026rdquo;) and you\u0026rsquo;ll see it in the URL.\n3. Turn on GitHub Pages In the repo on GitHub:\nSettings → Pages Build and deployment → Source: Deploy from a branch Branch: main, folder: / (root) Save Within a minute or two the site is live at:\nhttps://YOUR_USERNAME.github.io/your-app-name/ No credit card. No DNS. HTTPS included.\n4. (Optional) Make it installable Add a manifest.json, icons, and a simple service worker if you want offline support and a home-screen icon. Many LLM-generated apps already include these; if not, ask for “PWA manifest and service worker for offline caching.”\n5. Share the link Put the URL in your README, your bio, or a blog post (exactly like this one).\nWhy This Fits AI-Built Side Projects When you vibe-code with ChatGPT, Cursor, or Claude, you usually end up with:\nA single HTML file or a tiny static bundle No auth and no multi-user requirements A problem that only you (or your household) need solved this week Nursing Tracker started as \u0026ldquo;tap left or right and show time since last feed\u0026rdquo;. Pumping logs and stats came later, once we actually needed them. That\u0026rsquo;s a common pattern with LLM-built tools: ship the smallest version, use it for a few days, then ask for the next feature.\nThat profile is a perfect match for GitHub Pages:\nTypical LLM app GitHub Pages Static frontend Serves static files localStorage / no backend No server needed \u0026ldquo;Can I try it on my phone?\u0026rdquo; Add to Home Screen \u0026ldquo;I don\u0026rsquo;t want to pay for hosting\u0026rdquo; Free on public repos \u0026ldquo;I might tweak it later\u0026rdquo; Push to main, site updates You don\u0026rsquo;t need Vercel, Railway, or an app store review for a nursing timer, a habit tracker, or a one-off dashboard. Push the repo, flip Pages on, send the link.\nWhat I\u0026rsquo;d Do Differently Next Time Nothing dramatic, the point was speed and privacy. The app grew feature by feature because our routine changed, not because I planned a product roadmap upfront. If you outgrow local storage (sync across devices, shared accounts), then add a backend. For a household tracker that lives on a couple of phones, static + Pages is still enough.\nLinks Nursing Tracker (live app): joel-hanson.github.io/booby-tracker GitHub repo: github.com/Joel-hanson/booby-tracker GitHub Pages docs: docs.github.com/pages If you ship something similar, the whole loop is: prompt → static files → repo → Pages → home screen. That\u0026rsquo;s the whole release pipeline.\nFor more side projects and tooling posts, browse the blog archive.\n","permalink":"https://joel-hanson.github.io/posts/27-from-prompt-to-home-screen-booby-tracker-on-github-pages/","summary":"My wife was losing track of feeds, so I built a one-tap nursing timer that grew into a pumping log, and shipped it for free on GitHub Pages with no database and no app store.","title":"From Prompt to Home Screen: Shipping Nursing Tracker with GitHub Pages"},{"content":"If you produce millions of Kafka messages where the value is just a number — counters, IDs, timestamps, sequence numbers — you\u0026rsquo;re almost certainly paying for 8 bytes per message whether the value is 1 or 1_000_000_000.\nKafka\u0026rsquo;s built-in LongSerializer is simple and correct. But simple also means fixed-width: every long becomes 8 bytes on the wire and on disk. On high-volume, integer-heavy topics, that adds up fast in broker storage, replication traffic, and egress costs.\nThat\u0026rsquo;s the problem Bijou64 solves. It\u0026rsquo;s a compact, variable-length encoding for unsigned 64-bit integers from Ink \u0026amp; Switch. I wrapped it as a Java library with Kafka serializers so you can swap it in without changing your application logic.\nGitHub repo: github.com/Joel-hanson/bijou64\nWhat Is Bijou64? bijou64 (pronounced \u0026ldquo;bee-zoo-sixty-four\u0026rdquo;) stands for BIJective Offset U64. It\u0026rsquo;s a way to encode integers into 1–9 bytes instead of always using 8.\nThe idea is straightforward:\nValue range Bytes on the wire Example 0 – 247 1 byte 42 → [0x2A] 248 – 503 2 bytes 300 → [0xF8, 0x34] Larger values 3–9 bytes Uses a tag byte + big-endian payload Small numbers stay small. Big numbers grow only as much as they need to. And unlike some varint formats, bijou64 is canonically unique — each number has exactly one valid encoding, so you don\u0026rsquo;t get ambiguous \u0026ldquo;overlong\u0026rdquo; forms.\nIf you want the deep dive on the wire format, Ink \u0026amp; Switch\u0026rsquo;s bijou64 write-up and the upstream spec are excellent.\nWhy Should Kafka Users Care? Think about topics like:\nEvent counters and metrics Monotonic sequence numbers (1, 2, 3, …) Numeric IDs that start small and grow over time Timestamp-like values where most records cluster in a range For these workloads, most values are small. Bijou64 encodes them in 1–3 bytes instead of 8.\nWhat you gain Smaller payloads → less disk on brokers, less data to replicate Less network I/O → especially on producer-heavy pipelines Potential throughput bump → less bytes to serialize, compress, and ship What you don\u0026rsquo;t get Bijou64 is not a replacement for Avro, Protobuf, or JSON. It\u0026rsquo;s not for arbitrary structured objects. If your values are complex records, stick with schema-based serialization (and enable Kafka compression first).\nWhen to Use It (and When Not To) Use Bijou64 when:\nYour Kafka values are mostly numeric (Long / Integer) The topic is high volume and storage or egress costs matter You control both producer and consumer serialization Skip Bijou64 when:\nPayloads are JSON, Avro, Protobuf, or other structured formats Producers and consumers can\u0026rsquo;t share the same serializer/deserializer pair Topics are low volume and the savings aren\u0026rsquo;t worth the migration Rule of thumb: if you\u0026rsquo;d reach for LongSerializer today and your numbers are often small, Bijou64 is worth a look.\nKnow the Limits (Quick Gist) Bijou64 is narrow on purpose — that\u0026rsquo;s what makes it fast and simple:\nRange: unsigned 0 → 2⁶⁴ − 1 (Java long is treated as unsigned on the wire) Size: 1–9 bytes per value — small ints save the most; very large values can use 9 bytes (slightly more than Long\u0026rsquo;s fixed 8) Signed negatives: if you rely on negative Long values, stick with LongSerializer — Bijou64 does not preserve signed semantics the same way Kafka contract: producer and consumer must both use Bijou64Serializer / Bijou64Deserializer on the same field Payload shape: one integer per message — not JSON, Avro, Protobuf, or nested objects Migration: existing topics written with LongSerializer need a coordinated rollout or a new topic If your topic is counters, sequence numbers, or IDs that stay non-negative and mostly small, you\u0026rsquo;re in the sweet spot for cutting network transfer.\nHow It Works in Java The library exposes a simple encode/decode API:\n// Encode 300 → [0xF8, 0x34] byte[] encoded = Bijou64.encode(300); // Decode back to 300 long value = Bijou64.decode(encoded); Under the hood there are two implementations:\nNative (JNI) — wraps the upstream Rust crate for maximum speed Pure Java fallback — used automatically if the native library isn\u0026rsquo;t available Force pure Java if you need it:\nbijou64.useJava=true Drop-In Kafka Integration This is the part that matters for day-to-day use. Swap your serializers — no schema registry changes, no payload restructuring.\nProducer:\nkey.serializer=org.bijou64.perf.kafka.Bijou64Serializer value.serializer=org.bijou64.perf.kafka.Bijou64Serializer Consumer:\nkey.deserializer=org.bijou64.perf.kafka.Bijou64Deserializer value.deserializer=org.bijou64.perf.kafka.Bijou64Deserializer Maven:\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.bijou64\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;bijou64\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.2.0\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.bijou64\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;bijou64-kafka-serializers\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.2.0\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; Build from source (includes the Rust native library via submodule):\ngit clone https://github.com/Joel-hanson/bijou64.git cd bijou64 git submodule update --init --recursive mvn -B clean package Does It Actually Help? Benchmark Numbers I run an end-to-end Kafka producer benchmark in CI on every push to main: 50,000 sequential integers (1..N), Kafka 4.3 (KRaft), Java 17, comparing LongSerializer vs Bijou64 (JNI and pure Java), with and without zstd compression.\nResults from the 2026-06-06 CI run:\nMode Compression Median throughput Avg payload Long none 102,849 msg/s 8.0 bytes Long zstd 94,099 msg/s 8.0 bytes Bijou64 (JNI) none 114,590 msg/s 3.0 bytes Bijou64 (JNI) zstd 98,789 msg/s 3.0 bytes Bijou64 (Java) none 116,683 msg/s 3.0 bytes Bijou64 (Java) zstd 108,589 msg/s 3.0 bytes Takeaways:\n62% smaller payloads on sequential integers (3 vs 8 bytes) ~11–14% higher producer throughput vs LongSerializer without compression Pure Java is competitive with JNI on the CI runner — you won\u0026rsquo;t necessarily need native libs everywhere These are full producer runs (client + broker + network), not isolated microbenchmarks. Run the same test locally for your hardware:\ncd perf/kafka docker compose up -d ./scripts/compare-benchmarks.sh --ci --distribution sequential 50000 2 \\ bijou64-benchmark-topic localhost:9092 For encode/decode-only numbers:\n./scripts/run-jmh.sh A Mental Model Standard LongSerializer Bijou64 ───────────────────────── ───────────────────────── value: 42 value: 42 wire: [8 fixed bytes] wire: [1 byte] value: 300 value: 300 wire: [8 fixed bytes] wire: [2 bytes] value: 1_000_000 value: 1_000_000 wire: [8 fixed bytes] wire: [3 bytes] You\u0026rsquo;re not changing what you send — just how many bytes it takes to represent the same number.\nTry It in 5 Minutes Artifacts are published to GitHub Packages on every release tag (v0.2.0, etc.). You can pull them into an existing Kafka app without cloning the repo.\n1. Authenticate with GitHub Packages GitHub Packages requires a personal access token, even for public repos.\nCreate a classic PAT with the read:packages scope (or a fine-grained token with Packages: Read on Joel-hanson/bijou64).\nAdd credentials to ~/.m2/settings.xml:\n\u0026lt;settings xmlns=\u0026#34;http://maven.apache.org/SETTINGS/1.2.0\u0026#34; xmlns:xsi=\u0026#34;http://www.w3.org/2001/XMLSchema-instance\u0026#34; xsi:schemaLocation=\u0026#34;http://maven.apache.org/SETTINGS/1.2.0 https://maven.apache.org/xsd/settings-1.2.0.xsd\u0026#34;\u0026gt; \u0026lt;servers\u0026gt; \u0026lt;server\u0026gt; \u0026lt;id\u0026gt;github\u0026lt;/id\u0026gt; \u0026lt;username\u0026gt;YOUR_GITHUB_USERNAME\u0026lt;/username\u0026gt; \u0026lt;password\u0026gt;YOUR_GITHUB_PAT\u0026lt;/password\u0026gt; \u0026lt;/server\u0026gt; \u0026lt;/servers\u0026gt; \u0026lt;/settings\u0026gt; The \u0026lt;id\u0026gt;github\u0026lt;/id\u0026gt; must match the repository \u0026lt;id\u0026gt; in your pom.xml below.\n2. Add the dependency to your Maven project Point Maven at the package registry and pull in the Kafka serializers (which transitively includes the core library):\n\u0026lt;repositories\u0026gt; \u0026lt;repository\u0026gt; \u0026lt;id\u0026gt;github\u0026lt;/id\u0026gt; \u0026lt;url\u0026gt;https://maven.pkg.github.com/Joel-hanson/bijou64\u0026lt;/url\u0026gt; \u0026lt;/repository\u0026gt; \u0026lt;/repositories\u0026gt; \u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.bijou64\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;bijou64-kafka-serializers\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.2.0\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; If you only need encode/decode outside Kafka, depend on bijou64 instead of bijou64-kafka-serializers.\nGradle (Kotlin DSL):\nrepositories { maven { url = uri(\u0026#34;https://maven.pkg.github.com/Joel-hanson/bijou64\u0026#34;) credentials { username = providers.gradleProperty(\u0026#34;gpr.user\u0026#34;).get() password = providers.gradleProperty(\u0026#34;gpr.key\u0026#34;).get() } } } dependencies { implementation(\u0026#34;org.bijou64:bijou64-kafka-serializers:0.2.0\u0026#34;) } Set gpr.user and gpr.key in ~/.gradle/gradle.properties to your GitHub username and PAT.\n3. Wire up your Kafka producer and consumer Swap the serializers in your existing app — no changes to your Long business logic.\nproducer.properties:\nbootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.bijou64.perf.kafka.Bijou64Serializer consumer.properties:\nbootstrap.servers=localhost:9092 group.id=my-app key.deserializer=org.apache.kafka.common.serialization.StringDeserializer value.deserializer=org.bijou64.perf.kafka.Bijou64Deserializer auto.offset.reset=earliest Or in Java:\nProperties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, \u0026#34;localhost:9092\u0026#34;); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, Bijou64Serializer.class.getName()); try (KafkaProducer\u0026lt;String, Long\u0026gt; producer = new KafkaProducer\u0026lt;\u0026gt;(props)) { producer.send(new ProducerRecord\u0026lt;\u0026gt;(\u0026#34;counters\u0026#34;, \u0026#34;page-views\u0026#34;, 42L)); } Both producer and consumer must use Bijou64 serializers on the same topic. Mixing LongSerializer on one side and Bijou64Serializer on the other will corrupt your data.\n4. Run it mvn clean package # start your app, or use kafka-console-producer/consumer with the properties above The library falls back to pure Java automatically if the native JNI library isn\u0026rsquo;t on your classpath — no extra setup required for most deployments.\nWant to run the benchmarks locally? Clone the repo, start Kafka with Docker, and compare LongSerializer vs Bijou64:\ngit clone https://github.com/Joel-hanson/bijou64.git cd bijou64 git submodule update --init --recursive mvn -B clean package cd perf/kafka docker compose up -d ./scripts/compare-benchmarks.sh sequential 10000 2 \\ my-benchmark-topic localhost:9092 Credits This project implements the bijou64 wire format from Bijou by Ink \u0026amp; Switch. The native path uses their Rust crate via a git submodule and JNI. My repo adds the Java/Kafka layer, benchmarks, and CI.\nFinal Thoughts Kafka compression (lz4, zstd, gzip) is the first knob most teams turn. That\u0026rsquo;s the right default for general payloads. But when your topic is integer-heavy, you\u0026rsquo;re compressing 8-byte values that didn\u0026rsquo;t need to be 8 bytes in the first place.\nBijou64 attacks the problem at the serialization layer: same semantics, fewer bytes, drop-in Kafka integration. If your pipeline looks like counters, IDs, or sequential numbers at scale, it\u0026rsquo;s worth benchmarking on your own cluster.\nStar the repo if it\u0026rsquo;s useful → github.com/Joel-hanson/bijou64\nFor more Kafka tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/26-shrink-kafka-integer-payloads-with-bijou64/","summary":"Every Kafka record with a Long value you produce sends 8 bytes over the wire, even when the value is 42. Bijou64 encodes integers in 1–3 bytes with drop-in serializers, cutting network transfer and replication traffic on integer-heavy topics by up to 50% or more.","title":"Send Half the Bytes Over the Network: Cut Kafka Integer Payloads with Bijou64"},{"content":"I started Stream Wars as a demo project to explain how Kafka works using something more fun than log lines.\nInstead of having a producer script send fake JSON into a topic, every tap in the browser becomes a real event that flows through Kafka, Redis, and back to the UI in real time.\nStream Wars - a simple game to explain Kafka\nWhy I Built It The goal was to show three things:\nHow frontend actions (taps) become Kafka events How topics are designed around use-cases, not tables How you can see those events both in the UI and in Kafka tooling So I ended up with three main topics:\ngame-taps – every tap from a player (userId, team, timestamp, sessionId) game-updates – processed game state (team scores, total taps, leaderboard) user-metadata – connection metadata (browser, IP, language) for analytics How a Tap Becomes an Event The flow is deliberately simple:\nThe browser sends a POST to /api/tap when you tap. The Next.js API publishes a message to the game-taps topic. A Kafka consumer reads game-taps, updates Redis (scores, totals, user taps). The consumer publishes a summarized state to game-updates. A WebSocket server pushes that update to every connected client. From Kafka\u0026rsquo;s point of view, it\u0026rsquo;s just:\nOne producer writing tap events One consumer updating state One real-time fan-out via WebSockets The nice part is that you can open the game in your browser, start tapping, and at the same time run:\ndocker exec -it kafka ./opt/kafka/bin/kafka-console-consumer.sh \\ --bootstrap-server localhost:9092 \\ --topic game-taps \\ --from-beginning You literally see your taps as Kafka events in the terminal.\nThe Architecture ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Browser │ │ Next.js │ │ Kafka │ │ (Taps) │───►│ Producer │───►│ game-taps │ └─────────────┘ └─────────────┘ └─────────────┘ │ ▼ ┌─────────────┐ │ Consumer │ │ (Updates │ │ Redis) │ └─────────────┘ | ▼ ┌─────────────┐ │ WebSocket │ │ Broadcast │ └─────────────┘ Try It Yourself Start the Game # Clone and start git clone https://github.com/joel-hanson/stream-wars cd stream-wars docker-compose up Then open http://localhost:3000 and start tapping!\nWatch Events in Real-Time In another terminal, watch the tap events flow:\n# Watch game-taps topic docker exec -it kafka ./opt/kafka/bin/kafka-console-consumer.sh \\ --bootstrap-server localhost:9092 \\ --topic game-taps \\ --from-beginning \\ --property print.key=true \\ --property print.timestamp=true You\u0026rsquo;ll see events like:\n{\u0026#34;userId\u0026#34;:\u0026#34;abc123\u0026#34;,\u0026#34;username\u0026#34;:\u0026#34;Player1\u0026#34;,\u0026#34;team\u0026#34;:\u0026#34;blue\u0026#34;,\u0026#34;timestamp\u0026#34;:1703001234567,\u0026#34;sessionId\u0026#34;:\u0026#34;xyz\u0026#34;} {\u0026#34;userId\u0026#34;:\u0026#34;def456\u0026#34;,\u0026#34;username\u0026#34;:\u0026#34;Player2\u0026#34;,\u0026#34;team\u0026#34;:\u0026#34;red\u0026#34;,\u0026#34;timestamp\u0026#34;:1703001234568,\u0026#34;sessionId\u0026#34;:\u0026#34;abc\u0026#34;} Inspect Topics # List all topics docker exec -it kafka ./opt/kafka/bin/kafka-topics.sh \\ --bootstrap-server localhost:9092 \\ --list # Check topic details docker exec -it kafka ./opt/kafka/bin/kafka-topics.sh \\ --bootstrap-server localhost:9092 \\ --describe \\ --topic game-taps Key Learnings 1. Events as First-Class Citizens Each tap is an event with:\nA unique ID A timestamp User context (userId, team, sessionId) This makes it easy to replay, debug, and analyze later.\n2. Single Source of Truth Redis is the authoritative state store. The consumer:\nReads tap events from Kafka Updates Redis atomically Broadcasts state via WebSocket This eliminates race conditions and ensures accurate counts.\n3. Decoupling Frontend and Backend The frontend doesn\u0026rsquo;t know about Kafka. It just:\nSends HTTP requests Receives WebSocket updates Kafka handles the event streaming behind the scenes.\nUsing It as a Demo This tiny game turned into a good way to:\nExplain event-driven thinking (state derived from a stream of taps) Talk about idempotency and counting (why Redis is the source of truth) Show how Kafka sits between a frontend and a backend without anyone noticing Demonstrate real-time updates across multiple clients The Tech Stack Next.js 16 – Frontend and API routes Kafka (KRaft mode) – Event streaming (no Zookeeper needed!) Redis – Game state and leaderboard WebSockets – Real-time updates TypeScript – Type safety throughout Conclusion Stream Wars shows that you don\u0026rsquo;t need a complex system to demonstrate Kafka\u0026rsquo;s power. Sometimes the best way to explain event streaming is to make it tangible – where every action you take becomes an event you can see.\nIf you want to try it or adapt it for your own demos, the code is here:\nGitHub Repository For more Kafka tips and real-time application insights, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/25-building-stream-wars-a-real-time-multiplayer-tap-battle-game-with-kafka-and-websockets/","summary":"I built Stream Wars as a tiny tap game to demo how Kafka works, by turning every tap in the browser into a real Kafka event you can inspect.","title":"Stream Wars: A Simple Game to Explain Kafka"},{"content":"The Celery Question For years, when someone asked \u0026ldquo;How do I handle background tasks in Django?\u0026rdquo;, the answer was almost always \u0026ldquo;Use Celery\u0026rdquo;. And for good reason. Celery is mature, battle-tested, and has solved async task processing for countless Django applications. It has extensive documentation, a large ecosystem of plugins, and most Django developers have at least some experience with it.\nBut Celery also brings complexity. You need a message broker like Redis or RabbitMQ. You need to manage worker processes. You need to implement your own progress tracking if tasks need to report status. You need to handle state persistence separately if your tasks need to remember things across restarts. As your application grows, this infrastructure overhead grows with it.\nI recently rebuilt a Django application that used Celery for handling email tasks, file processing, and data analysis jobs. The application worked fine, but I kept thinking about the infrastructure I was maintaining. Redis for the message broker and results backend. Multiple Celery worker processes. Flower for monitoring. Custom database models for tracking task progress. It was a lot of moving parts for what should be a straightforward problem.\nThat\u0026rsquo;s when I started experimenting with Restate as an alternative approach to async task processing in Django.\nUnderstanding the Architecture Difference Before diving into code, let\u0026rsquo;s understand how these systems differ architecturally.\nCelery uses a message queue pattern. Your Django application puts tasks into a queue, workers pull tasks from the queue, and results go into a backend. State management and progress tracking are your responsibility. You store that information in your database or Redis and query it when needed.\nDjango → Redis Queue → Celery Workers → Results Backend → Database Restate uses a service invocation pattern with built-in state management. Your Django application calls Restate services via HTTP. Restate handles durability, retries, and state persistence internally. The services themselves are stateless, but Restate\u0026rsquo;s virtual objects provide durable state storage.\nDjango → Restate Runtime → Restate Services → Database The fundamental difference is where responsibility for reliability and state lives. In Celery, you build it yourself. In Restate, it\u0026rsquo;s built into the runtime.\nA Typical Celery Task Let\u0026rsquo;s look at how you\u0026rsquo;d implement an email sending task with Celery. This is representative of real production code:\nfrom celery import shared_task from django.conf import settings from .models import EmailTask @shared_task(bind=True) def send_email(self, email_id): try: email_task = EmailTask.objects.get(id=email_id) email_task.task_id = self.request.id email_task.status = \u0026#34;PROGRESS\u0026#34; email_task.save() # Simulate email sending with progress updates for i in range(1, 6): self.update_state( state=\u0026#39;PROGRESS\u0026#39;, meta={ \u0026#39;current\u0026#39;: i, \u0026#39;total\u0026#39;: 5, \u0026#39;status\u0026#39;: f\u0026#39;Step {i}/5: Processing email...\u0026#39; } ) time.sleep(1) # Send the email email_task.status = \u0026#34;SUCCESS\u0026#34; email_task.sent_at = datetime.now() email_task.save() return {\u0026#39;status\u0026#39;: \u0026#39;Email sent successfully\u0026#39;} except Exception as exc: email_task.status = \u0026#34;FAILURE\u0026#34; email_task.save() raise self.retry(exc=exc, countdown=60, max_retries=3) This works, but notice what you\u0026rsquo;re responsible for. You\u0026rsquo;re managing the database model for tracking task state. You\u0026rsquo;re manually updating progress in both Celery\u0026rsquo;s state and your database. You\u0026rsquo;re implementing retry logic. You\u0026rsquo;re handling exceptions and state transitions. This is boilerplate that every task needs.\nThe Same Task with Restate Here\u0026rsquo;s the equivalent task implemented with Restate:\nimport restate from datetime import datetime from typing import Dict, Any email_service = restate.Service(\u0026#34;EmailService\u0026#34;) task_tracker = restate.VirtualObject(\u0026#34;TaskTracker\u0026#34;) @email_service.handler(\u0026#34;send_email\u0026#34;) async def send_email_handler(ctx: restate.Context, task_data: Dict[str, Any]) -\u0026gt; Dict[str, Any]: task_id = task_data.get(\u0026#34;task_id\u0026#34;) to_email = task_data.get(\u0026#34;to_email\u0026#34;) subject = task_data.get(\u0026#34;subject\u0026#34;) message = task_data.get(\u0026#34;message\u0026#34;) # Simulate email sending with progress updates for i in range(1, 6): await ctx.object_client(task_tracker).set_state( f\u0026#34;email_progress_{task_id}\u0026#34;, {\u0026#34;current\u0026#34;: i, \u0026#34;total\u0026#34;: 5, \u0026#34;status\u0026#34;: f\u0026#34;Step {i}/5: Processing email...\u0026#34;} ) await asyncio.sleep(1) # Send the email result = { \u0026#34;status\u0026#34;: \u0026#34;Email sent successfully\u0026#34;, \u0026#34;to\u0026#34;: to_email, \u0026#34;subject\u0026#34;: subject, \u0026#34;completed_at\u0026#34;: datetime.now().isoformat() } await ctx.object_client(task_tracker).set_state( f\u0026#34;email_result_{task_id}\u0026#34;, result ) return result The difference is subtle but significant. There\u0026rsquo;s no explicit exception handling for retries because Restate handles that automatically. Progress tracking happens through virtual objects, which persist state durably without requiring a database model. The code focuses purely on the email sending logic.\nState Management: The Key Difference In the Celery version, if you want to track task progress, you typically create a Django model:\nclass EmailTask(models.Model): task_id = models.CharField(max_length=255, blank=True) to_email = models.EmailField() subject = models.CharField(max_length=200) message = models.TextField() status = models.CharField(max_length=20, default=\u0026#39;PENDING\u0026#39;) sent_at = models.DateTimeField(null=True, blank=True) Then you query this model to check task status. If your worker crashes mid-task, you need recovery logic to detect and handle stale tasks.\nRestate\u0026rsquo;s virtual objects provide durable state without database models. State persists across restarts automatically. If your service crashes mid-execution, Restate resumes exactly where it left off when the service restarts. You don\u0026rsquo;t write recovery logic because the runtime guarantees completion.\nInvoking Tasks from Django Views In Celery, you invoke tasks with .delay() or .apply_async():\nfrom tasks.tasks import send_email def create_email(request): email_task = EmailTask.objects.create( to_email=request.POST[\u0026#39;email\u0026#39;], subject=request.POST[\u0026#39;subject\u0026#39;], message=request.POST[\u0026#39;message\u0026#39;] ) result = send_email.delay(email_task.id) email_task.task_id = result.id email_task.save() return redirect(\u0026#39;email_list\u0026#39;) With Restate, you make HTTP calls to the runtime:\nimport requests RESTATE_ENDPOINT = os.environ.get(\u0026#34;RESTATE_ENDPOINT\u0026#34;, \u0026#34;http://restate:8080\u0026#34;) def create_email(request): email_task = EmailTask.objects.create( to_email=request.POST[\u0026#39;email\u0026#39;], subject=request.POST[\u0026#39;subject\u0026#39;], message=request.POST[\u0026#39;message\u0026#39;] ) response = requests.post( f\u0026#34;{RESTATE_ENDPOINT}/EmailService/send_email\u0026#34;, json={ \u0026#34;task_id\u0026#34;: str(email_task.id), \u0026#34;to_email\u0026#34;: email_task.to_email, \u0026#34;subject\u0026#34;: email_task.subject, \u0026#34;message\u0026#34;: email_task.message } ) return redirect(\u0026#39;email_list\u0026#39;) The Restate approach is more explicit about being a remote call, which some developers prefer. It\u0026rsquo;s clear that you\u0026rsquo;re invoking a separate service rather than a Python function.\nComplex Workflows and Orchestration Where Restate particularly shines is in complex, multi-step workflows. Imagine a file processing pipeline that validates a file, processes its contents, generates a report, and sends notifications.\nIn Celery, you\u0026rsquo;d use chains or chords:\nfrom celery import chain workflow = chain( validate_file.s(file_id), process_contents.s(), generate_report.s(), send_notification.s() ) workflow.apply_async() This works, but error handling across steps is tricky. If step 3 fails, you need custom logic to determine what to retry and what to skip. State tracking across the workflow requires careful planning.\nRestate has built-in workflow support:\nworkflow_orchestrator = restate.Workflow(\u0026#34;WorkflowOrchestrator\u0026#34;) @workflow_orchestrator.main() async def run_file_workflow(ctx: restate.WorkflowContext, workflow_data: Dict[str, Any]) -\u0026gt; Dict[str, Any]: file_id = workflow_data.get(\u0026#34;file_id\u0026#34;) # Each step is durable and resumable validation_result = await ctx.service_client(file_service).validate_file({\u0026#34;file_id\u0026#34;: file_id}) if validation_result[\u0026#34;valid\u0026#34;]: processing_result = await ctx.service_client(file_service).process_contents({\u0026#34;file_id\u0026#34;: file_id}) report_result = await ctx.service_client(report_service).generate_report(processing_result) await ctx.service_client(notification_service).send_notification(report_result) return {\u0026#34;status\u0026#34;: \u0026#34;completed\u0026#34;, \u0026#34;file_id\u0026#34;: file_id} If this workflow crashes at any point, Restate resumes from the last completed step. You don\u0026rsquo;t need to track which steps completed manually. The workflow is just async Python code, making it easier to understand and maintain than Celery\u0026rsquo;s chain/chord abstractions.\nMonitoring and Observability Celery typically uses Flower for monitoring. Flower provides a web interface showing active tasks, completed tasks, and worker status. It\u0026rsquo;s a separate application you need to deploy and maintain.\nRestate includes a dashboard at port 9070 that shows all service invocations, their status, and execution history. It\u0026rsquo;s built into the runtime, so there\u0026rsquo;s nothing extra to deploy.\nFor basic monitoring, Restate\u0026rsquo;s built-in dashboard is often sufficient. For advanced use cases like custom metrics or alerting, both systems require integration with external monitoring tools.\nPerformance Considerations Celery is highly optimized for throughput. If you\u0026rsquo;re processing thousands of simple tasks per second, Celery\u0026rsquo;s message queue architecture is hard to beat. It\u0026rsquo;s been refined over many years to handle high-volume task processing efficiently.\nRestate adds durability guarantees that come with some overhead. Each service invocation is persisted, which ensures reliability but takes time. For tasks where you need absolute guarantees of completion, this overhead is worthwhile. For simple, stateless tasks that can be retried without consequences, Celery might be faster.\nThe trade-off is reliability versus raw throughput. Restate prioritizes reliability. Celery prioritizes throughput but leaves reliability concerns to you.\nInfrastructure Requirements A Celery setup needs:\nMessage broker (Redis or RabbitMQ) Celery worker processes Results backend (typically Redis or your database) Monitoring (Flower) Your Django application A Restate setup needs:\nRestate runtime Restate service process Your Django application Restate has fewer components, but the runtime is a new dependency you might not be familiar with. Celery\u0026rsquo;s components are well-understood by most Django developers.\nWhen to Choose What After rebuilding my Django application with both approaches, here\u0026rsquo;s my perspective on when each makes sense.\nChoose Celery when you need high-throughput task processing and are comfortable managing the infrastructure. If your tasks are relatively simple and you don\u0026rsquo;t need complex state management, Celery\u0026rsquo;s maturity and ecosystem make it a solid choice. If your team already knows Celery, the learning curve for staying with it is zero.\nChoose Restate when you have complex workflows with significant state management needs. If your tasks must complete reliably even through failures, Restate\u0026rsquo;s durability guarantees simplify your code significantly. If you\u0026rsquo;re building something new and can adopt modern patterns, Restate reduces the infrastructure complexity you need to manage.\nFor existing applications already using Celery, migration to Restate requires careful planning. You\u0026rsquo;re changing architecture fundamentally, not just swapping libraries. The benefits are real, but so is the effort of migration.\nRunning Both Implementations I\u0026rsquo;ve created two complete implementations of the same Django application. One uses Celery, the other uses Restate. Both handle email tasks, file processing, and data analysis with progress tracking.\nThe complete source code for both implementations is available on GitHub: django-celery-restate-example\nTo try the Celery version:\ncd django-celery docker-compose up Access the application at http://localhost:8000 and Flower monitoring at http://localhost:5555.\nTo try the Restate version:\ncd django-restate docker-compose up Access the application at http://localhost:8000 and Restate dashboard at http://localhost:9070.\nCompare how the two implementations handle the same features. Look at the task definitions, how progress is tracked, how errors are handled, and how monitoring works. The differences become clear when you see them side by side.\nMigration Strategy If you\u0026rsquo;re considering moving from Celery to Restate, start small. Pick a single task type, implement it with Restate, and run both systems in parallel. Monitor how the Restate version performs and how your team adapts to the different patterns.\nThe service-based architecture of Restate actually makes this parallel operation straightforward. Your Django views can choose to invoke either Celery tasks or Restate services based on feature flags or environment configuration. This lets you migrate incrementally rather than all at once.\nConclusion Celery isn\u0026rsquo;t going away, nor should it. It\u0026rsquo;s a mature, capable system that serves Django applications well. But Restate represents a different philosophy about reliability and state management that\u0026rsquo;s worth understanding.\nCelery says \u0026ldquo;here\u0026rsquo;s a powerful task queue, build your reliability layer on top of it\u0026rdquo;. Restate says \u0026ldquo;reliability and state management are hard problems, we\u0026rsquo;ll handle them so you can focus on your application logic\u0026rdquo;.\nBoth approaches work. The right choice depends on your specific needs, your team\u0026rsquo;s experience, and what trade-offs make sense for your application. Understanding both options helps you make that decision thoughtfully.\nThe complete examples for both implementations are available on GitHub at django-celery-restate-example for you to explore and compare. See how each approach handles the same problems and decide which patterns resonate with how you think about building reliable systems.\n","permalink":"https://joel-hanson.github.io/posts/24-from-celery-to-restate-rethinking-async-tasks-in-django/","summary":"\u003ch2 id=\"the-celery-question\"\u003eThe Celery Question\u003c/h2\u003e\n\u003cp\u003eFor years, when someone asked \u0026ldquo;How do I handle background tasks in Django?\u0026rdquo;, the answer was almost always \u0026ldquo;Use Celery\u0026rdquo;. And for good reason. Celery is mature, battle-tested, and has solved async task processing for countless Django applications. It has extensive documentation, a large ecosystem of plugins, and most Django developers have at least some experience with it.\u003c/p\u003e\n\u003cp\u003eBut Celery also brings complexity. You need a message broker like Redis or RabbitMQ. You need to manage worker processes. You need to implement your own progress tracking if tasks need to report status. You need to handle state persistence separately if your tasks need to remember things across restarts. As your application grows, this infrastructure overhead grows with it.\u003c/p\u003e","title":"From Celery to Restate: Rethinking Async Tasks in Django"},{"content":"The Challenge of Reliable Background Operations Building web applications with Flask is straightforward until you need to handle operations that can fail. Database operations timeout, external APIs become unavailable, and network connections drop. These failures are inevitable in production systems, yet handling them gracefully often requires significant engineering effort.\nRecently, I was trying to build a Todo application where users could create, update, and delete tasks. On the surface, this seems simple. But when you start thinking about production scenarios, questions emerge. What happens if the database connection fails mid-operation? How do you handle retries without duplicating operations? What if the server crashes while processing a delete request?\nI could have built custom retry logic, implemented a job queue, and added manual state tracking. But that would mean writing and maintaining infrastructure code instead of focusing on the application itself. That\u0026rsquo;s when I discovered Restate and its approach to durable execution.\nWhat is Durable Execution? Before diving into the implementation, let\u0026rsquo;s understand what durable execution means. When you mark an operation as durable, the system guarantees it will complete even if failures occur along the way. If your application crashes mid-execution, Restate picks up exactly where it left off once your service restarts. No lost operations, no duplicate executions, no manual recovery logic needed.\nThis is fundamentally different from traditional error handling. Instead of catching exceptions and implementing retry logic throughout your codebase, you declare operations as durable and let Restate handle the complexity of ensuring they complete successfully.\nStarting with a Basic Flask Application Let\u0026rsquo;s look at a standard Flask application that manages todos. Here\u0026rsquo;s how a typical route handles creating a new todo:\nfrom flask import Blueprint, redirect, request, url_for from models.database import db from models.todo import Todo todo_routes = Blueprint(\u0026#34;todo_routes\u0026#34;, __name__) @todo_routes.route(\u0026#34;/add\u0026#34;, methods=[\u0026#34;POST\u0026#34;]) def add_todo(): task = request.form.get(\u0026#34;task\u0026#34;) if task: new_todo = Todo(task=task) db.session.add(new_todo) db.session.commit() return redirect(url_for(\u0026#34;todo_routes.index\u0026#34;)) This code works fine in development, but in production, the db.session.commit() call can fail for numerous reasons. You could wrap it in a try-except block, implement retry logic, and add logging. But this approach has limitations. Each route needs its own error handling, retry policies must be carefully tuned, and you\u0026rsquo;re still responsible for tracking operation state across failures.\nIntroducing Restate to the Architecture Rather than modifying every route with error handling code, we can introduce Restate as a reliability layer. The architecture becomes:\nFlask Routes → Restate Services → Database Operations Your Flask routes remain clean and focused on HTTP handling. Restate services encapsulate the actual database operations with built-in durability and retry capabilities.\nCreating Restate Services First, let\u0026rsquo;s define a Restate service that handles todo operations. Create a new file for your Restate services:\nimport restate from datetime import timedelta from dataclasses import asdict from pydantic import BaseModel from models.todo import Todo from services.flask_app import app, db todo_service = restate.Service(\u0026#34;TodoService\u0026#34;) class TodoRequest(BaseModel): task: str @todo_service.handler() async def add_todo(ctx: restate.Context, req: TodoRequest) -\u0026gt; dict: def db_operation(): with app.app_context(): todo = Todo(task=req.task) db.session.add(todo) db.session.commit() return asdict(todo) await ctx.run_typed(\u0026#34;db_add\u0026#34;, db_operation) return {\u0026#34;message\u0026#34;: \u0026#34;Todo added successfully\u0026#34;} Notice how the database operation is wrapped in ctx.run_typed(). This is where Restate\u0026rsquo;s durability kicks in. The operation becomes atomic and recoverable. If it fails, Restate automatically retries with exponential backoff. If your application crashes, Restate persists the execution state and resumes when your service comes back online.\nConfiguring Retry Policies Different operations have different reliability requirements. For a delete operation, you might want more aggressive retry behavior:\nclass TodoItemRequest(BaseModel): todo_id: int @todo_service.handler() async def delete_todo(ctx: restate.Context, req: TodoItemRequest) -\u0026gt; None: def db_operation(): with app.app_context(): todo = db.session.get(Todo, req.todo_id) if todo is None: raise ValueError(f\u0026#34;Todo with id {req.todo_id} not found\u0026#34;) db.session.delete(todo) db.session.commit() await ctx.run_typed( \u0026#34;db_delete\u0026#34;, db_operation, restate.RunOptions(max_attempts=3, max_retry_duration=timedelta(seconds=30)) ) The RunOptions parameter lets you control exactly how Restate handles failures for this specific operation. You can set maximum retry attempts, retry duration, and other policies without cluttering your business logic with retry code.\nUpdating Flask Routes Now update your Flask routes to call Restate services instead of directly manipulating the database:\nimport requests import os RESTATE_ENDPOINT = os.environ.get(\u0026#34;RESTATE_ENDPOINT\u0026#34;, \u0026#34;http://restate:8080\u0026#34;) @todo_routes.route(\u0026#34;/add\u0026#34;, methods=[\u0026#34;POST\u0026#34;]) def add_todo(): task = request.form.get(\u0026#34;task\u0026#34;) if task: response = requests.post( f\u0026#34;{RESTATE_ENDPOINT}/TodoService/add_todo\u0026#34;, json={\u0026#34;task\u0026#34;: task} ) return redirect(url_for(\u0026#34;todo_routes.index\u0026#34;)) The route is now simpler and focused solely on HTTP concerns. It receives the request, forwards it to Restate, and returns a response. All reliability concerns are handled by Restate.\nSetting Up the Restate Server You need a separate server process to host your Restate services. Create a restate_server.py file:\nimport asyncio import restate from hypercorn.asyncio import serve from hypercorn.config import Config from services.todo_service import todo_service app = restate.app(services=[todo_service]) def main(): config = Config() config.bind = [\u0026#34;0.0.0.0:9080\u0026#34;] asyncio.run(serve(app, config)) if __name__ == \u0026#34;__main__\u0026#34;: main() This creates a separate service that runs alongside your Flask application. The Restate runtime sits between them, handling invocations, managing state, and ensuring durability.\nDocker Compose Configuration The complete system involves three main components: your Flask application, the Restate service server, and the Restate runtime. Here\u0026rsquo;s how they work together in Docker Compose:\nservices: web: build: . ports: - \u0026#34;5050:5050\u0026#34; environment: - DATABASE_URL=postgresql://postgres:postgres@db:5432/todos - RESTATE_ENDPOINT=http://restate:8080 command: python src/app.py depends_on: - db - restate restate_app: build: . environment: - DATABASE_URL=postgresql://postgres:postgres@db:5432/todos command: python src/restate_server.py ports: - \u0026#34;9080:9080\u0026#34; depends_on: - db restate: image: docker.restate.dev/restatedev/restate:latest ports: - \u0026#34;8080:8080\u0026#34; - \u0026#34;9070:9070\u0026#34; depends_on: - restate_app db: image: postgres:15 environment: - POSTGRES_PASSWORD=postgres - POSTGRES_DB=todos ports: - \u0026#34;5432:5432\u0026#34; The restate service is the runtime that manages executions. Your Flask app (web) sends requests to it, and it forwards them to your Restate services (restate_app) with all the durability guarantees.\nObservability and Monitoring One of Restate\u0026rsquo;s strengths is its built-in observability. The runtime includes a dashboard accessible at http://localhost:9070 that shows all service invocations, their current state, and execution history.\nWhen an operation fails and retries, you can see exactly what happened in the dashboard. When your application crashes and resumes, you can track which operations were in-flight and how they completed. This visibility is invaluable for debugging production issues.\nPerformance Considerations Adding Restate introduces an extra network hop between your Flask application and your database operations. For most applications, this latency is negligible compared to the database operation itself. However, you should consider this trade-off.\nThe benefit is that Restate handles durability, retries, and state management that you would otherwise implement manually. For operations where reliability is critical, this is almost always a worthwhile trade-off. For simple read operations that don\u0026rsquo;t require durability, you might keep them as direct database calls in your Flask routes.\nWhen to Use Restate with Flask Restate shines in scenarios where operation reliability matters:\nFinancial Transactions: Operations that must complete exactly once, even if systems fail.\nExternal API Calls: When your Flask application integrates with third-party services that might be unreliable.\nMulti-Step Workflows: Complex operations that span multiple services or databases.\nBackground Jobs: Long-running tasks that need to survive application restarts.\nFor simple CRUD applications with low reliability requirements, Restate might be overkill. But for production systems where failures are costly, it provides a robust foundation for reliable operations.\nGetting Started I\u0026rsquo;ve created a complete example with two implementations: a basic Flask application and the same application enhanced with Restate. The repository includes full Docker Compose configurations for both versions, making it easy to compare the approaches side by side.\nThe complete source code for both implementations is available on GitHub: flask-restate-example\nTo try it yourself:\n# Basic Flask version cd flask-todo docker-compose up # Flask with Restate version cd flask-todo-restate docker-compose up The basic version demonstrates traditional Flask development with direct database operations. The Restate version shows how to add durability without changing your core application logic. Compare the two implementations to see how Restate changes the architecture and where the reliability benefits emerge.\nConclusion Building reliable systems is hard. Restate doesn\u0026rsquo;t make the hard parts disappear, but it consolidates them into a well-tested runtime rather than scattering reliability concerns throughout your application code.\nYour Flask routes stay focused on HTTP handling. Your business logic stays focused on application concerns. And Restate handles the complex problems of durability, retries, and state management.\nThis separation of concerns makes your code easier to understand, test, and maintain. More importantly, it makes your application more reliable in production without requiring you to become an expert in distributed systems.\nThe complete example implementations are available on GitHub at flask-restate-example for you to explore and experiment with.\n","permalink":"https://joel-hanson.github.io/posts/23-adding-durable-execution-to-flask-with-restate/","summary":"\u003ch2 id=\"the-challenge-of-reliable-background-operations\"\u003eThe Challenge of Reliable Background Operations\u003c/h2\u003e\n\u003cp\u003eBuilding web applications with Flask is straightforward until you need to handle operations that can fail. Database operations timeout, external APIs become unavailable, and network connections drop. These failures are inevitable in production systems, yet handling them gracefully often requires significant engineering effort.\u003c/p\u003e\n\u003cp\u003eRecently, I was trying to build a Todo application where users could create, update, and delete tasks. On the surface, this seems simple. But when you start thinking about production scenarios, questions emerge. What happens if the database connection fails mid-operation? How do you handle retries without duplicating operations? What if the server crashes while processing a delete request?\u003c/p\u003e","title":"Adding Durable Execution to Flask Applications with Restate"},{"content":"The Problem: Connecting AI Assistants to Your APIs As AI assistants like Claude and VS Code Copilot become more powerful, developers are discovering a critical bottleneck: these AI tools can\u0026rsquo;t directly interact with our existing REST APIs.\nAfter building Django DRF to MCP converters and Kafka MCP servers, I realized a fundamental challenge: Every API requires custom MCP server implementation.\nYou shouldn\u0026rsquo;t need to write custom code for every API integration. You shouldn\u0026rsquo;t have to learn Model Context Protocol\u0026rsquo;s internals just to connect Claude to your weather API, GitHub API, or internal company APIs.\nThat\u0026rsquo;s when ContextLayer was born.\nWhat is ContextLayer? ContextLayer is an open-source platform that transforms any REST API into a Model Context Protocol (MCP) server through a visual, web-based interface. No coding required. No API modifications needed. No MCP protocol knowledge necessary.\nThink of ContextLayer as a universal translator between REST APIs and AI assistants:\n┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ AI Assistant │───▶│ ContextLayer │───▶│ REST API │ │ (Claude, etc) │ │ MCP Bridge │ │ (Any API) │ └─────────────────┘ └─────────────────┘ └─────────────────┘ ContextLayer\u0026rsquo;s web interface - transform any REST API into an MCP server through a visual dashboard\nThe Core Innovation ContextLayer bridges the gap between two worlds:\nREST APIs: The standard format used by 99% of web APIs Model Context Protocol: The protocol that AI assistants use to interact with external tools Instead of requiring developers to manually create MCP servers for each API, ContextLayer provides:\nVisual Configuration: Web-based dashboard to configure your API bridges Automatic Tool Generation: Converts REST endpoints into MCP tools automatically Multiple Authentication Methods: Supports Bearer tokens, API keys, OAuth, and Basic auth OpenAPI Integration: Import existing OpenAPI/Swagger specs for instant setup Zero Code Changes: Your existing APIs remain completely unchanged Why ContextLayer Exists The Real-World Problem Imagine you want Claude to help you manage your GitHub repositories, check weather data, or interact with your company\u0026rsquo;s internal APIs. Currently, you\u0026rsquo;d need to:\nLearn the Model Context Protocol specification Write custom MCP server code for each API Handle authentication, error handling, and protocol compliance Maintain separate codebases for each integration Debug protocol-level issues when things break This is too much work for something that should be simple.\nThe ContextLayer Solution With ContextLayer, you:\nSign up for a free account (or host it yourself) Point ContextLayer at your REST API (or upload an OpenAPI spec) Configure authentication if needed Click \u0026ldquo;Start Bridge\u0026rdquo; Use your API with Claude immediately That\u0026rsquo;s it. Five minutes from signup to AI-powered API interaction.\nHow ContextLayer Works Architecture Overview ContextLayer consists of three main components:\nWeb Dashboard: Visual interface for creating and managing API bridges Bridge Engine: Transforms REST API calls into MCP-compatible tools MCP Server: Protocol-compliant endpoint that AI assistants connect to The Bridge Creation Process Step 1: Define Your API Creating a new MCP server bridge in ContextLayer is straightforward through the visual interface:\nThe bridge creation interface - define your API name, description, and base URL\nSimply provide:\nBridge Name: A descriptive name for your MCP server Description: What the API does and its purpose Base URL: The root URL of your REST API OpenAPI Spec (optional): Upload an existing OpenAPI/Swagger specification for automatic configuration Step 2: Configure Authentication ContextLayer supports multiple authentication methods through an intuitive interface:\nConfigure authentication for your API - supports Bearer tokens, API keys, Basic auth, and more\nSupported authentication types:\nNo Authentication: For public APIs Bearer Token: Token-based authentication API Key: Header or query parameter-based authentication Basic Authentication: Username and password authentication OAuth 2.0: Coming soon for advanced authorization flows All credentials are encrypted and securely stored.\nStep 3: Automatic MCP Tool Generation ContextLayer automatically converts your REST endpoints into MCP tools through the visual tool editor:\nDefine and edit MCP tools - specify endpoint paths, methods, parameters, and request bodies\nFor each endpoint, you can configure:\nTool Name: The name AI assistants will use to call this tool Description: What the tool does (helps AI understand when to use it) HTTP Method: GET, POST, PUT, DELETE, PATCH Endpoint Path: The API path with parameter placeholders Parameters: Path, query, and header parameters with types and descriptions Request Body: JSON schema for POST/PUT requests ContextLayer generates the complete MCP tool schema automatically.\nStep 4: Connect to AI Assistants Once your bridge is configured, you can see all your MCP servers in the dashboard:\nView and manage all your MCP server bridges - start, stop, and get connection URLs\nTo connect to Claude Desktop, copy the MCP server URL and add it to your Claude configuration:\n{ \u0026#34;mcpServers\u0026#34;: { \u0026#34;your-api-name\u0026#34;: { \u0026#34;command\u0026#34;: \u0026#34;npx\u0026#34;, \u0026#34;args\u0026#34;: [\u0026#34;-y\u0026#34;, \u0026#34;mcp-remote\u0026#34;, \u0026#34;https://contextlayer.tech/mcp/your-bridge-id\u0026#34;] } } } Save this configuration in:\nmacOS: ~/Library/Application Support/Claude/claude_desktop_config.json Windows: %APPDATA%/Claude/claude_desktop_config.json Now you can ask Claude:\n\u0026ldquo;What\u0026rsquo;s the current weather in New York?\u0026rdquo;\nClaude will automatically use your ContextLayer MCP Server to call the weather API and return the results.\nReal-World Use Cases 1. GitHub API Integration Transform GitHub\u0026rsquo;s REST API into MCP tools for AI-assisted repository management:\n# Before ContextLayer: Write custom MCP server code # After ContextLayer: Configure in 2 minutes # You can now ask Claude: # \u0026#34;Show me all open issues in my repository\u0026#34; # \u0026#34;Create a new branch called feature/user-authentication\u0026#34; # \u0026#34;Comment on PR #42 with a review\u0026#34; 2. Weather \u0026amp; Data APIs Connect public APIs like OpenWeatherMap, APIs.gov, or NASA data:\n# Claude can now: # - Check weather forecasts # - Search government databases # - Access real-time data feeds 3. Internal Company APIs Bridge internal APIs that your team uses daily:\n# Connect to your: # - CRM system # - Inventory management # - Customer support tools # - Internal dashboards 4. E-commerce \u0026amp; Business APIs Integrate Stripe, Shopify, or other business-critical APIs:\n# Enable AI-assisted: # - Order management # - Payment processing # - Inventory tracking # - Customer analytics Getting Started with ContextLayer Option 1: Use the Hosted Version Visit contextlayer.tech Sign up with Google (free account) Create your first MCP Server bridge through the visual dashboard Configure your API endpoints and authentication Copy the MCP server URL Connect to Claude Desktop using the provided configuration ContextLayer dashboard - manage all your MCP server bridges in one place\nOption 2: Self-Hosted Deployment For privacy-conscious organizations or custom deployments:\n# Clone the repository git clone https://github.com/Joel-hanson/contextlayer.git cd contextlayer # Start with Docker Compose (includes PostgreSQL) docker-compose up -d # Or run manually npm install npm run db:migrate npm run dev Visit http://localhost:3000 and start creating bridges.\nAdvanced Features OpenAPI/Swagger Import If your API already has an OpenAPI specification, ContextLayer can import it automatically:\n# Upload your OpenAPI spec (JSON or YAML) # ContextLayer will: # - Extract all endpoints # - Parse request/response schemas # - Configure authentication methods # - Generate MCP tools automatically Getting Involved ContextLayer is open source:\nContribute GitHub Repository: github.com/Joel-hanson/contextlayer Issues: Report bugs or request features Pull Requests: Submit improvements Documentation: Help improve guides and examples Pre-built Connectors Templates We\u0026rsquo;re building a library of pre-configured connectors for popular APIs:\nGitHub OpenWeatherMap Slack Stripe Shopify And many more\u0026hellip; Check out existing connectors or contribute your own!\nConclusion ContextLayer solves a real problem: connecting AI assistants to the APIs you already use shouldn\u0026rsquo;t require custom code for every integration.\nWhether you\u0026rsquo;re:\nIndividual developers wanting Claude to help with daily API tasks Small teams integrating internal tools with AI assistants Enterprises needing a scalable solution for API-to-AI integration API providers wanting to offer MCP compatibility without rewriting ContextLayer provides a simple, visual, code-free way to bridge REST APIs and Model Context Protocol.\nTry it today: contextlayer.tech\nSelf-host it: github.com/Joel-hanson/contextlayer\nJoin the community: GitHub Discussions\nRelated Articles API Integration \u0026amp; MCP Convert Your Django DRF Project to MCP in 5 Minutes - Transform Django REST Framework APIs into MCP servers Breaking Kafka Walls: Building an MCP Server for Your Kafka Cluster - Learn how to integrate Kafka with AI assistants using MCP API Development Turn GitHub into Your Free Data Platform: Building APIs with GitHub Actions - Create powerful APIs using GitHub\u0026rsquo;s infrastructure Build Custom Kafka Connectors Fast with This Open Source Template - Streamline your data pipeline development Automation \u0026amp; Productivity How I Automated My Markdown Publishing on Medium: No More Manual Work - Automate your content publishing workflow Useful Git Aliases for a Productive Workflow - Boost your development productivity with Git shortcuts Stay Connected Website: contextlayer.tech GitHub: Joel-hanson/contextlayer Blog: joel-hanson.github.io LinkedIn: Joel-hanson Found ContextLayer useful? Star the repository and share it with your team. Together, we\u0026rsquo;re making AI assistant integration accessible to everyone.\n","permalink":"https://joel-hanson.github.io/posts/22-introducing-contextlayer-transform-any-rest-api-into-an-mcp-server/","summary":"\u003ch2 id=\"the-problem-connecting-ai-assistants-to-your-apis\"\u003eThe Problem: Connecting AI Assistants to Your APIs\u003c/h2\u003e\n\u003cp\u003eAs AI assistants like \u003ca href=\"https://claude.ai/\"\u003eClaude\u003c/a\u003e and \u003ca href=\"https://github.com/features/copilot\"\u003eVS Code Copilot\u003c/a\u003e become more powerful, developers are discovering a critical bottleneck: \u003cstrong\u003ethese AI tools can\u0026rsquo;t directly interact with our existing REST APIs\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eAfter building \u003ca href=\"/posts/20-convert-your-django-drf-project-to-mcp-in-5-minutes/\"\u003eDjango DRF to MCP converters\u003c/a\u003e and \u003ca href=\"/posts/16-breaking-kafka-walls-building-an-mcp-server-for-your-kafka-cluster/\"\u003eKafka MCP servers\u003c/a\u003e, I realized a fundamental challenge: \u003cstrong\u003eEvery API requires custom MCP server implementation\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eYou shouldn\u0026rsquo;t need to write custom code for every API integration. You shouldn\u0026rsquo;t have to learn Model Context Protocol\u0026rsquo;s internals just to connect Claude to your weather API, GitHub API, or internal company APIs.\u003c/p\u003e","title":"Introducing ContextLayer: Transform Any REST API into an MCP Server in Minutes"},{"content":"The Problem: Getting Real-time Data into Kafka Have you ever tried to stream Wikipedia changes or other real-time Server-Sent Events (SSE) updates into Kafka? If you have, you probably wrote custom Kafka producer and consumer code to handle these events, managed reconnections, and built your own error handling.\nThere had to be a better way.\nIntroducing the SSE Kafka Connector I\u0026rsquo;m excited to share my new open-source project: a generic Kafka Connect connector for Server-Sent Events. It lets you stream data from any SSE endpoint straight into Kafka topics—no custom code required.\nGithub Repository What Can You Do With It? Stream Wikipedia Changes { \u0026#34;sse.url\u0026#34;: \u0026#34;https://stream.wikimedia.org/v2/stream/recentchange\u0026#34;, \u0026#34;data.format\u0026#34;: \u0026#34;json\u0026#34;, \u0026#34;topic.strategy\u0026#34;: \u0026#34;static\u0026#34;, \u0026#34;kafka.topic\u0026#34;: \u0026#34;wiki-changes\u0026#34; } Watch page edits, detect vandalism, or track trends across Wikipedia in real-time.\nMonitor GitHub Events { \u0026#34;sse.url\u0026#34;: \u0026#34;https://github-events-proxy.example.com/events\u0026#34;, \u0026#34;sse.headers\u0026#34;: \u0026#34;Authorization:Bearer YOUR_TOKEN\u0026#34;, \u0026#34;topic.strategy\u0026#34;: \u0026#34;field\u0026#34;, \u0026#34;topic.field\u0026#34;: \u0026#34;$.repository.name\u0026#34; } Get notified about new issues, commits, and PRs as they happen.\nTrack Financial Updates { \u0026#34;sse.url\u0026#34;: \u0026#34;https://market-data.example.com/stocks\u0026#34;, \u0026#34;json.key.field\u0026#34;: \u0026#34;$.symbol\u0026#34;, \u0026#34;json.value.fields\u0026#34;: \u0026#34;$.price,$.change\u0026#34;, \u0026#34;kafka.topic\u0026#34;: \u0026#34;stock-updates\u0026#34; } Stream market data for real-time analytics and alerts.\nSee It in Action # Start the Docker environment docker-compose up -d # Deploy the connector with Wikimedia config curl -X POST -H \u0026#34;Content-Type: application/json\u0026#34; \\ --data @config/json/wikimedia-sse-connector.json \\ http://localhost:8083/connectors # Watch the data flow into Kafka docker exec kafka kafka-console-consumer \\ --bootstrap-server kafka:9092 \\ --topic wikimedia-changes --from-beginning How It Works The connector:\nEstablishes a connection to any SSE endpoint Filters events based on what you\u0026rsquo;re interested in Processes the data (JSON, XML, plain text) Routes events to the right Kafka topics Why I Built This I wanted a simple, reusable way to integrate SSE streams with Kafka without reinventing the wheel each time. This connector abstracts away the complexity, letting you focus on building applications instead of plumbing. Also I wanted to tryout real world event streaming usecases with Kafka Connect especially the wikimedia stream which is a great source of real time data.\nTry It Today Ready to simplify your event streaming? Check out the GitHub repository and give it a star if you find it useful!\nQuestions or ideas? Open an issue or contribute to the project. I\u0026rsquo;d love to hear how you\u0026rsquo;re using it.\nStay Connected Blog: https://joel-hanson.github.io/ GitHub: Joel-hanson LinkedIn: Joel-hanson For more Kafka Connect tips and open-source tools, follow the blog series and star the repository.\n","permalink":"https://joel-hanson.github.io/posts/21-sse-kafka-connector/","summary":"\u003ch2 id=\"the-problem-getting-real-time-data-into-kafka\"\u003eThe Problem: Getting Real-time Data into Kafka\u003c/h2\u003e\n\u003cp\u003eHave you ever tried to stream Wikipedia changes or other real-time Server-Sent Events (SSE) updates into Kafka? If you have, you probably wrote custom Kafka producer and consumer code to handle these events, managed reconnections, and built your own error handling.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThere had to be a better way.\u003c/strong\u003e\u003c/p\u003e\n\u003ch2 id=\"introducing-the-sse-kafka-connector\"\u003eIntroducing the SSE Kafka Connector\u003c/h2\u003e\n\u003cp\u003eI\u0026rsquo;m excited to share my new open-source project: a \u003cstrong\u003egeneric Kafka Connect connector for Server-Sent Events\u003c/strong\u003e. It lets you stream data from any SSE endpoint straight into Kafka topics—no custom code required.\u003c/p\u003e","title":"SSE Kafka Connector: Stream Real-time Events into Kafka"},{"content":"The Problem: When APIs Meet AI Assistants A few weeks ago, I was experimenting with Model Context Protocol (MCP) for a Kafka integration project. The experience was eye-opening - being able to interact with complex systems through natural language felt like a glimpse into the future of development workflows.\nAfter successfully getting MCP working with Kafka, I started thinking about other systems I could connect. That\u0026rsquo;s when I looked at one of my old Django projects.\nThe question hit me: \u0026ldquo;What if I could expose my existing Django APIs to Claude through MCP or any other MCP client?\u0026rdquo;\nMy first instinct was to start mapping out individual MCP tools for each endpoint. But as I began sketching out the implementation, I realized I was about to rebuild everything I already had: well-tested ViewSets, comprehensive serializers, validated business logic.\nThen it occurred to me - what if this could be completely automatic? What if any Django developer could add MCP support to their existing project with minimal changes? The more I thought about it, the more convinced I became that this should be a easy solution.\nAfter some experimentation, I built exactly that: a universal bridge that transforms any Django REST Framework project into an MCP-compatible server in under five minutes, without changing a single line of existing code.\nWhat You\u0026rsquo;ll Achieve This comprehensive guide will show you how to add Model Context Protocol support to your existing Django DRF project, enabling AI assistants to:\nInteract with all your models through your existing ViewSets Execute custom business logic you\u0026rsquo;ve defined with @action decorators Understand your data structures automatically from DRF serializers Respect your validation rules using your existing DRF validation logic By the end of this tutorial, you\u0026rsquo;ll have a fully functional MCP server that works with Claude Desktop, VS Code, Warp Terminal, and any other MCP-compatible client.\nPrerequisites This guide assumes you have:\nA Django project with Django REST Framework installed At least one ModelViewSet defined in your application Basic familiarity with Django URL configuration Python 3.8+ and Django 3.2+ installed Here\u0026rsquo;s an example of what we\u0026rsquo;ll be working with:\n# Your existing ViewSet (remains completely unchanged) class UserViewSet(ModelViewSet): queryset = User.objects.all() serializer_class = UserSerializer @action(detail=True, methods=[\u0026#39;post\u0026#39;]) def activate(self, request, pk=None): user = self.get_object() user.is_active = True user.save() return Response({\u0026#39;status\u0026#39;: \u0026#39;activated\u0026#39;}) Step 1: Add the MCP View Download the MCP view class and add it to your Django project:\n# In your Django project root wget https://raw.githubusercontent.com/Joel-hanson/django-drf-mcp/main/mcp_view.py This single file contains everything needed to convert your ViewSets to MCP tools. The implementation is lightweight and follows Django best practices for maximum compatibility.\nNote: You can also manually download the file from the django-drf-mcp repository if you prefer.\nStep 2: Update Your URL Configuration Add the MCP endpoint to your main urls.py:\n# urls.py from django.contrib import admin from django.urls import path, include from mcp_view import MCPView # Add this import urlpatterns = [ path(\u0026#39;admin/\u0026#39;, admin.site.urls), path(\u0026#39;mcp/\u0026#39;, MCPView.as_view(), name=\u0026#39;mcp\u0026#39;), # Add this line path(\u0026#39;api/\u0026#39;, include(\u0026#39;your_app.urls\u0026#39;)), # Your existing URLs ] That\u0026rsquo;s it. No configuration files, no setup scripts, no database changes. The system will automatically discover all your ViewSets.\nStep 3: Test Your MCP Endpoint Start your Django development server and verify the MCP endpoint:\npython manage.py runserver Test the tools discovery:\ncurl -X POST http://127.0.0.1:8000/mcp/ \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -d \u0026#39;{\u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 1, \u0026#34;method\u0026#34;: \u0026#34;tools/list\u0026#34;, \u0026#34;params\u0026#34;: {}}\u0026#39; You should see tools automatically generated for all your ViewSets.\nWhat Gets Auto-Discovered The system automatically finds and converts:\nAll Your ViewSets # These ViewSets automatically become MCP tools: class UserViewSet(ModelViewSet): queryset = User.objects.all() serializer_class = UserSerializer class ProductViewSet(ModelViewSet): queryset = Product.objects.all() serializer_class = ProductSerializer class OrderViewSet(ModelViewSet): queryset = Order.objects.all() serializer_class = OrderSerializer CRUD Operations Each ViewSet automatically gets these MCP tools:\nlist_{app}_{model} - List all objects create_{app}_{model} - Create new object retrieve_{app}_{model} - Get object by ID update_{app}_{model} - Update existing object destroy_{app}_{model} - Delete object Custom Actions Your @action methods become MCP tools automatically:\nclass UserViewSet(ModelViewSet): # ... standard ViewSet code ... @action(detail=True, methods=[\u0026#39;post\u0026#39;]) def activate(self, request, pk=None): # This becomes: activate_users_user pass @action(detail=False) def active_users(self, request): # This becomes: active_users_users_user pass Schema Generation JSON schemas are automatically created from your DRF serializers:\nclass UserSerializer(serializers.ModelSerializer): class Meta: model = User fields = [\u0026#39;username\u0026#39;, \u0026#39;email\u0026#39;, \u0026#39;first_name\u0026#39;, \u0026#39;last_name\u0026#39;] # Automatically generates: # { # \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, # \u0026#34;properties\u0026#34;: { # \u0026#34;username\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;maxLength\u0026#34;: 150}, # \u0026#34;email\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;format\u0026#34;: \u0026#34;email\u0026#34;}, # \u0026#34;first_name\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;maxLength\u0026#34;: 30}, # \u0026#34;last_name\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;maxLength\u0026#34;: 30} # }, # \u0026#34;required\u0026#34;: [\u0026#34;username\u0026#34;, \u0026#34;email\u0026#34;] # } Complete Example: Before and After Let\u0026rsquo;s examine a typical Django application to see exactly what changes:\nBefore: Standard DRF API # models.py class User(models.Model): username = models.CharField(max_length=150, unique=True) email = models.EmailField() is_active = models.BooleanField(default=True) # serializers.py class UserSerializer(serializers.ModelSerializer): class Meta: model = User fields = [\u0026#39;id\u0026#39;, \u0026#39;username\u0026#39;, \u0026#39;email\u0026#39;, \u0026#39;is_active\u0026#39;] # views.py class UserViewSet(ModelViewSet): queryset = User.objects.all() serializer_class = UserSerializer @action(detail=True, methods=[\u0026#39;post\u0026#39;]) def activate(self, request, pk=None): user = self.get_object() user.is_active = True user.save() return Response({\u0026#39;status\u0026#39;: \u0026#39;activated\u0026#39;}) # urls.py router = DefaultRouter() router.register(r\u0026#39;users\u0026#39;, UserViewSet) urlpatterns = [path(\u0026#39;api/\u0026#39;, include(router.urls))] After: Same Code + MCP Support # models.py - UNCHANGED class User(models.Model): username = models.CharField(max_length=150, unique=True) email = models.EmailField() is_active = models.BooleanField(default=True) # serializers.py - UNCHANGED class UserSerializer(serializers.ModelSerializer): class Meta: model = User fields = [\u0026#39;id\u0026#39;, \u0026#39;username\u0026#39;, \u0026#39;email\u0026#39;, \u0026#39;is_active\u0026#39;] # views.py - UNCHANGED class UserViewSet(ModelViewSet): queryset = User.objects.all() serializer_class = UserSerializer @action(detail=True, methods=[\u0026#39;post\u0026#39;]) def activate(self, request, pk=None): user = self.get_object() user.is_active = True user.save() return Response({\u0026#39;status\u0026#39;: \u0026#39;activated\u0026#39;}) # urls.py - ONE LINE ADDED from mcp_view import MCPView # Added router = DefaultRouter() router.register(r\u0026#39;users\u0026#39;, UserViewSet) urlpatterns = [ path(\u0026#39;api/\u0026#39;, include(router.urls)), path(\u0026#39;mcp/\u0026#39;, MCPView.as_view(), name=\u0026#39;mcp\u0026#39;), # Added ] Result: Auto-Generated MCP Tools { \u0026#34;tools\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;list_your_app_user\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;List User in your_app\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;create_your_app_user\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Create User in your_app\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;retrieve_your_app_user\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Retrieve User in your_app\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;update_your_app_user\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Update User in your_app\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;destroy_your_app_user\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Destroy User in your_app\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;activate_your_app_user\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Activate a user\u0026#34; } ] } Step 4: Configure Claude Desktop Create or update your Claude Desktop configuration file:\n{ \u0026#34;mcpServers\u0026#34;: { \u0026#34;my-django-project\u0026#34;: { \u0026#34;command\u0026#34;: \u0026#34;npx\u0026#34;, \u0026#34;args\u0026#34;: [\u0026#34;-y\u0026#34;, \u0026#34;mcp-remote\u0026#34;, \u0026#34;http://127.0.0.1:8000/mcp/\u0026#34;] } } } Save this as claude_desktop_config.json in:\nmacOS: ~/Library/Application Support/Claude/ Windows: %APPDATA%/Claude/ Linux: ~/.config/Claude/ Important Notes:\nInstall mcp-remote first: npm install -g mcp-remote Restart Claude Desktop after configuration changes Ensure your Django server is running on http://127.0.0.1:8000 Make sure your Django project has CORS_ALLOW_ALL_ORIGINS = True in settings for local development For production deployments, consider using django-cors-headers for proper CORS configuration.\nWarp Terminal Configuration Warp Terminal supports MCP through its AI features. Configure it by adding this to your Warp MCP configuration:\n{ \u0026#34;django-project\u0026#34;: { \u0026#34;command\u0026#34;: \u0026#34;npx\u0026#34;, \u0026#34;args\u0026#34;: [\u0026#34;-y\u0026#34;, \u0026#34;mcp-remote\u0026#34;, \u0026#34;http://127.0.0.1:8000/mcp/\u0026#34;], \u0026#34;env\u0026#34;: {}, \u0026#34;working_directory\u0026#34;: null } } VS Code Configuration Configure VS Code to work with your MCP server using the official MCP extension:\nCreate VS Code settings (.vscode/mcp.json):\n{ \u0026#34;servers\u0026#34;: { \u0026#34;django-project\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;http\u0026#34;, \u0026#34;url\u0026#34;: \u0026#34;http://127.0.0.1:8000/mcp/\u0026#34; } } } Alternative: Direct HTTP Configuration For any MCP client that supports HTTP endpoints directly:\n{ \u0026#34;servers\u0026#34;: { \u0026#34;django-project\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;http\u0026#34;, \u0026#34;url\u0026#34;: \u0026#34;http://127.0.0.1:8000/mcp/\u0026#34;, \u0026#34;headers\u0026#34;: { \u0026#34;Content-Type\u0026#34;: \u0026#34;application/json\u0026#34; } } } } Testing Your Configuration Verify your setup works across all clients:\n# Test the endpoint directly curl -X POST http://127.0.0.1:8000/mcp/ \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -d \u0026#39;{\u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 1, \u0026#34;method\u0026#34;: \u0026#34;tools/list\u0026#34;, \u0026#34;params\u0026#34;: {}}\u0026#39; You should see a list of your auto-generated Django tools.\nStep 5: Start Using AI with Your Data Restart Claude Desktop and begin interacting with your Django data through natural language:\nList your data:\n\u0026ldquo;Show me all users in the system\u0026rdquo;\nCreate records:\n\u0026ldquo;Create a new user with username \u0026lsquo;johndoe\u0026rsquo; and email \u0026lsquo;john@example.com\u0026rsquo;\u0026rdquo;\nExecute custom actions:\n\u0026ldquo;Activate the user with ID 5\u0026rdquo;\nComplex operations:\n\u0026ldquo;Show me all inactive users and then activate them one by one\u0026rdquo;\nClaude will automatically use your MCP tools to interact with your Django data while respecting all your existing validation rules and business logic.\nAdvanced Configuration Options ViewSet Compatibility The current implementation works specifically with ModelViewSet classes. Here\u0026rsquo;s the compatibility breakdown:\nView Type Support Status Notes ModelViewSet ✅ Full support Automatic CRUD + custom actions ReadOnlyModelViewSet ⚠️ Partial Only list/retrieve operations ViewSet ❌ Not supported No standard CRUD pattern APIView ❌ Not supported No auto-discovery possible GenericAPIView ❌ Not supported Would need custom implementation Extending Support for Other ViewSet Types You can extend the implementation to support other ViewSet types:\nfrom rest_framework.viewsets import ViewSet, ReadOnlyModelViewSet from mcp_view import CustomMCPView class ExtendedMCPView(CustomMCPView): def get_viewset_classes(self): \u0026#34;\u0026#34;\u0026#34;Support multiple ViewSet types.\u0026#34;\u0026#34;\u0026#34; viewset_classes = [] for app_config in apps.get_app_configs(): if app_config.name.startswith(\u0026#34;django.\u0026#34;): continue try: views_module = importlib.import_module(f\u0026#34;{app_config.name}.views\u0026#34;) for attr_name in dir(views_module): attr = getattr(views_module, attr_name) if inspect.isclass(attr): # Support ModelViewSet (full CRUD) if issubclass(attr, ModelViewSet) and attr != ModelViewSet: viewset_classes.append((attr, app_config.name, \u0026#39;full\u0026#39;)) # Support ReadOnlyModelViewSet (list/retrieve only) elif issubclass(attr, ReadOnlyModelViewSet) and attr != ReadOnlyModelViewSet: viewset_classes.append((attr, app_config.name, \u0026#39;readonly\u0026#39;)) # Support custom ViewSet with @action methods elif issubclass(attr, ViewSet) and attr != ViewSet: viewset_classes.append((attr, app_config.name, \u0026#39;custom\u0026#39;)) except (ImportError, AttributeError): continue return viewset_classes def generate_tools_for_viewset(self, viewset_class, app_name, viewset_type): \u0026#34;\u0026#34;\u0026#34;Generate tools based on ViewSet type.\u0026#34;\u0026#34;\u0026#34; tools = [] if viewset_type == \u0026#39;full\u0026#39;: # Full CRUD for ModelViewSet tools.extend(self.get_crud_tools(viewset_class, app_name)) elif viewset_type == \u0026#39;readonly\u0026#39;: # Only list and retrieve for ReadOnlyModelViewSet tools.extend(self.get_readonly_tools(viewset_class, app_name)) elif viewset_type == \u0026#39;custom\u0026#39;: # Only @action methods for custom ViewSet tools.extend(self.get_action_tools(viewset_class, app_name)) return tools Working with APIView Classes For APIView classes, you\u0026rsquo;d need a different approach since they don\u0026rsquo;t follow standard patterns:\nclass APIViewMCPView(CustomMCPView): def get_custom_tools(self): \u0026#34;\u0026#34;\u0026#34;Manually define tools for APIView endpoints.\u0026#34;\u0026#34;\u0026#34; return [ { \u0026#34;name\u0026#34;: \u0026#34;custom_api_endpoint\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Call a custom API endpoint\u0026#34;, \u0026#34;inputSchema\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;data\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Request data\u0026#34;} } } } ] def handle_custom_tool(self, tool_name: str, arguments: dict) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;Handle custom APIView calls.\u0026#34;\u0026#34;\u0026#34; if tool_name == \u0026#34;custom_api_endpoint\u0026#34;: # Create a request and call your APIView manually factory = RequestFactory() request = factory.post(\u0026#39;/custom/\u0026#39;, data=arguments.get(\u0026#39;data\u0026#39;, {})) from myapp.views import MyCustomAPIView view = MyCustomAPIView() response = view.post(request) return json.dumps(response.data) Adding Custom Business Logic Tools Extend the system with your own custom tools:\nclass MyProjectMCPView(CustomMCPView): def get_custom_tools(self): return [ { \u0026#34;name\u0026#34;: \u0026#34;user_statistics\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Get user statistics for the platform\u0026#34;, \u0026#34;inputSchema\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;period\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;enum\u0026#34;: [\u0026#34;day\u0026#34;, \u0026#34;week\u0026#34;, \u0026#34;month\u0026#34;], \u0026#34;description\u0026#34;: \u0026#34;Statistics period\u0026#34; } } } } ] def handle_custom_tool(self, tool_name: str, arguments: dict) -\u0026gt; str: if tool_name == \u0026#34;user_statistics\u0026#34;: period = arguments.get(\u0026#34;period\u0026#34;, \u0026#34;month\u0026#34;) active_users = User.objects.filter(is_active=True).count() total_users = User.objects.count() return f\u0026#34;Statistics for {period}: {active_users}/{total_users} users active\u0026#34; Common Use Cases Content Management Systems \u0026ldquo;Show me all blog posts published this month\u0026rdquo; \u0026ldquo;Create a new blog post with title \u0026lsquo;Django MCP Integration\u0026rsquo;\u0026rdquo;\nSimilar to how we automated Markdown publishing workflows, this approach streamlines content management through AI.\nData Analysis and Reporting \u0026ldquo;How many orders were placed yesterday?\u0026rdquo; \u0026ldquo;Show me users who haven\u0026rsquo;t logged in for 30 days\u0026rdquo;\nFor complex data processing tasks, consider combining this with real-time processing solutions or RAG systems for enhanced analytics.\nAdministrative Tasks \u0026ldquo;Deactivate all users with invalid email addresses\u0026rdquo; \u0026ldquo;Generate a report of recent product orders\u0026rdquo;\nCustomer Support Operations \u0026ldquo;Find user with email john@example.com\u0026rdquo; \u0026ldquo;Show all orders for user ID 123\u0026rdquo;\nTroubleshooting Common Issues Tool Not Found Error Error: Tool \u0026#39;list_myapp_user\u0026#39; not found Solution: Verify that your ViewSet inherits from ModelViewSet and has a queryset attribute properly defined.\nSchema Validation Error Error: Required field \u0026#39;username\u0026#39; missing Solution: Your DRF serializer validation rules are working correctly. Ensure all required fields are provided when creating or updating records.\nImport Error ImportError: cannot import name \u0026#39;MCPView\u0026#39; Solution: Confirm that mcp_view.py is in your project root directory or accessible in your Python path.\nProduction Considerations Performance Optimization Caching: ViewSet discovery occurs once at startup, minimizing runtime overhead Security: Consider adding authentication middleware if your MCP endpoint needs access control. Learn more about Django authentication Rate Limiting: Implement rate limiting for production APIs to prevent abuse using django-ratelimit Database Optimization: Use select_related() and prefetch_related() in your ViewSets for better query performance Scaling for Large Projects # For projects with many ViewSets, consider selective exposure class OptimizedMCPView(CustomMCPView): def get_viewset_classes(self): # Only expose ViewSets you actually want as MCP tools from myapp.views import UserViewSet, ProductViewSet return [ (UserViewSet, \u0026#39;myapp\u0026#39;), (ProductViewSet, \u0026#39;myapp\u0026#39;), ] Why This Approach Works Zero Breaking Changes Your existing DRF APIs continue to function exactly as before. MCP support is purely additive, requiring no modifications to your existing codebase.\nLeverages Existing Infrastructure Uses your existing models, serializers, and ViewSets without modification Respects your validation rules and business logic Works seamlessly with your existing authentication and permission systems Automatic Scaling Adding new ViewSets automatically creates new MCP tools Custom actions become MCP tools instantly No configuration files to maintain or update Production Ready Type-safe implementation with comprehensive error handling Follows Django and DRF best practices Minimal performance impact on your existing APIs Developer Experience Transformation Before MCP Integration # Manual API testing required multiple steps curl -X POST http://localhost:8000/api/users/ \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -d \u0026#39;{\u0026#34;username\u0026#34;:\u0026#34;john\u0026#34;,\u0026#34;email\u0026#34;:\u0026#34;john@example.com\u0026#34;}\u0026#39; After MCP Integration Developer: \u0026ldquo;Create a user named John with email john@example.com\u0026rdquo;\nClaude: \u0026ldquo;I\u0026rsquo;ll create that user for you right now.\u0026rdquo;\nUser created successfully with ID 42\nComplex Operations Comparison Before (Multiple API calls):\ncurl http://localhost:8000/api/users/ curl -X POST http://localhost:8000/api/users/5/activate/ curl http://localhost:8000/api/orders/?user=5 After (Natural language):\nDeveloper: \u0026ldquo;Show me user 5, activate them, then show their orders\u0026rdquo;\nClaude: \u0026ldquo;Here\u0026rsquo;s user 5\u0026rsquo;s details\u0026hellip; Now activating them\u0026hellip; Here are their 3 orders\u0026hellip;\u0026rdquo;\nReal Project Examples E-commerce Platform # Existing ViewSets automatically become: # - list_products_product # - create_orders_order # - update_inventory_item # - activate_users_user # AI can now: # \u0026#34;Show me low-stock products\u0026#34; # \u0026#34;Create an order for user@example.com with product ID 123\u0026#34; # \u0026#34;Find all orders from the last week\u0026#34; Content Management System # Existing ViewSets: # - BlogPostViewSet # - CategoryViewSet # - CommentViewSet # AI workflows: # \u0026#34;Create a blog post about Django MCP integration\u0026#34; # \u0026#34;Show me all unpublished posts from this month\u0026#34; # \u0026#34;Moderate comments on post \u0026#39;MCP Tutorial\u0026#39;\u0026#34; Customer Support Platform # Existing ViewSets: # - TicketViewSet # - CustomerViewSet # - AgentViewSet # AI assistance: # \u0026#34;Show me all open tickets assigned to John\u0026#34; # \u0026#34;Create a high-priority ticket for customer@email.com\u0026#34; # \u0026#34;Find tickets about billing issues\u0026#34; Resources \u0026amp; Links Complete Implementation: django-drf-mcp repository MCP View Source: mcp_view.py Usage Examples: mcp_usage_example.py Model Context Protocol: Official MCP Documentation Django REST Framework: Official DRF Documentation Claude AI: Claude Desktop Application Related Tools and Libraries mcp-remote: NPM Package for HTTP MCP Servers django-cors-headers: CORS Handling for Django django-ratelimit: Rate Limiting for Django Conclusion Converting your Django DRF project to support MCP takes less than 5 minutes and opens up powerful AI-assisted workflows. You don\u0026rsquo;t need to change any existing code - just add one file and one URL pattern.\nThe beauty of this approach is that it:\nRespects your existing architecture Scales with your project growth Requires zero maintenance Enables powerful AI workflows Whether you\u0026rsquo;re managing users, processing orders, or handling content, AI assistants can now interact with your Django data using natural language - while respecting all your existing business logic and validation rules.\nThis integration represents a significant step forward in making APIs more accessible and user-friendly through natural language interfaces. As AI continues to evolve, having your Django applications MCP-ready positions you at the forefront of this technological shift.\nReady to give your Django project AI superpowers? Download the MCP view and let me know how it transforms your development workflow!\nRelated Articles AI \u0026amp; Development Integration Breaking Kafka Walls: Building an MCP Server for Your Kafka Cluster - Learn how to integrate Kafka with AI assistants using MCP Build Real-Time Web Apps with Minimal Code - Discover modern approaches to building responsive applications API Development \u0026amp; Integration Build Custom Kafka Connectors Fast with This Open Source Template - Streamline your data pipeline development Turn GitHub into Your Free Data Platform: Building APIs with GitHub Actions - Create powerful APIs using GitHub\u0026rsquo;s infrastructure Automation \u0026amp; Productivity How I Automated My Markdown Publishing on Medium: No More Manual Work - Automate your content publishing workflow Useful Git Aliases for a Productive Workflow - Boost your development productivity with Git shortcuts Data Engineering \u0026amp; Processing Setting Up RAG with PostgreSQL and pgvector - Build intelligent data retrieval systems Deduplication for Kafka Using Flink - Handle duplicate data in real-time streaming Stay Connected Blog: https://joel-hanson.github.io/ GitHub: Joel-hanson LinkedIn: Joel-hanson Found this tutorial helpful? Share it with your fellow developers and help them supercharge their Django projects with AI capabilities!\n","permalink":"https://joel-hanson.github.io/posts/20-convert-your-django-drf-project-to-mcp-in-5-minutes/","summary":"\u003ch2 id=\"the-problem-when-apis-meet-ai-assistants\"\u003eThe Problem: When APIs Meet AI Assistants\u003c/h2\u003e\n\u003cp\u003eA few weeks ago, I was experimenting with \u003ca href=\"https://modelcontextprotocol.io/\"\u003eModel Context Protocol (MCP)\u003c/a\u003e for a \u003ca href=\"/posts/16-breaking-kafka-walls-building-an-mcp-server-for-your-kafka-cluster/\"\u003eKafka integration project\u003c/a\u003e. The experience was eye-opening - being able to interact with complex systems through natural language felt like a glimpse into the future of development workflows.\u003c/p\u003e\n\u003cp\u003eAfter successfully getting MCP working with Kafka, I started thinking about other systems I could connect. That\u0026rsquo;s when I looked at one of my old \u003ca href=\"https://www.djangoproject.com/\"\u003eDjango\u003c/a\u003e projects.\u003c/p\u003e","title":"Convert Your Django DRF Project to MCP in 5 Minutes"},{"content":"What if I told you that you could build, host, and maintain data APIs completely free using just GitHub? Here\u0026rsquo;s how I built a production-ready data platform without spending a penny on hosting.\nThe Problem: APIs Are Expensive (Or Are They?) Building data-driven applications usually means expensive cloud hosting, database costs, and server maintenance. Most developers assume they need AWS, Google Cloud, or Azure to run automated data collection and serve APIs. But what if there was a better way?\nWhen I built my Antarctic Iceberg Tracker, I discovered that GitHub offers everything needed for a complete data platform:\nFree automation with GitHub Actions (2,000 minutes/month) Free hosting via GitHub Pages Free CDN through raw.githubusercontent.com URLs Free storage in repositories (up to 1GB per repo) That\u0026rsquo;s when I realized GitHub isn\u0026rsquo;t just for code—it\u0026rsquo;s a powerful data platform.\nThe GitHub-as-a-Platform Architecture Instead of traditional cloud infrastructure, this approach uses GitHub\u0026rsquo;s native features as building blocks:\nData Collection Layer GitHub Actions serves as your cron job scheduler and processing engine. It can run Python scripts, install dependencies, and execute complex data workflows on Ubuntu virtual machines.\nData Storage Layer Your repository becomes the database. JSON files, CSVs, or any structured data format can be stored and versioned automatically. Every data update creates a commit, giving you complete audit trails.\nAPI Layer Raw GitHub URLs become your API endpoints. Files like api/latest.json are instantly accessible via https://raw.githubusercontent.com/\u0026lt;USERNAME\u0026gt;/\u0026lt;REPO\u0026gt;/main/api/latest.json with built-in CDN caching.\nFrontend Layer GitHub Pages hosts your interactive dashboards, documentation, and user interfaces. No separate hosting needed.\nReal-World Example: Weather Forecast to API in 30 Minutes Let me show you exactly how to transform a weather website into a public API using this approach. We\u0026rsquo;ll scrape weather data and serve it as a clean JSON API:\nStep 1: Repository Structure weather-api-project/ ├── .github/workflows/ │ └── hourly-weather.yml ├── src/ │ └── weather_scraper.py ├── api/ │ ├── current.json │ └── forecast.json ├── data/ │ └── historical_weather.json └── docs/ └── index.html Step 2: The Weather Scraper import requests import json from datetime import datetime, timedelta from bs4 import BeautifulSoup import re def scrape_weather_data(): \u0026#34;\u0026#34;\u0026#34;Scrape weather data from a public weather service\u0026#34;\u0026#34;\u0026#34; try: # Using OpenWeatherMap\u0026#39;s free tier as an example # You can also scrape weather.com, weather.gov, etc. city = \u0026#34;New York\u0026#34; url = f\u0026#34;https://wttr.in/{city}?format=j1\u0026#34; # JSON weather API response = requests.get(url, timeout=30) response.raise_for_status() weather_data = response.json() # Process and clean the data current_weather = extract_current_weather(weather_data) forecast_data = extract_forecast_data(weather_data) # Save current weather to API endpoint current_api = { \u0026#34;location\u0026#34;: city, \u0026#34;last_updated\u0026#34;: datetime.now().isoformat(), \u0026#34;current\u0026#34;: current_weather } # Save forecast to API endpoint forecast_api = { \u0026#34;location\u0026#34;: city, \u0026#34;last_updated\u0026#34;: datetime.now().isoformat(), \u0026#34;forecast\u0026#34;: forecast_data } # Write to API files with open(\u0026#39;api/current.json\u0026#39;, \u0026#39;w\u0026#39;) as f: json.dump(current_api, f, indent=2) with open(\u0026#39;api/forecast.json\u0026#39;, \u0026#39;w\u0026#39;) as f: json.dump(forecast_api, f, indent=2) print(f\u0026#34;Successfully updated weather data for {city}\u0026#34;) return True except Exception as e: print(f\u0026#34;Error scraping weather data: {e}\u0026#34;) return False def extract_current_weather(data): \u0026#34;\u0026#34;\u0026#34;Extract current weather conditions\u0026#34;\u0026#34;\u0026#34; current = data.get(\u0026#39;current_condition\u0026#39;, [{}])[0] return { \u0026#34;temperature_f\u0026#34;: int(current.get(\u0026#39;temp_F\u0026#39;, 0)), \u0026#34;temperature_c\u0026#34;: int(current.get(\u0026#39;temp_C\u0026#39;, 0)), \u0026#34;condition\u0026#34;: current.get(\u0026#39;weatherDesc\u0026#39;, [{}])[0].get(\u0026#39;value\u0026#39;, \u0026#39;Unknown\u0026#39;), \u0026#34;humidity\u0026#34;: int(current.get(\u0026#39;humidity\u0026#39;, 0)), \u0026#34;wind_speed_mph\u0026#34;: int(current.get(\u0026#39;windspeedMiles\u0026#39;, 0)), \u0026#34;wind_direction\u0026#34;: current.get(\u0026#39;winddir16Point\u0026#39;, \u0026#39;N\u0026#39;), \u0026#34;feels_like_f\u0026#34;: int(current.get(\u0026#39;FeelsLikeF\u0026#39;, 0)), \u0026#34;uv_index\u0026#34;: int(current.get(\u0026#39;uvIndex\u0026#39;, 0)) } def extract_forecast_data(data): \u0026#34;\u0026#34;\u0026#34;Extract 3-day forecast\u0026#34;\u0026#34;\u0026#34; forecast = [] for day_data in data.get(\u0026#39;weather\u0026#39;, [])[:3]: # 3-day forecast date = day_data.get(\u0026#39;date\u0026#39;) forecast.append({ \u0026#34;date\u0026#34;: date, \u0026#34;max_temp_f\u0026#34;: int(day_data.get(\u0026#39;maxtempF\u0026#39;, 0)), \u0026#34;min_temp_f\u0026#34;: int(day_data.get(\u0026#39;mintempF\u0026#39;, 0)), \u0026#34;max_temp_c\u0026#34;: int(day_data.get(\u0026#39;maxtempC\u0026#39;, 0)), \u0026#34;min_temp_c\u0026#34;: int(day_data.get(\u0026#39;mintempC\u0026#39;, 0)), \u0026#34;condition\u0026#34;: day_data.get(\u0026#39;hourly\u0026#39;, [{}])[0].get(\u0026#39;weatherDesc\u0026#39;, [{}])[0].get(\u0026#39;value\u0026#39;, \u0026#39;Unknown\u0026#39;), \u0026#34;chance_of_rain\u0026#34;: int(day_data.get(\u0026#39;hourly\u0026#39;, [{}])[0].get(\u0026#39;chanceofrain\u0026#39;, 0)), \u0026#34;sunrise\u0026#34;: day_data.get(\u0026#39;astronomy\u0026#39;, [{}])[0].get(\u0026#39;sunrise\u0026#39;, \u0026#39;\u0026#39;), \u0026#34;sunset\u0026#34;: day_data.get(\u0026#39;astronomy\u0026#39;, [{}])[0].get(\u0026#39;sunset\u0026#39;, \u0026#39;\u0026#39;) }) return forecast if __name__ == \u0026#34;__main__\u0026#34;: success = scrape_weather_data() exit(0 if success else 1) Step 3: GitHub Actions Automation name: Hourly Weather Update on: schedule: - cron: \u0026#34;0 * * * *\u0026#34; # Every hour workflow_dispatch: # Manual trigger option jobs: scrape-weather: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v4 with: token: ${{ secrets.GITHUB_TOKEN }} - name: Set up Python uses: actions/setup-python@v4 with: python-version: \u0026#34;3.12\u0026#34; - name: Install dependencies run: | python -m pip install --upgrade pip pip install requests beautifulsoup4 lxml - name: Run weather scraper run: python src/weather_scraper.py - name: Commit and push changes run: | git config --local user.email \u0026#34;action@github.com\u0026#34; git config --local user.name \u0026#34;GitHub Action\u0026#34; git add api/ git diff --staged --quiet || git commit -m \u0026#34;Weather update $(date)\u0026#34; git push scrape-and-update: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v4 with: token: ${{ secrets.GITHUB_TOKEN }} - name: Set up Python uses: actions/setup-python@v4 with: python-version: \u0026#34;3.12\u0026#34; - name: Install dependencies run: | python -m pip install --upgrade pip pip install requests beautifulsoup4 lxml - name: Run data scraper run: python src/scraper.py - name: Commit and push changes run: | git config --local user.email \u0026#34;action@github.com\u0026#34; git config --local user.name \u0026#34;GitHub Action\u0026#34; git add . git diff --staged --quiet || git commit -m \u0026#34;Automated data update $(date)\u0026#34; git push Step 4: Instant Weather API Endpoints Once the workflow runs, your weather data becomes instantly accessible:\n// Get current weather fetch( \u0026#34;https://raw.githubusercontent.com/\u0026lt;USERNAME\u0026gt;/weather-api/main/api/current.json\u0026#34; ) .then((response) =\u0026gt; response.json()) .then((data) =\u0026gt; { console.log(`Current temperature: ${data.current.temperature_f}°F`); console.log(`Condition: ${data.current.condition}`); }); // Get 3-day forecast fetch( \u0026#34;https://raw.githubusercontent.com/\u0026lt;USERNAME\u0026gt;/weather-api/main/api/forecast.json\u0026#34; ) .then((response) =\u0026gt; response.json()) .then((data) =\u0026gt; { data.forecast.forEach((day) =\u0026gt; { console.log( `${day.date}: ${day.min_temp_f}-${day.max_temp_f}°F, ${day.condition}` ); }); }); Advanced Patterns for Production Use Error Handling and Reliability import time import random def scrape_with_retry(url, max_retries=3): \u0026#34;\u0026#34;\u0026#34;Robust scraping with exponential backoff\u0026#34;\u0026#34;\u0026#34; for attempt in range(max_retries): try: response = requests.get(url, timeout=30) response.raise_for_status() return response except requests.RequestException as e: if attempt \u0026lt; max_retries - 1: wait_time = (2 ** attempt) + random.uniform(0, 1) print(f\u0026#34;Attempt {attempt + 1} failed, retrying in {wait_time:.1f}s\u0026#34;) time.sleep(wait_time) else: raise e Data Validation and Quality def validate_data(data): \u0026#34;\u0026#34;\u0026#34;Ensure data quality before publishing\u0026#34;\u0026#34;\u0026#34; required_fields = [\u0026#39;id\u0026#39;, \u0026#39;latitude\u0026#39;, \u0026#39;longitude\u0026#39;, \u0026#39;date\u0026#39;] for item in data: # Check required fields if not all(field in item for field in required_fields): raise ValueError(f\u0026#34;Missing required fields in: {item}\u0026#34;) # Validate coordinates lat, lon = item[\u0026#39;latitude\u0026#39;], item[\u0026#39;longitude\u0026#39;] if not (-90 \u0026lt;= lat \u0026lt;= 90) or not (-180 \u0026lt;= lon \u0026lt;= 180): raise ValueError(f\u0026#34;Invalid coordinates: {lat}, {lon}\u0026#34;) return True Smart Caching and Updates def should_update_data(current_data, new_data): \u0026#34;\u0026#34;\u0026#34;Only update if data has actually changed\u0026#34;\u0026#34;\u0026#34; if not current_data: return True # Compare checksums or key metrics current_count = len(current_data.get(\u0026#39;icebergs\u0026#39;, [])) new_count = len(new_data.get(\u0026#39;icebergs\u0026#39;, [])) return current_count != new_count Beyond Basic Scraping: Advanced Use Cases Multi-Source Data Aggregation def aggregate_multiple_sources(): \u0026#34;\u0026#34;\u0026#34;Combine data from multiple APIs/websites\u0026#34;\u0026#34;\u0026#34; sources = [ {\u0026#39;name\u0026#39;: \u0026#39;NASA\u0026#39;, \u0026#39;url\u0026#39;: \u0026#39;https://nasa.gov/api/data\u0026#39;}, {\u0026#39;name\u0026#39;: \u0026#39;NOAA\u0026#39;, \u0026#39;url\u0026#39;: \u0026#39;https://noaa.gov/api/weather\u0026#39;}, {\u0026#39;name\u0026#39;: \u0026#39;USGS\u0026#39;, \u0026#39;url\u0026#39;: \u0026#39;https://usgs.gov/api/earthquakes\u0026#39;} ] aggregated_data = {} for source in sources: try: data = fetch_and_process(source[\u0026#39;url\u0026#39;]) aggregated_data[source[\u0026#39;name\u0026#39;]] = data except Exception as e: print(f\u0026#34;Failed to fetch from {source[\u0026#39;name\u0026#39;]}: {e}\u0026#34;) return aggregated_data Time Series Data Management def manage_historical_data(new_data): \u0026#34;\u0026#34;\u0026#34;Maintain rolling window of historical data\u0026#34;\u0026#34;\u0026#34; try: with open(\u0026#39;data/historical.json\u0026#39;, \u0026#39;r\u0026#39;) as f: history = json.load(f) except FileNotFoundError: history = [] # Add new data with timestamp history.append({ \u0026#39;timestamp\u0026#39;: datetime.now().isoformat(), \u0026#39;data\u0026#39;: new_data }) # Keep only last 30 days cutoff = datetime.now() - timedelta(days=30) history = [ entry for entry in history if datetime.fromisoformat(entry[\u0026#39;timestamp\u0026#39;]) \u0026gt; cutoff ] with open(\u0026#39;data/historical.json\u0026#39;, \u0026#39;w\u0026#39;) as f: json.dump(history, f, indent=2) GitHub Pages: Your Free Frontend Create an interactive dashboard using GitHub Pages:\n\u0026lt;!DOCTYPE html\u0026gt; \u0026lt;html\u0026gt; \u0026lt;head\u0026gt; \u0026lt;title\u0026gt;Data Dashboard\u0026lt;/title\u0026gt; \u0026lt;script src=\u0026#34;https://cdn.jsdelivr.net/npm/chart.js\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; \u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;div id=\u0026#34;dashboard\u0026#34;\u0026gt; \u0026lt;h1\u0026gt;Real-Time Data Dashboard\u0026lt;/h1\u0026gt; \u0026lt;canvas id=\u0026#34;dataChart\u0026#34;\u0026gt;\u0026lt;/canvas\u0026gt; \u0026lt;div id=\u0026#34;stats\u0026#34;\u0026gt;\u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;script\u0026gt; async function loadData() { const response = await fetch( \u0026#34;https://raw.githubusercontent.com/\u0026lt;USERNAME\u0026gt;/\u0026lt;REPO\u0026gt;/main/api/latest.json\u0026#34; ); const data = await response.json(); // Update statistics document.getElementById(\u0026#34;stats\u0026#34;).innerHTML = ` \u0026lt;p\u0026gt;Last Updated: ${new Date( data.last_updated ).toLocaleString()}\u0026lt;/p\u0026gt; \u0026lt;p\u0026gt;Total Items: ${data.total_icebergs}\u0026lt;/p\u0026gt; `; // Create chart const ctx = document.getElementById(\u0026#34;dataChart\u0026#34;).getContext(\u0026#34;2d\u0026#34;); new Chart(ctx, { type: \u0026#34;line\u0026#34;, data: { labels: data.icebergs.map((item) =\u0026gt; item.date), datasets: [ { label: \u0026#34;Data Points\u0026#34;, data: data.icebergs.map((item) =\u0026gt; item.value), }, ], }, }); } loadData(); // Refresh every 5 minutes setInterval(loadData, 5 * 60 * 1000); \u0026lt;/script\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; Real Success Stories: What You Can Build Financial Data APIs Stock market indicators from multiple sources Cryptocurrency price aggregation Economic indicators dashboard Environmental Monitoring Air quality data from government APIs Weather pattern analysis Climate change indicators Social Media Analytics Trending topics aggregation Sentiment analysis pipelines Social media metrics dashboards Government Data Portals Public records aggregation Legislative tracking systems Census data visualization Limitations and When to Graduate GitHub Limits to Consider The free approach works great but has boundaries:\nRepository size: 1GB per repository Actions minutes: 2,000 minutes/month (about 67 minutes/day) API rate limits: 5,000 requests/hour/IP regardless of authentication File size: 100MB maximum per file When to Move to Paid Services Consider upgrading when you need:\nHigh-frequency updates (more than hourly) Large datasets (multi-GB storage requirements) Real-time processing (sub-minute latency) Advanced analytics (machine learning pipelines) Commercial reliability (SLA guarantees) Getting Started: Your First Data Platform 1. Fork the Template I\u0026rsquo;ve created a complete template repository with:\nPre-configured GitHub Actions workflows Example scraper with error handling API generation scripts Basic dashboard template Comprehensive documentation Template Repository: https://github.com/joel-hanson/github-data-platform-template\n2. Customize for Your Data Source git clone https://github.com/joel-hanson/github-data-platform-template.git cd github-data-platform-template # Update src/scraper.py with your data source # Modify .github/workflows/daily-scrape.yml for your schedule # Customize docs/index.html for your dashboard 3. Configure Repository Permissions Before your GitHub Actions can commit changes back to your repository, you need to enable the proper permissions:\nGo to your repository settings Navigate to Actions → General Scroll down to Workflow permissions Select Read and write permissions Click Save This allows your automated workflows to commit the scraped data back to your repository.\n4. Deploy and Monitor git add . git commit -m \u0026#34;Initial setup for my data platform\u0026#34; git push origin main # Enable GitHub Pages in repository settings # Your API will be live at: https://username.github.io/repo-name/ Best Practices for Sustainable Data Platforms Respectful Scraping Ethics Always follow these principles when scraping data:\n# Include proper delays time.sleep(random.uniform(1, 3)) # Random delay between requests # Respect robots.txt import urllib.robotparser rp = urllib.robotparser.RobotFileParser() rp.set_url(\u0026#34;https://example.com/robots.txt\u0026#34;) rp.can_fetch(\u0026#34;*\u0026#34;, url) # Use appropriate headers headers = { \u0026#39;User-Agent\u0026#39;: \u0026#39;Mozilla/5.0 (compatible; DataBot/1.0; +https://github.com/username/repo)\u0026#39;, \u0026#39;Accept\u0026#39;: \u0026#39;text/html,application/json\u0026#39;, } Monitoring and Alerting # Add to your GitHub Actions workflow - name: Notify on failure if: failure() uses: actions/github-script@v6 with: script: | github.rest.issues.create({ owner: context.repo.owner, repo: context.repo.repo, title: \u0026#39;Data scraping failed\u0026#39;, body: \u0026#39;Automated data collection workflow failed. Please check logs.\u0026#39; }) Documentation and Maintenance Create clear documentation for your data platform:\n# API Documentation ## Endpoints ### GET /api/latest.json Returns the most recent dataset **Response Format:** ```json { \u0026#34;last_updated\u0026#34;: \u0026#34;2025-07-26T12:00:00Z\u0026#34;, \u0026#34;total_items\u0026#34;: 97, \u0026#34;data\u0026#34;: [...] } ``` Rate Limits No authentication required Cached for 5 minutes via GitHub CDN Updated daily at 6:00 AM UTC The Economics: Why This Matters Cost Comparison Traditional Cloud Approach:\nAWS Lambda: $0.20 per 1M requests RDS Database: $25/month minimum CloudWatch: $10/month for logging API Gateway: $3.50 per million calls Total: ~$40-100/month for basic setup GitHub Approach:\nGitHub Actions: Free (2,000 minutes/month for private repos) GitHub Pages: Free hosting Raw file CDN: Free bandwidth Version control: Free Total: $0/month For small to medium-scale data projects, the savings are substantial while maintaining professional-grade reliability.\nWhat\u0026rsquo;s Next: Advanced Automation Once you master the basics, consider these advanced patterns:\nMulti-Repository Data Pipelines Source repo for raw data collection Processing repo for data transformation API repo for serving clean endpoints Dashboard repo for visualization Cross-Repository Automation - name: Trigger downstream processing uses: peter-evans/repository-dispatch@v2 with: token: ${{ secrets.PERSONAL_ACCESS_TOKEN }} repository: username/processing-repo event-type: data-updated Webhook Integration # Notify external services when data updates def send_webhook_notification(data): webhook_url = \u0026#34;https://hooks.slack.com/services/...\u0026#34; requests.post(webhook_url, json={ \u0026#34;text\u0026#34;: f\u0026#34;Data updated: {len(data)} new records\u0026#34; }) Conclusion: Your Free Data Empire GitHub\u0026rsquo;s combination of Actions, Pages, and raw file serving creates a surprisingly powerful data platform. You get enterprise-grade infrastructure, global CDN, automatic SSL, and professional reliability—all for free.\nThe approach I\u0026rsquo;ve outlined powers real production systems serving thousands of users. From tracking Antarctic icebergs to monitoring financial markets, the pattern scales beautifully for most data-driven applications.\nReady to build your data platform? Start with the template repository and customize it for your data source. Within an hour, you\u0026rsquo;ll have a fully automated data collection and API serving system running on GitHub\u0026rsquo;s infrastructure.\nData Sources \u0026amp; Credits This approach was inspired by real-world experience building the Antarctic Iceberg Tracker using NASA\u0026rsquo;s publicly available data:\nOriginal Project Antarctic Iceberg Tracker: https://github.com/Joel-hanson/Iceberg-locations Live Demo: https://joel-hanson.github.io/Iceberg-locations/ Template Repositories I\u0026rsquo;ve created complete working examples that demonstrate this approach:\nGitHub Data Platform Template: https://github.com/joel-hanson/github-data-platform-template\nGeneric template with example scraper and dashboard Pre-configured GitHub Actions workflows Complete documentation and setup guide Weather API Project: https://github.com/Joel-hanson/weather-api-project\nReal-world example scraping weather data Hourly automated updates Interactive weather dashboard with charts Ready to transform GitHub into your personal data platform? Fork the template, customize for your data source, and launch your free API in minutes. The power of enterprise-grade data infrastructure is just a git push away.\nFor more Kafka Connect tips and open-source tools, follow the blog series and star the repository, weather API project, and Iceberg Tracker.\n","permalink":"https://joel-hanson.github.io/posts/19-turn-git-hub-into-your-free-data-platform-building-ap-is-with-git-hub-actions/","summary":"\u003cp\u003e\u003cem\u003eWhat if I told you that you could build, host, and maintain data APIs completely free using just GitHub? Here\u0026rsquo;s how I built a production-ready data platform without spending a penny on hosting.\u003c/em\u003e\u003c/p\u003e\n\u003ch2 id=\"the-problem-apis-are-expensive-or-are-they\"\u003eThe Problem: APIs Are Expensive (Or Are They?)\u003c/h2\u003e\n\u003cp\u003eBuilding data-driven applications usually means expensive cloud hosting, database costs, and server maintenance. Most developers assume they need AWS, Google Cloud, or Azure to run automated data collection and serve APIs. But what if there was a better way?\u003c/p\u003e","title":"Turn GitHub into Your Free Data Platform: Building APIs with GitHub Actions"},{"content":"Ever wondered where those massive icebergs the size of cities are drifting? I built a system to track them in real-time using NASA satellite data.\nThe Problem: Lost Giants in the Southern Ocean Antarctic icebergs are some of the most fascinating phenomena on Earth. These floating ice mountains—some larger than entire countries—break off from ice shelves and drift across the Southern Ocean for years. But tracking their movements has always been challenging.\nWhen iceberg A23A (nicknamed the \u0026ldquo;world\u0026rsquo;s largest iceberg\u0026rdquo;) started moving again after being grounded for 30+ years, I realized there wasn\u0026rsquo;t an easy way for curious people like me to follow these giants\u0026rsquo; journeys.\nThat\u0026rsquo;s when I decided to build one.\nTry it out \u0026amp; contribute: GitHub Repo\nThe Original Motivation: Tracking Iceberg A68 My journey began with a specific goal: tracking Iceberg A68, which was once the world\u0026rsquo;s largest iceberg. When A68 calved from the Larsen C ice shelf in 2017, it captured global attention as it began its epic journey across the Southern Ocean.\nThe challenge was that while NASA\u0026rsquo;s Scatterometer Climate Record Pathfinder (SCP) database at BYU provided excellent tracking data, it wasn\u0026rsquo;t easily accessible to developers or the general public. The data existed, but there was no programmatic way to access it or integrate it into applications.\nI wanted to:\nCreate an API that could programmatically access A68\u0026rsquo;s location data Demonstrate web scraping techniques for extracting scientific data Build a reusable system that could track not just A68, but any iceberg in the database Make the data accessible through modern web technologies (JSON APIs, interactive maps) What started as a focused effort to track one remarkable iceberg evolved into a comprehensive system capable of monitoring 97+ icebergs across multiple observation periods. The project became a perfect example of how to transform static scientific data into dynamic, accessible information.\nWhat I Built: A Complete Iceberg Tracking Ecosystem The Antarctic Iceberg Tracker isn\u0026rsquo;t just a simple data scraper—it\u0026rsquo;s a comprehensive system that:\nReal-Time Data Collection The system automatically scrapes NASA\u0026rsquo;s Scatterometer Climate Record Pathfinder (SCP) database, tracking 97+ unique icebergs across 37 observation dates. It processes over 1,800 location records with precise GPS coordinates, ensuring comprehensive coverage of Antarctic iceberg movements.\nInteractive Visualization I built a beautiful web-based map with a professional UI inspired by shadcn/ui design principles. The interface features real-time iceberg positions with color-coded series (A, B, C, D series) and detailed popup information showing coordinates, observation dates, and data freshness indicators.\nMovement Animations Integration with NASA SCP historical movement GIFs Visual drift patterns showing years of iceberg journeys Direct links to animation files for each tracked iceberg API for Developers The system provides RESTful JSON endpoints for external integrations with real-time data access via GitHub raw URLs. It includes JSONP support for cross-origin requests, making it easy for web applications to integrate iceberg tracking data.\nAvailable API Endpoints:\nLatest Data: https://raw.githubusercontent.com/Joel-hanson/Iceberg-locations/main/api/latest.json All Historical Data: https://raw.githubusercontent.com/Joel-hanson/Iceberg-locations/main/iceberg_location.json JSONP Support: https://raw.githubusercontent.com/Joel-hanson/Iceberg-locations/main/api/latest.jsonp The Technical Journey: From Monolith to Modular After the initial version was working, I identified several areas that needed improvement to make the system more robust and maintainable:\nProblem 1: Inconsistent NASA Data Handling The original version had issues with NASA\u0026rsquo;s SCP database because it contains mixed coordinate formats—some in decimal degrees, others in degrees-minutes-seconds (DMS). This caused positioning errors on the map. I updated the system with a smart parser:\ndef convert_coordinates(lat, lon): # Handle both -7545.0 (DMS) and -75.75 (decimal) formats if abs(lat) \u0026gt;= 100: # DMS format detected degrees = int(abs(lat) / 100) minutes = abs(lat) % 100 return -(degrees + minutes/60) if lat \u0026lt; 0 else (degrees + minutes/60) return lat # Already decimal Problem 2: Incorrect Date Ordering The previous version used simple string sorting for dates, which made \u0026ldquo;11/11/21\u0026rdquo; appear \u0026ldquo;newer\u0026rdquo; than \u0026ldquo;01/04/22\u0026rdquo;, showing outdated data as current. I fixed this with proper datetime parsing:\ndef get_latest_date(data): dates_with_dt = [(d, datetime.strptime(d, \u0026#34;%m/%d/%y\u0026#34;)) for d in data.keys()] dates_with_dt.sort(key=lambda x: x[1]) # Sort by actual datetime return dates_with_dt[-1][0] # Return latest date string Automated Data Updates with GitHub Actions One of the most powerful features of this system is its automated data collection pipeline. Instead of manually checking for new iceberg data, the system runs autonomously:\nname: Daily Data Collection on: schedule: - cron: \u0026#34;0 0 * * *\u0026#34; # Every day at midnight jobs: scrape: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v2 - name: Set up Python uses: actions/setup-python@v2 with: python-version: \u0026#34;3.9\u0026#34; - name: Install dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt - name: Run data scraper run: python main.py scrape How It Works The system uses scheduled GitHub Actions to run the scraper daily, automatically checking for new iceberg data from NASA\u0026rsquo;s database. This automated data processing workflow collects, validates, and updates iceberg positions without any manual intervention, then regenerates the interactive map with the latest information.\nDaily Data Collection GitHub Actions automatically runs the scraper every day at a configured time, ensuring the interactive map and API always reflect the latest available NASA data. Once configured, the system operates with zero maintenance required.\nSmart Update Process The automated workflow performs these steps:\nScrapes the NASA SCP database for new iceberg positions Processes and validates the collected data for accuracy Updates JSON APIs with the latest information Regenerates the interactive map with new positions Commits changes back to the repository automatically Reliability Features The system includes comprehensive error handling that gracefully retries if NASA\u0026rsquo;s servers are temporarily unavailable. It validates all data for malformed coordinates or missing information before publishing, maintains previous data if new collection fails, and provides detailed logs to monitor automation health.\nBenefits of Automation This automated approach ensures visitors always see the most recent iceberg positions without delays. The system can easily scale to track hundreds of icebergs without increasing maintenance burden, provides complete transparency through GitHub commit history, and leverages free GitHub Actions instead of costly hosting solutions.\nThis automation transforms a manual data collection task into a reliable, self-maintaining system that keeps iceberg tracking data current 24/7.\nSee It In Action Try the Live Demo Visit the interactive map: View Iceberg Locations\nUse the API // Get latest iceberg data fetch( \u0026#34;https://raw.githubusercontent.com/Joel-hanson/Iceberg-locations/main/api/latest.json\u0026#34; ) .then((response) =\u0026gt; response.json()) .then((data) =\u0026gt; console.log(`Tracking ${data.total_icebergs} icebergs!`)); Install \u0026amp; Run Locally git clone https://github.com/Joel-hanson/Iceberg-locations.git cd Iceberg-locations pip install -r requirement.txt # Generate interactive map python main.py map # Show current stats python main.py info # View animation URLs python main.py animations Data Sources \u0026amp; Credits This project wouldn\u0026rsquo;t be possible without the excellent work of researchers and institutions who make iceberg tracking data publicly available:\nPrimary Data Source NASA Scatterometer Climate Record Pathfinder (SCP) - Hosted by Brigham Young University Website: https://www.scp.byu.edu/ Iceberg Database: https://www.scp.byu.edu/data/iceberg/ A68 Tracking: https://www.scp.byu.edu/data/iceberg/A68tracking.html Research Credit Special thanks to the researchers at BYU\u0026rsquo;s Microwave Earth Remote Sensing (MERS) Laboratory who maintain this valuable dataset and make it freely accessible for research and educational purposes.\nImportant Note This project is built for educational and research purposes. All data is sourced from publicly available NASA/BYU databases. If you use this project or its data, please consider citing the original SCP database in your work.\nInterested in tracking Earth\u0026rsquo;s giants? Fork the repo, contribute improvements, or just explore the fascinating world of Antarctic icebergs through the interactive map!\nFor more Kafka Connect tips and open-source tools, follow the blog series and star the repository.\n","permalink":"https://joel-hanson.github.io/posts/18-tracking-antarctic-giants-building-a-real-time-iceberg-monitor-with-nasa-data/","summary":"\u003cp\u003e\u003cem\u003eEver wondered where those massive icebergs the size of cities are drifting? I built a system to track them in real-time using NASA satellite data.\u003c/em\u003e\u003c/p\u003e\n\u003ch2 id=\"the-problem-lost-giants-in-the-southern-ocean\"\u003eThe Problem: Lost Giants in the Southern Ocean\u003c/h2\u003e\n\u003cp\u003eAntarctic icebergs are some of the most fascinating phenomena on Earth. These floating ice mountains—some larger than entire countries—break off from ice shelves and drift across the Southern Ocean for years. But tracking their movements has always been challenging.\u003c/p\u003e","title":"Tracking Antarctic Giants: Building a Real-Time Iceberg Monitor with NASA Data"},{"content":"🔗 GitHub Repository: github.com/Joel-hanson/kafka-connector-template\n⭐ Star the repo to stay updated!\nIn the first part of this series, we explored how an open-source Kafka Connector Template accelerates connector development. Now, let\u0026rsquo;s tackle a critical next step: writing integration tests to ensure your connector works flawlessly in real-world scenarios. I recently enhanced the template with a testing framework (see PR #1), and here\u0026rsquo;s how you can leverage it.\nBuilding on our Kafka Connector Template, this post dives deep into integration testing - the crucial but often overlooked aspect of connector development.\nWhy Integration Tests Are Critical for Kafka Connectors The mental model for integration testing in Kafka Connect is to validate the end-to-end functionality of your connectors. This means testing how your connector interacts with a real Kafka cluster, ensuring it can produce and consume messages, handle errors, and manage configurations correctly.\nThe whole setup is designed to run integration tests against a real Kafka cluster using TestContainers, which provides a lightweight and isolated environment for testing. This allows you to focus on writing tests that validate your connector\u0026rsquo;s functionality without worrying about the underlying infrastructure.\nTesting Framework Foundation: The integration testing framework in our template is based on the robust testing approach from the Aiven JDBC Connector for Apache Kafka. This proven foundation provides battle-tested patterns for Kafka Connect integration testing, including TestContainers setup, embedded Connect runtime, and comprehensive test utilities. I have adapted and enhanced these patterns to create a reusable testing framework that works with any custom connector.\nThe integration test starts off with the AbstractIT.java class, which serves as the foundation for our integration tests and provides the necessary infrastructure for running tests against a real Kafka cluster. The ConnectRunner.java class is where the magic happens - it creates an embedded Kafka Connect runtime for testing. It also manages the lifecycle of connectors, allowing us to focus on writing tests without worrying about the underlying infrastructure.\nAbstractIT.java - The AbstractIT class is the backbone of our integration testing framework. It provides:\nConnectRunner.java - The ConnectRunner class is where the magic happens - it creates an embedded Kafka Connect runtime for testing. It also manages the lifecycle of connectors, allowing us to focus on writing tests without worrying about the underlying infrastructure.\nThe Reality of Kafka Connect Development Building a Kafka connector isn\u0026rsquo;t just about implementing the SourceTask and SinkTask interfaces. Real-world connectors must handle:\nComplex Data Flows: Data transformations, schema evolution, and error handling Distributed Environments: Multiple tasks, offset management, and fault tolerance Kafka Integration: Topic management, consumer group coordination, and serialization External Systems: Database connections, API calls, and network failures Unit tests alone cannot validate these interactions. You need integration tests that verify your connector works end-to-end with:\nReal Kafka clusters Actual topic creation and message flow Connector lifecycle management Error scenarios and recovery What Integration Tests Catch That Unit Tests Miss // Unit test - tests in isolation @Test void testSinkTaskProcessRecord() { SinkRecord record = createTestRecord(); task.put(Collections.singletonList(record)); // Verify method calls, but not actual Kafka integration } // Integration test - tests real-world scenarios @Test void testSinkConnectorEndToEnd() throws Exception { // Start real Kafka cluster // Create connector with actual configuration getConnectRunner().createConnector(connectorConfig); // Send real messages to Kafka producer.send(new ProducerRecord\u0026lt;\u0026gt;(SINK_TOPIC_NAME, \u0026#34;key\u0026#34;, jsonMessage)); // Verify connector processes messages and interacts with external system await().atMost(Duration.ofSeconds(30)) .untilAsserted(() -\u0026gt; assertThat(externalSystemCalls).isNotEmpty()); } @Test public void testSourceConnectorBasicFunctionality() throws Exception { // Create and start the source connector getConnectRunner().createConnector(connectorConfig); // Wait for connector to produce messages await().atMost(Duration.ofSeconds(30)).pollInterval(Duration.ofSeconds(2)).untilAsserted(() -\u0026gt; { ConsumerRecords\u0026lt;String, String\u0026gt; records = consumeFromSourceTopic(); // Verify that messages were produced assertThat(records.count()).isGreaterThan(0); }); } Understanding Integration Tests in the Kafka Connect Context What Makes Kafka Connect Integration Testing Unique Kafka Connect introduces several layers of complexity:\nConnector Framework: The Connect runtime manages connector lifecycle Distributed Coordination: Multiple workers coordinate through Kafka topics Offset Management: Connector state is stored in Kafka topics Plugin System: Connectors are loaded as plugins with class isolation REST API: Configuration and monitoring through HTTP endpoints Integration Test Hierarchy ┌─────────────────────────────────────────┐ │ Integration Tests │ ├─────────────────────────────────────────┤ │ • End-to-end connector behavior │ │ • Real Kafka cluster interaction │ │ • Configuration validation │ │ • Error handling and recovery │ │ • Performance under load │ └─────────────────────────────────────────┘ ↑ ┌─────────────────────────────────────────┐ │ Unit Tests │ ├─────────────────────────────────────────┤ │ • Individual method testing │ │ • Configuration parsing │ │ • Data transformation logic │ │ • Mocked external dependencies │ └─────────────────────────────────────────┘ Types of Integration Tests for Connectors Configuration Tests: Validate connector configuration handling Functionality Tests: Test core source/sink operations Error Handling Tests: Verify graceful failure scenarios Performance Tests: Ensure connector meets throughput requirements Compatibility Tests: Test with different Kafka versions Writing Effective Integration Tests Test Structure Pattern Our integration tests follow a consistent pattern:\n@Test public void testConnectorFeature() throws Exception { // 1. Setup: Configure the connector Map\u0026lt;String, String\u0026gt; config = createConnectorConfig(); // 2. Action: Start the connector getConnectRunner().createConnector(config); // 3. Interact: Send/receive data sendTestData(); // 4. Verify: Assert expected behavior await().atMost(Duration.ofSeconds(30)) .untilAsserted(() -\u0026gt; assertExpectedOutcome()); // 5. Cleanup: Handled by @AfterEach } Example: Source Connector Integration Test @Test public void testSourceConnectorBasicFunctionality() throws Exception { log.info(\u0026#34;Testing basic source connector functionality\u0026#34;); // Create and start the source connector Map\u0026lt;String, String\u0026gt; connectorConfig = createSourceConnectorConfig(); getConnectRunner().createConnector(connectorConfig); // Wait for connector to produce messages await().atMost(Duration.ofSeconds(30)) .pollInterval(Duration.ofSeconds(2)) .untilAsserted(() -\u0026gt; { // Consume messages from the source topic consumer.subscribe(Collections.singletonList(SOURCE_TOPIC_NAME)); ConsumerRecords\u0026lt;String, String\u0026gt; records = consumer.poll(Duration.ofSeconds(5)); assertThat(records).isNotEmpty(); assertThat(records.count()).isGreaterThan(0); // Verify message structure for (ConsumerRecord\u0026lt;String, String\u0026gt; record : records) { assertThat(record.key()).isNotNull(); assertThat(record.value()).isNotNull(); // Validate JSON structure JsonNode jsonNode = objectMapper.readTree(record.value()); assertThat(jsonNode.has(\u0026#34;id\u0026#34;)).isTrue(); assertThat(jsonNode.has(\u0026#34;timestamp\u0026#34;)).isTrue(); } }); log.info(\u0026#34;Basic source connector test completed successfully\u0026#34;); } Now let\u0026rsquo;s look at the AbstractIT.java class, which serves as the foundation for our integration tests and provides the necessary infrastructure for running tests against a real Kafka cluster. The ConnectRunner.java class is where the magic happens - it creates an embedded Kafka Connect runtime for testing. It also manages the lifecycle of connectors, allowing us to focus on writing tests without worrying about the underlying infrastructure.\nDeep Dive: AbstractIT.java - The Foundation The AbstractIT class is the backbone of our integration testing framework. It provides:\nTestContainers Integration @Testcontainers public abstract class AbstractIT { @Container protected static final KafkaContainer kafkaContainer = new KafkaContainer(KAFKA_IMAGE_NAME) .withNetwork(Network.newNetwork()) .withStartupTimeout(CONTAINER_STARTUP_TIMEOUT) .withExposedPorts(9092, 8083); } KafkaContainer: Starts a real Kafka instance in Docker Network Isolation: Each test suite runs in its own Docker network Exposed Ports: Maps Kafka ports for communication, 9092 is the default Kafka port and 8083 is for Kafka Connect REST API Startup Timeout: Ensures Kafka is ready before tests run Automatic Cleanup: Containers are stopped after tests Why TestContainers?\nReal Kafka: Not mocked, but actual Apache Kafka running in Docker Isolation: Each test suite gets a fresh Kafka cluster Reproducibility: Consistent environment across development and CI Version Testing: Easy to test against multiple Kafka versions Infrastructure Setup The AbstractIT class handles complex setup in setUp():\n@BeforeEach void setUp() throws Exception { // 1. Initialize Kafka clients (producer, consumer, admin) setupKafkaClients(); // 2. Create necessary topics setupTopics(); // 3. Start embedded Kafka Connect runtime setupKafkaConnect(); } Kafka Clients: Initializes producer and consumer for test data generation and verification Topic Management: Creates required Kafka topics for testing Plugin Directory: Sets up the directory structure for connector plugins Kafka Connect Initialization: Starts the Connect runtime with the configured plugins 1. Kafka Client Configuration private void setupKafkaClients() { // Producer configuration for test data generation Properties producerProps = new Properties(); producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaContainer.getBootstrapServers()); producerProps.put(ProducerConfig.ACKS_CONFIG, \u0026#34;all\u0026#34;); producerProps.put(ProducerConfig.RETRIES_CONFIG, 3); producer = new KafkaProducer\u0026lt;\u0026gt;(producerProps); // Consumer configuration for result verification Properties consumerProps = new Properties(); consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, \u0026#34;earliest\u0026#34;); consumerProps.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); consumer = new KafkaConsumer\u0026lt;\u0026gt;(consumerProps); } Producer: Sends test data to Kafka topics Consumer: Reads messages from Kafka topics for verification Bootstrap Servers: Connects to the Kafka cluster started by TestContainers Acknowledgments: Configured to wait for all replicas to acknowledge messages Retries: Configured to retry sending messages up to 3 times in case of transient errors Auto Offset Reset: Ensures the consumer starts reading from the earliest messages in the topic Enable Auto Commit: Disabled to manually control offset commits during tests Consumer Group: Each test can run in its own consumer group to isolate message consumption 2. Topic Management private void setupTopics() throws ExecutionException, InterruptedException { List\u0026lt;String\u0026gt; topicNames = List.of( TEST_TOPIC_NAME, SINK_TOPIC_NAME, SOURCE_TOPIC_NAME, ConnectRunner.OFFSET_TOPIC, // connect-offsets ConnectRunner.CONFIG_TOPIC, // connect-configs ConnectRunner.STATUS_TOPIC // connect-status ); List\u0026lt;NewTopic\u0026gt; topicsToCreate = topicNames.stream() .filter(name -\u0026gt; !existingTopics.contains(name)) .map(name -\u0026gt; { NewTopic topic = new NewTopic(name, 1, (short) 1); // Special configuration for Connect internal topics if (isConnectInternalTopic(name)) { Map\u0026lt;String, String\u0026gt; configs = new HashMap\u0026lt;\u0026gt;(); configs.put(\u0026#34;cleanup.policy\u0026#34;, \u0026#34;compact\u0026#34;); topic.configs(configs); } return topic; }) .collect(Collectors.toList()); adminClient.createTopics(topicsToCreate).all().get(); } Topic Creation: Creates necessary topics for testing, including internal Kafka Connect topics Compact Cleanup Policy: Ensures that internal topics like offsets and configs are compacted Existing Topics Check: Avoids recreating topics that already exist, preventing errors 3. Plugin Directory Setup The plugin directory setup is crucial for loading the connector JARs:\nprivate static Path setupPluginDir() throws Exception { Path testDir = Files.createTempDirectory(\u0026#34;kafka-connector-template-\u0026#34;); String destFilePath = \u0026#34;./target/kafka-connector-template-0.0.2-jar-with-dependencies.jar\u0026#34;; Path distFile = Paths.get(destFilePath); // Verify JAR exists (built by Maven) assert Files.exists(distFile); // Create plugin directory structure Path pluginDir = Paths.get(testDir.toString(), \u0026#34;plugins/kafka-connector-template/\u0026#34;); Files.createDirectories(pluginDir); // Copy connector JAR to plugin directory Files.copy(distFile, pluginDir.resolve(distFile.getFileName())); return pluginDir; } This is done to ensure that the connector JAR is available for Kafka Connect to load during tests. The directory structure matches what Kafka Connect expects, allowing it to find and use the connector plugin.\nTemporary Directory: Creates a unique directory for each test run JAR Verification: Ensures the connector JAR is built and available Directory Structure: Sets up the expected plugin directory structure for Kafka Connect JAR Copying: Copies the connector JAR to the plugin directory for Kafka Connect to load Plugin Path: Kafka Connect uses this directory to load the connector plugin 4. Kafka Connect Runtime Initialization private void setupKafkaConnect() throws Exception { Path pluginDir = setupPluginDir(); // Create ConnectRunner instance connectRunner = new ConnectRunner(kafkaContainer.getBootstrapServers(), pluginDir); // Start the Connect runtime connectRunner.start(); } ConnectRunner: Manages the embedded Kafka Connect runtime Plugin Directory: Loads the connector JAR from the test resources Kafka Connect Initialization: Starts the Connect runtime with the configured plugins 5. Cleanup @AfterEach void tearDown() throws Exception { // Stop Kafka Connect runtime if (connectRunner != null) { connectRunner.stop(); } // Close Kafka clients if (producer != null) { producer.close(); } if (consumer != null) { consumer.close(); } // Cleanup temporary directories if (pluginDir != null \u0026amp;\u0026amp; Files.exists(pluginDir)) { Files.walk(pluginDir) .sorted(Comparator.reverseOrder()) .map(Pa th::toFile) .forEach(File::delete); } } Resource Management: Ensures all resources are cleaned up after each test Kafka Connect Shutdown: Stops the embedded Connect runtime Kafka Client Cleanup: Closes producer and consumer connections Helper Methods for Test Cases // Connector configuration builders protected Map\u0026lt;String, String\u0026gt; createBasicSinkConnectorConfig() { Map\u0026lt;String, String\u0026gt; config = createBasicConnectorConfig(); config.put(\u0026#34;connector.class\u0026#34;, \u0026#34;com.example.kafka.connect.sink.ExampleSinkConnector\u0026#34;); config.put(\u0026#34;topics\u0026#34;, SINK_TOPIC_NAME); config.put(\u0026#34;name\u0026#34;, \u0026#34;test-sink-connector\u0026#34;); return config; } // Topic creation utilities protected void createTopic(String topicName, int partitions) throws ExecutionException, InterruptedException { NewTopic topic = new NewTopic(topicName, partitions, (short) 1); adminClient.createTopics(List.of(topic)).all().get(); } Deep Dive: ConnectRunner.java - The Engine The ConnectRunner class is where the magic happens - it creates an embedded Kafka Connect runtime for testing. It also manages the lifecycle of connectors, allowing us to focus on writing tests without worrying about the underlying infrastructure.\nWhy Embedded Connect? Instead of starting a separate Connect process, we embed the Connect runtime:\nBenefits:\nControl: Direct access to Connect internals for testing Speed: No process startup overhead Debugging: Easy to debug connector issues Isolation: Each test gets a fresh Connect instance Trade-offs:\nComplexity: More complex setup than REST API testing Memory: Higher memory usage during tests Connect Runtime Architecture public final class ConnectRunner { private Herder herder; // Manages connector lifecycle private Worker worker; // Executes connector tasks private ConnectRestServer restServer; // HTTP API (optional for tests) private Connect\u0026lt;StandaloneHerder\u0026gt; connect; // Main Connect runtime } Startup Process void start() { // 1. Create worker configuration Map\u0026lt;String, String\u0026gt; workerProps = createWorkerConfig(); // 2. Initialize plugin system Plugins plugins = new Plugins(workerProps); DistributedConfig config = new DistributedConfig(workerProps); // 3. Create offset backing store (uses Kafka topics) OffsetBackingStore offsetBackingStore = createOffsetBackingStore(config, plugins); // 4. Create worker worker = new Worker(workerId, time, plugins, config, offsetBackingStore, overridePolicy); // 5. Create herder (standalone mode for testing) herder = new StandaloneHerder(worker, kafkaClusterId, overridePolicy); // 5. Initialize REST API (optional) final RestClient restClient = new RestClient(config); restServer = new ConnectRestServer(10, restClient, workerProps); restServer.initializeServer(); restServer.initializeResources(herder); // 6. Start the Connect runtime connect = new Connect(herder, restServer); connect.start(); } The code above outlines the key steps in starting the Connect runtime:\nWorker Configuration: Sets up the worker properties, including Kafka connection and converters Plugin System: Initializes the plugin system to load connectors Offset Backing Store: Configures the offset storage using Kafka topics Header Creation: Creates the herder instance that manages connector lifecycle and state. It handles connector configuration, status updates, and task management. Without this, the Connect runtime cannot function. Rest API Initialization: Initializes the REST API for connector management (optional for tests) but useful for debugging and monitoring Worker Creation: Creates the worker instance that executes connector tasks. Workers are responsible for running the connector tasks and managing their lifecycle. It handles task assignment, execution, and error recovery. Worker Configuration The worker configuration is crucial for the Connect runtime to function correctly. Some of the key properties include:\nprivate Map\u0026lt;String, String\u0026gt; createWorkerConfig() { Map\u0026lt;String, String\u0026gt; workerProps = new HashMap\u0026lt;\u0026gt;(); // Kafka connection workerProps.put(DistributedConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); // Default converters (connectors can override) workerProps.put(DistributedConfig.KEY_CONVERTER_CLASS_CONFIG, \u0026#34;org.apache.kafka.connect.json.JsonConverter\u0026#34;); workerProps.put(DistributedConfig.VALUE_CONVERTER_CLASS_CONFIG, \u0026#34;org.apache.kafka.connect.json.JsonConverter\u0026#34;); // Plugin system workerProps.put(DistributedConfig.PLUGIN_PATH_CONFIG, pluginDir.toString()); // Connect internal topics workerProps.put(DistributedConfig.GROUP_ID_CONFIG, KAFKA_CONNECT_GROUP_ID); workerProps.put(DistributedConfig.OFFSET_STORAGE_TOPIC_CONFIG, OFFSET_TOPIC); workerProps.put(DistributedConfig.CONFIG_TOPIC_CONFIG, CONFIG_TOPIC); workerProps.put(DistributedConfig.STATUS_STORAGE_TOPIC_CONFIG, STATUS_TOPIC); return workerProps; } Kafka Connection: Specifies the Kafka cluster to connect to Default Converters: Sets up JSON converters for key and value serialization Plugin System: Points to the directory containing connector plugins Connect Internal Topics: Configures topics for offsets, configs, and statuses Offset Storage Configuration One of the most complex parts is setting up offset storage:\nThe following method creates an offset backing store that uses Kafka topics to store offsets, which is crucial for maintaining connector state across restarts and failures.\nWhy do we need this?\nIt allows connectors to track their progress and resume from the last processed offset, ensuring no data loss or duplication. private OffsetBackingStore createOffsetBackingStore(DistributedConfig config, Plugins plugins) { // TopicAdmin for managing the offset topic Supplier\u0026lt;TopicAdmin\u0026gt; topicAdminSupplier = () -\u0026gt; { Map\u0026lt;String, Object\u0026gt; adminConfig = new HashMap\u0026lt;\u0026gt;(); adminConfig.put(\u0026#34;bootstrap.servers\u0026#34;, bootstrapServers); adminConfig.put(\u0026#34;client.id\u0026#34;, \u0026#34;connect-worker-offset-admin\u0026#34;); return new TopicAdmin(adminConfig); }; // Key converter for offset serialization Converter keyConverter = plugins.newConverter(config, \u0026#34;key.converter\u0026#34;, ClassLoaderUsage.CURRENT_CLASSLOADER); // Configure converter Map\u0026lt;String, Object\u0026gt; converterConfig = new HashMap\u0026lt;\u0026gt;(); converterConfig.put(\u0026#34;schemas.enable\u0026#34;, \u0026#34;false\u0026#34;); keyConverter.configure(converterConfig, true); // Create Kafka-based offset store KafkaOffsetBackingStore offsetBackingStore = new KafkaOffsetBackingStore( topicAdminSupplier, () -\u0026gt; OFFSET_TOPIC, keyConverter ); offsetBackingStore.configure(config); return offsetBackingStore; } The code above outlines the key components involved in offset management, which are:\nTopicAdmin: Manages the offset topic creation and configuration Key Converter: Serializes keys for offset storage KafkaOffsetBackingStore: Implements the offset storage using Kafka topics Connector Management public void createConnector(Map\u0026lt;String, String\u0026gt; config) throws ExecutionException, InterruptedException { String connectorName = config.get(ConnectorConfig.NAME_CONFIG); // Use callback for async operation FutureCallback\u0026lt;Herder.Created\u0026lt;ConnectorInfo\u0026gt;\u0026gt; cb = new FutureCallback\u0026lt;\u0026gt;((error, info) -\u0026gt; { if (error != null) { LOGGER.error(\u0026#34;Failed to create connector: {}\u0026#34;, connectorName, error); } else { LOGGER.info(\u0026#34;Created connector {}\u0026#34;, info.result().name()); } }); // Submit connector creation to herder herder.putConnectorConfig(connectorName, config, false, cb); // Wait for completion Herder.Created\u0026lt;ConnectorInfo\u0026gt; connectorInfoCreated = cb.get(); if (!connectorInfoCreated.created()) { throw new RuntimeException(\u0026#34;Failed to create connector: \u0026#34; + connectorName); } } The createConnector method is responsible for creating a new connector in the Connect runtime. It uses the Herder to submit the connector configuration and waits for the operation to complete.\nConnector Name: Extracts the connector name from the configuration FutureCallback: Handles the asynchronous response from the herder Herder Interaction: Submits the connector configuration to the herder for processing Error Handling: Throws an exception if the connector creation fails TestContainers: Real Kafka in Tests Why TestContainers Over Embedded Kafka Many Kafka testing frameworks use embedded Kafka, but TestContainers provides:\nReal Environment: Actual Kafka Docker image, not a test simulation Version Flexibility: Easy to test against different Kafka versions Network Isolation: Each test suite gets isolated networking Resource Management: Automatic cleanup of containers Container Configuration private static final String DEFAULT_KAFKA_TAG = \u0026#34;3.9.1\u0026#34;; private static final DockerImageName KAFKA_IMAGE_NAME = DockerImageName.parse(\u0026#34;apache/kafka\u0026#34;).withTag(DEFAULT_KAFKA_TAG); @Container protected static final KafkaContainer kafkaContainer = new KafkaContainer(KAFKA_IMAGE_NAME) .withNetwork(Network.newNetwork()) .withStartupTimeout(CONTAINER_STARTUP_TIMEOUT) .withExposedPorts(9092, 8083); Docker Image: Uses the official Apache Kafka Docker image Network Isolation: Creates a new Docker network for the test suite Startup Timeout: Ensures Kafka is ready before tests run Exposed Ports: Maps Kafka ports for communication (9092 for Kafka, 8083 for Kafka Connect REST API) Automatic Cleanup: Containers are stopped after tests, ensuring no leftover state Container Lifecycle Test Suite Start ↓ ┌─────────────────┐ │ Container Start │ ← Automatic (TestContainers) └─────────────────┘ ↓ ┌─────────────────┐ │ Network Setup │ ← Isolated Docker network └─────────────────┘ ↓ ┌─────────────────┐ │ Port Mapping │ ← Random host ports → container ports └─────────────────┘ ↓ ┌─────────────────┐ │ Health Check │ ← Wait for Kafka to be ready └─────────────────┘ ↓ Test Execution ↓ (after all tests) ┌─────────────────┐ │ Container Stop │ ← Automatic cleanup └─────────────────┘ Advanced Container Configuration For more complex scenarios, you can customize the container:\n@Container protected static final KafkaContainer kafkaContainer = new KafkaContainer(KAFKA_IMAGE_NAME) .withNetwork(Network.newNetwork()) .withStartupTimeout(Duration.ofMinutes(5)) .withExposedPorts(9092, 8083) // Custom environment variables .withEnv(\u0026#34;KAFKA_AUTO_CREATE_TOPICS_ENABLE\u0026#34;, \u0026#34;true\u0026#34;) .withEnv(\u0026#34;KAFKA_NUM_PARTITIONS\u0026#34;, \u0026#34;3\u0026#34;) // Custom startup command .withCommand(\u0026#34;sh\u0026#34;, \u0026#34;-c\u0026#34;, \u0026#34;start-kafka.sh \u0026amp;\u0026amp; tail -f /dev/null\u0026#34;) // Volume mounting for custom configuration .withFileSystemBind(\u0026#34;./config/server.properties\u0026#34;, \u0026#34;/opt/kafka/config/server.properties\u0026#34;); Configuration Deep Dive Connector Configuration Hierarchy Test Configuration Hierarchy: 1. Base Configuration (AbstractIT.createBasicConnectorConfig()) ├── key.converter: StringConverter ├── value.converter: JsonConverter ├── value.converter.schemas.enable: false └── tasks.max: 1 2. Connector-Specific Configuration ├── Sink: createBasicSinkConnectorConfig() │ ├── connector.class: ExampleSinkConnector │ ├── topics: sink-topic │ └── name: test-sink-connector │ └── Source: createBasicSourceConnectorConfig() ├── connector.class: ExampleSourceConnector ├── topic: source-topic └── name: test-source-connector 3. Test-Specific Overrides ├── Custom batch sizes ├── Specific timeouts └── Test identifiers Worker Configuration The Connect worker requires specific configuration for testing:\n// Essential worker configuration workerProps.put(DistributedConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); workerProps.put(DistributedConfig.GROUP_ID_CONFIG, \u0026#34;connect-cluster\u0026#34;); // Offset management (where connector state is stored) workerProps.put(DistributedConfig.OFFSET_STORAGE_TOPIC_CONFIG, \u0026#34;connect-offsets\u0026#34;); workerProps.put(DistributedConfig.CONFIG_TOPIC_CONFIG, \u0026#34;connect-configs\u0026#34;); workerProps.put(DistributedConfig.STATUS_STORAGE_TOPIC_CONFIG, \u0026#34;connect-status\u0026#34;); // Plugin loading workerProps.put(DistributedConfig.PLUGIN_PATH_CONFIG, pluginDir.toString()); // Converters (can be overridden by connectors) workerProps.put(DistributedConfig.KEY_CONVERTER_CLASS_CONFIG, \u0026#34;org.apache.kafka.connect.json.JsonConverter\u0026#34;); workerProps.put(DistributedConfig.VALUE_CONVERTER_CLASS_CONFIG, \u0026#34;org.apache.kafka.connect.json.JsonConverter\u0026#34;); These settings ensure that the Connect worker can communicate with the Kafka cluster, manage offsets, and load the necessary plugins for connector functionality.\nConnect Internal Topics Kafka Connect uses three internal topics for coordination:\nconnect-offsets: Stores source connector offsets\nCleanup policy: compact Contains: Source connector position tracking The topic should be created with a compact cleanup policy to ensure that only the latest offsets are retained, preventing unnecessary data growth. connect-configs: Stores connector configurations\nCleanup policy: compact Contains: Connector and task configurations connect-status: Stores connector and task status\nCleanup policy: compact Contains: Health and status information Topic Configuration in Tests private void setupTopics() throws ExecutionException, InterruptedException { List\u0026lt;NewTopic\u0026gt; topicsToCreate = topicNames.stream() .map(name -\u0026gt; { NewTopic topic = new NewTopic(name, 1, (short) 1); // Configure Connect internal topics if (name.equals(OFFSET_TOPIC) || name.equals(CONFIG_TOPIC) || name.equals(STATUS_TOPIC)) { Map\u0026lt;String, String\u0026gt; configs = new HashMap\u0026lt;\u0026gt;(); configs.put(\u0026#34;cleanup.policy\u0026#34;, \u0026#34;compact\u0026#34;); topic.configs(configs); } return topic; }) .collect(Collectors.toList()); adminClient.createTopics(topicsToCreate).all().get(); } Best Practices and Real-World Examples 1. Test Organization Structure your integration tests by functionality:\nTest Organization Structure src/test/java/com/example/kafka/connect/integration/ ├── AbstractIT.java # Base test infrastructure ├── ConnectRunner.java # Embedded Connect runtime ├── ConnectorConfigurationIT.java # Configuration validation tests ├── ExampleSinkConnectorIT.java # Sink connector functionality └── ExampleSourceConnectorIT.java # Source connector functionality 2. Comprehensive Configuration Testing @Test public void testConnectorConfigValidation() { ExampleSinkConnector connector = new ExampleSinkConnector(); // Test valid configuration Map\u0026lt;String, String\u0026gt; validConfig = createValidSinkConfig(); Config result = connector.validate(validConfig); assertThat(result.configValues()).allMatch(cv -\u0026gt; cv.errorMessages().isEmpty()); // Test invalid configuration Map\u0026lt;String, String\u0026gt; invalidConfig = new HashMap\u0026lt;\u0026gt;(); // Missing required \u0026#39;topics\u0026#39; property Config invalidResult = connector.validate(invalidConfig); boolean hasTopicsError = invalidResult.configValues().stream() .anyMatch(cv -\u0026gt; cv.name().equals(\u0026#34;topics\u0026#34;) \u0026amp;\u0026amp; !cv.errorMessages().isEmpty()); assertThat(hasTopicsError).isTrue(); } Valid Configuration: Ensure the connector accepts valid configurations without errors Invalid Configuration: Test that missing or incorrect configurations produce appropriate error messages 3. Error Handling and Recovery @Test public void testConnectorErrorHandling() throws Exception { // Start connector with configuration that will cause errors Map\u0026lt;String, String\u0026gt; config = createSinkConnectorConfig(); config.put(\u0026#34;external.system.url\u0026#34;, \u0026#34;http://nonexistent-host:9999\u0026#34;); connectRunner.createConnector(config); // Send messages that should trigger errors producer.send(new ProducerRecord\u0026lt;\u0026gt;(SINK_TOPIC_NAME, \u0026#34;key\u0026#34;, \u0026#34;value\u0026#34;)).get(); // Verify connector handles errors gracefully await().atMost(Duration.ofSeconds(30)) .untilAsserted(() -\u0026gt; { // Check connector status or error logs // Verify messages are retried or sent to DLQ }); } Error Scenarios: Simulate common error conditions (e.g., network failures, invalid data) Graceful Recovery: Ensure the connector can recover from errors without losing data 4. Performance Testing @Test public void testSourceConnectorThroughput() throws Exception { Map\u0026lt;String, String\u0026gt; config = createSourceConnectorConfig(); config.put(\u0026#34;batch.size\u0026#34;, \u0026#34;1000\u0026#34;); config.put(\u0026#34;poll.interval.ms\u0026#34;, \u0026#34;100\u0026#34;); connectRunner.createConnector(config); // Measure message production rate long startTime = System.currentTimeMillis(); int expectedMessages = 10000; await().atMost(Duration.ofMinutes(2)) .untilAsserted(() -\u0026gt; { consumer.subscribe(Collections.singletonList(SOURCE_TOPIC_NAME)); ConsumerRecords\u0026lt;String, String\u0026gt; records = consumer.poll(Duration.ofSeconds(1)); totalMessages += records.count(); assertThat(totalMessages).isGreaterThanOrEqualTo(expectedMessages); }); long duration = System.currentTimeMillis() - startTime; double messagesPerSecond = expectedMessages / (duration / 1000.0); log.info(\u0026#34;Throughput: {} messages/second\u0026#34;, messagesPerSecond); assertThat(messagesPerSecond).isGreaterThan(100); // Minimum expected throughput } 5. Multi-Version Testing Test your connector against multiple Kafka versions:\n@ParameterizedTest @ValueSource(strings = {\u0026#34;3.8.1\u0026#34;, \u0026#34;3.9.1\u0026#34;, \u0026#34;4.0.0\u0026#34;}) public void testConnectorCompatibility(String kafkaVersion) { // Use different Kafka container versions try (KafkaContainer kafka = new KafkaContainer( DockerImageName.parse(\u0026#34;apache/kafka\u0026#34;).withTag(kafkaVersion))) { kafka.start(); // Run connector tests against this Kafka version testBasicConnectorFunctionality(kafka.getBootstrapServers()); } } Conclusion Integration testing for Kafka connectors is complex but essential. The framework we\u0026rsquo;ve built provides:\nReal Kafka Environment: Using TestContainers for authentic testing Embedded Connect Runtime: Full control over connector lifecycle Comprehensive Coverage: Configuration, functionality, and error scenarios Developer Productivity: Fast feedback and easy debugging Key Takeaways Start with the Template: Use our Kafka Connector Template as your foundation Test Early and Often: Don\u0026rsquo;t wait until the end to add integration tests Test Real Scenarios: Use actual Kafka, not mocks, for meaningful results Cover Edge Cases: Test error conditions, high load, and configuration variants Automate Everything: Include integration tests in your CI/CD pipeline Next Steps Clone the Kafka Connector Template Run the existing integration tests: mvn verify Adapt the tests for your specific connector requirements Add performance and compatibility tests as needed Remember: Good integration tests are the difference between a connector that works in development and one that thrives in production. Invest the time upfront - your future self (and your operations team) will thank you.\nFor more Kafka Connect tips and open-source tools, follow the blog series and star the template repository.\n","permalink":"https://joel-hanson.github.io/posts/mastering-integration-testing-for-kafka-connectors-a-complete-guide/","summary":"Integration testing is crucial for Kafka Connectors to ensure they work correctly with external systems. This guide provides a comprehensive approach to setting up and executing integration tests for Kafka Connectors, covering everything from environment setup to best practices for testing source and sink connectors.","title":"Mastering Integration Testing for Kafka Connectors: A Complete Guide"},{"content":"Ever wished you could talk to your Kafka cluster using natural language—or call it directly from Claude Desktop or any other LLM client? With Model Context Protocol (MCP) and the FastMCP SDK, you can expose all your Kafka operations as discrete, promptable tools.\nIn this guide you’ll learn how to setup a MCP server to do the following:\nList Kafka topics Create \u0026amp; delete topics Produce messages Troubleshoot issues via a free‑form prompt Try it out \u0026amp; contribute: GitHub Repo\nProject Setup git clone https://github.com/joel-hanson/kafka-mcp-server.git cd kafka-mcp-server # Create environment conda create -n kafka-mcp python=3.10 -y conda activate kafka-mcp # Install dependencies pip install -r requirements.txt Requirements:\nPython 3.10+ A running Kafka broker (e.g. localhost:9092; you can docker compose up a quick cluster) MCP CLI tools (mcp[cli]) and kafka-python Optional: Claude Desktop or any MCP‑compatible LLM client File Structure kafka-mcp-server/ ├── server.py # FastMCP entrypoint: registers @tool, @resource, @prompt ├── kafka_utils.py # Connection loader \u0026amp; reusable KafkaManager ├── kafka_troubleshoot.py # Static troubleshooting guide constant ├── requirements.txt └── README.md server.py – your main MCP server. Decorate functions with\n@mcp.tool() for command‑style operations, @mcp.resource() for raw JSON endpoints, @mcp.prompt() for free‑form guidance. kafka_utils.py – handles parsing kafka.properties, instantiates and caches your KafkaAdminClient/Producer/Consumer, and stores it on lifespan_context.kafka_manager.\n– contains KAFKA_TROUBLESHOOTING_GUIDE, a templated Markdown guide you plug into your @prompt().\nDefining Your Tools 1. Initialize \u0026amp; Connect from mcp.server.fastmcp import Context, FastMCP mcp = FastMCP(\u0026#34;Kafka MCP Server\u0026#34;, lifespan=kafka_lifespan) @mcp.tool() def kafka_initialize_connection(path: str) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;Load kafka.properties and connect to the cluster.\u0026#34;\u0026#34;\u0026#34; try: manager = KafkaManager.from_properties(path) ctx.request_context.lifespan_context.kafka_manager = manager return \u0026#34;Connected to Kafka cluster.\u0026#34; except Exception as e: return f\u0026#34;Connection failed: {e}\u0026#34; 2. List Topics (rich example) from mcp.server.fastmcp import Context, FastMCP mcp = FastMCP(\u0026#34;Kafka MCP Server\u0026#34;, lifespan=kafka_lifespan) @mcp.tool() def kafka_list_topics(ctx: Context) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;List all topics in the Kafka cluster\u0026#34;\u0026#34;\u0026#34; manager = ctx.request_context.lifespan_context.kafka_manager if not manager: return \u0026#34;Error: Not connected. Use kafka_initialize_connection first.\u0026#34; try: topics = manager.list_topics() if not topics: return \u0026#34;No topics found in the Kafka cluster.\u0026#34; lines = [ f\u0026#34;• {t[\u0026#39;name\u0026#39;]} (partitions: {t[\u0026#39;partitions\u0026#39;]}, replication: {t[\u0026#39;replication_factor\u0026#39;]})\u0026#34; for t in topics ] return \u0026#34;Topics in Kafka cluster:\\n\u0026#34; + \u0026#34;\\n\u0026#34;.join(lines) except Exception as e: return f\u0026#34;Error listing topics: {e}\u0026#34; 3. Raw JSON Resource from mcp.server.fastmcp import Context, FastMCP mcp = FastMCP(\u0026#34;Kafka MCP Server\u0026#34;, lifespan=kafka_lifespan) @mcp.resource(\u0026#34;kafka://topics\u0026#34;) def get_all_topics() -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;Provide all Kafka topics as JSON\u0026#34;\u0026#34;\u0026#34; manager = mcp.get_context().request_context.lifespan_context.kafka_manager if not manager: return \u0026#34;Not connected to Kafka cluster\u0026#34; try: return json.dumps(manager.list_topics(), indent=2) except Exception as e: return f\u0026#34;Error fetching topics: {e}\u0026#34; 4. Troubleshooting Prompt from kafka_troubleshoot import KAFKA_TROUBLESHOOTING_GUIDE from mcp.server.fastmcp import Context, FastMCP mcp = FastMCP(\u0026#34;Kafka MCP Server\u0026#34;, lifespan=kafka_lifespan) @mcp.prompt() def kafka_troubleshoot(issue: str) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;Get comprehensive troubleshooting guidance for Kafka issues\u0026#34;\u0026#34;\u0026#34; return KAFKA_TROUBLESHOOTING_GUIDE.format(issue=issue) 5. (Bonus) Produce from mcp.server.fastmcp import Context, FastMCP mcp = FastMCP(\u0026#34;Kafka MCP Server\u0026#34;, lifespan=kafka_lifespan) @mcp.tool() def kafka_produce(topic: str, message: str) -\u0026gt; str: \u0026#34;\u0026#34;\u0026#34;Send a message to a Kafka topic\u0026#34;\u0026#34;\u0026#34; prod = ctx.request_context.lifespan_context.kafka_manager.producer try: prod.send(topic, message.encode(\u0026#39;utf-8\u0026#39;)) prod.flush() return f\u0026#34;Message sent to `{topic}`: {message}\u0026#34; except Exception as e: return f\u0026#34;Error producing message: {e}\u0026#34; Quick Start 1. Clone \u0026amp; Set Up git clone https://github.com/joel-hanson/kafka-mcp-server.git cd kafka-mcp-server conda create -n kafka-mcp python=3.10 -y conda activate kafka-mcp pip install -r requirements.txt 2. Start Kafka with Docker (Recommended) Before running your MCP server, bring up a local Kafka cluster:\ndocker compose up -d This starts a Kafka broker on localhost:9092. You can now use the kafka.properties file to connect.\nbootstrap.servers=localhost:9092 client.id=kafka-mcp-client 3. Running the MCP Server Option 1: Dev mode with Inspector (Recommended) mcp dev server.py This launches the server with the MCP Inspector, a GUI for exploring tools and prompts.\nOption 2: Headless python server.py Sprinkle logging.debug(...) in your tool functions to trace errors in real time.\nWhy use Docker Compose? It gives you a reliable, reproducible Kafka environment for local development—no need to install Kafka manually.\nExample Usage Flow First, connect to Kafka:\nUse kafka_initialize_connection with your properties file \u0026lt;path\u0026gt; OR Initialize connection with kafka using the properties file \u0026lt;path\u0026gt; List existing topics:\nUse kafka_list_topics to see all topics OR List all the topics Create a new topic:\nUse kafka_create_topic with name \u0026#34;my-topic\u0026#34;, partitions 3, replication_factor 1 OR Create me a topic with name \u0026#34;my-topic\u0026#34; Get topic details:\nUse kafka_get_topic_info for \u0026#34;my-topic\u0026#34; OR Please share the topics details of \u0026#34;my-topics\u0026#34; Integrating with Claude Desktop Configure claude_desktop_config.json:\n{ \u0026#34;mcpServers\u0026#34;: { \u0026#34;Kafka MCP Server\u0026#34;: { \u0026#34;command\u0026#34;: \u0026#34;/opt/miniconda3/envs/kafka-mcp/bin/python\u0026#34;, \u0026#34;args\u0026#34;: [\u0026#34;server.py\u0026#34;] } } } Now Claude will auto‑discover and surface your new Kafka tools.\nScreenshots 1. Listing available tools 2. Connecting to the Kafka cluster 3. Listing topics 4. Deleting a topic 5. Detailed topic information 6. Producing messages 7. Consuming messages via shell Final Thoughts You’ve just transformed your Kafka cluster into a first‑class, LLM‑driven API—ideal for internal dev‑tools, rapid troubleshooting, or even exposing safe, controlled access to non‑Kafka experts. Next up:\nSchema Registry \u0026amp; Avro/JSON serialization Consumer group and offset‑management tools Cluster‑wide metrics \u0026amp; health checks ACLs, security hardening, and multi‑tenant isolation Feel free to fork, extend, and contribute back at https://github.com/Joel-hanson/kafka-mcp-server. Happy building!\nFor more Kafka Connect tips and open-source tools, follow the blog series For more on MCP and LLM integration, check out the MCP documentation\n","permalink":"https://joel-hanson.github.io/posts/16-breaking-kafka-walls-building-an-mcp-server-for-your-kafka-cluster/","summary":"A step‑by‑step guide to using FastMCP and the MCP protocol to expose Kafka operations (topic management, produce/consume, troubleshooting) as LLM‑accessible tools.","title":"Building an MCP Server for Your Kafka Cluster"},{"content":" 🚀 Launch Markdown2Medium Converter The Problem: Writing Once, Publishing Twice As a technical writer, I love using Markdown for my blog posts. My current setup involves:\nWriting in Markdown Publishing to my Hugo static site As well as sharing on Medium for broader reach But here\u0026rsquo;s the catch: Medium doesn’t natively support Markdown. Every time I wanted to cross-post, I had to:\nCopy my Markdown content Manually reformat it in Medium’s editor Fix broken code blocks, images, and headings This was time-consuming and error-prone. I needed a better way.\nThe Solution: A Markdown-to-Medium Converter I built a simple web tool that:\n✅ Converts Markdown to Medium-friendly Content\n✅ Preserves code blocks, images, and formatting\n✅ Provides a live preview before publishing\n✅ Lets me copy content\nNow, my workflow is seamless:\nWrite in Markdown (as usual) Paste into my converter tool Copy the content → Paste into Medium No more manual tweaks!\nTry It Yourself https://markdown2medium.vercel.app/\nYou can find the tool here: GitHub Repo\nHow It Works The tool uses:\nmarked.js for Markdown parsing DOM manipulation to add Medium’s required CSS classes Clipboard API for easy copying Example:\n# My Post This is **bold** text. ```python print(\u0026#34;Hello, Medium!\u0026#34;) ```_ Gets converted to:\n\u0026lt;h1 class=\u0026#34;graf graf--h1\u0026#34;\u0026gt;My Post\u0026lt;/h1\u0026gt; \u0026lt;p class=\u0026#34;graf graf--p\u0026#34;\u0026gt;This is \u0026lt;strong class=\u0026#34;markup--strong markup--p-strong\u0026#34;\u0026gt;bold\u0026lt;/strong\u0026gt; text.\u0026lt;/p\u0026gt; \u0026lt;pre class=\u0026#34;graf graf--pre\u0026#34;\u0026gt;\u0026lt;code class=\u0026#34;markup--code markup--pre-code\u0026#34;\u0026gt;print(\u0026#34;Hello, Medium!\u0026#34;)\u0026lt;/code\u0026gt;\u0026lt;/pre\u0026gt; Future Plans: Automation with GitHub Actions Next step: A GitHub Action that:\nWatches for new Markdown posts in my Hugo blog Auto-converts them to Medium-ready Content Publishes directly via Medium’s API This would make cross-posting completely hands-off!\nFinal Thoughts If you’re a Markdown writer who also publishes on Medium, this tool can save you hours. No more manual reformatting—just write once, publish everywhere.\n🔗 Try it out \u0026amp; contribute: GitHub Repo\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/15-how-i-automated-my-markdown-publishing-on-medium-no-more-manual-work/","summary":"Tired of manually reformatting Markdown for Medium? Discover my open-source tool that converts Hugo-friendly Markdown into perfectly formatted Medium posts instantly.","title":"How I Automated My Markdown Publishing on Medium (No More Manual Work!)"},{"content":"Tombstone records are a fundamental concept in Kafka, especially when working with compacted topics. These special records—consisting of a key and a null value—signal that a key should be removed during the compaction process. Let\u0026rsquo;s explore how to create these tombstones using the Kafka console producer.\nWhat Are Tombstone Records? A tombstone record in Kafka is simply a record with:\nA normal key (identifying which record to remove) A null value (signaling deletion) When Kafka\u0026rsquo;s log compaction runs, it sees these tombstones and removes all records with that key, including the tombstone itself (after a retention period).\nCreating Tombstones With kafka-console-producer.sh KIP-810 introduced a simple but powerful feature: the ability to explicitly mark values as null in the Kafka Console Producer using a special \u0026ldquo;null marker\u0026rdquo; string.\nThis feature allows you to directly produce records with null values from the command line, making operations like tombstoning much more straightforward.\nThe traditional challenge was that the console producer didn\u0026rsquo;t have a straightforward way to specify null values. Thanks to the null.marker property, creating tombstones is now simple:\nStep 1: Start the Console Producer with Null Marker bin/kafka-console-producer.sh --bootstrap-server localhost:9092 \\ --topic my-compacted-topic \\ --property parse.key=true \\ --property key.separator=: \\ --property null.marker=NULL Let\u0026rsquo;s break down these parameters:\n--bootstrap-server: Your Kafka broker address --topic: The target topic (typically a compacted topic) parse.key=true: Enables key-value parsing key.separator=:: Defines the character separating keys from values null.marker=NULL: Defines what string will be interpreted as a null value Step 2: Send Your Tombstone Records Once the producer is running, you can send tombstone records by using your defined null marker:\n# Format: key:value user123:NULL product456:NULL order789:NULL Each of these entries creates a tombstone record for the respective key.\nA Real-World Example Example 1: Tombstoning Records in a Compacted Topic Let\u0026rsquo;s see tombstones in action with a user database example:\n# First, add some regular records user1:{\u0026#34;name\u0026#34;:\u0026#34;Alice\u0026#34;,\u0026#34;email\u0026#34;:\u0026#34;alice@example.com\u0026#34;} user2:{\u0026#34;name\u0026#34;:\u0026#34;Bob\u0026#34;,\u0026#34;email\u0026#34;:\u0026#34;bob@example.com\u0026#34;} user3:{\u0026#34;name\u0026#34;:\u0026#34;Charlie\u0026#34;,\u0026#34;email\u0026#34;:\u0026#34;charlie@example.com\u0026#34;} # Now, let\u0026#39;s delete user2 with a tombstone user2:NULL After compaction runs, all records with the key \u0026ldquo;user2\u0026rdquo; will be removed from the topic.\nLet\u0026rsquo;s say you have a user-profiles topic that is compacted and contains the latest information about each user:\n# Initial state of the topic user1:{\u0026#34;name\u0026#34;:\u0026#34;Alice\u0026#34;,\u0026#34;email\u0026#34;:\u0026#34;alice@example.com\u0026#34;} user2:{\u0026#34;name\u0026#34;:\u0026#34;Bob\u0026#34;,\u0026#34;email\u0026#34;:\u0026#34;bob@example.com\u0026#34;} user3:{\u0026#34;name\u0026#34;:\u0026#34;Charlie\u0026#34;,\u0026#34;email\u0026#34;:\u0026#34;charlie@example.com\u0026#34;} When a user deletes their account, you need to tombstone their record:\n# Send a tombstone record to mark user456 for eventual deletion user2:NULL After compaction, user456\u0026rsquo;s record will be removed entirely.\nExample 2: Clearing Values in a Schema Evolution Scenario Imagine you\u0026rsquo;re evolving your event schema and want to remove a field:\n# Original records order123:{\u0026#34;id\u0026#34;:123,\u0026#34;customer\u0026#34;:\u0026#34;Alice\u0026#34;,\u0026#34;deprecated_field\u0026#34;:\u0026#34;value\u0026#34;} order124:{\u0026#34;id\u0026#34;:124,\u0026#34;customer\u0026#34;:\u0026#34;Bob\u0026#34;,\u0026#34;deprecated_field\u0026#34;:\u0026#34;value\u0026#34;} # Clear the deprecated field with null order123:{\u0026#34;id\u0026#34;:123,\u0026#34;customer\u0026#34;:\u0026#34;Alice\u0026#34;,\u0026#34;deprecated_field\u0026#34;:null} order124:{\u0026#34;id\u0026#34;:124,\u0026#34;customer\u0026#34;:\u0026#34;Bob\u0026#34;,\u0026#34;deprecated_field\u0026#34;:null} Using the console producer with null marker makes these operations simple.\nVerifying Your Tombstones To confirm your tombstones were created correctly, you can consume from the topic with the --from-beginning flag and include null values:\nbin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \\ --topic my-compacted-topic \\ --from-beginning \\ --property print.key=true \\ --property print.value=true \\ --property key.separator=: \\ --property print.null.value=true Note the important print.null.value=true parameter that ensures null values are displayed (typically as \u0026ldquo;null\u0026rdquo;).\nConclusion Creating tombstone records from the Kafka console producer is straightforward with the null marker feature. This powerful capability lets you easily manage data deletion in compacted topics directly from the command line, making development and operational tasks much simpler.\nNext time you need to clean up data in a compacted topic, remember this pattern—it\u0026rsquo;s a clean, standard way to handle data removal in the Kafka ecosystem.\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/14-creating-tombstone-records-using-kafka-console-producer-sh-a-quick-guide/","summary":"A practical guide to creating tombstone records for Kafka compacted topics using the kafka-console-producer.sh command-line tool with the null marker feature","title":"Creating Tombstone Records Using kafka-console-producer.sh: A Quick Guide"},{"content":"🔗 GitHub Repository: github.com/Joel-hanson/kafka-connector-template\n⭐ Star the repo to stay updated!\nApache Kafka is a powerful distributed event streaming platform, and Kafka Connect makes it easy to integrate Kafka with external systems. While many pre-built connectors exist, real-world applications often need custom connectors tailored for proprietary systems, custom logic, or advanced error handling.\nThat’s where this production-ready template comes in—it removes the boilerplate and gives you everything you need to build, test, and deploy connectors with ease.\nUse This Template to Kickstart Your Own Connector Instead of cloning and manually renaming everything, you can use the GitHub Template feature to instantly create your own connector project with this structure.\nSteps to Create Your Project from This Template Go to the Template Repo\ngithub.com/Joel-hanson/kafka-connector-template\nClick “Use this template”\nAt the top of the repo, hit the green Use this template button.\nFill in Your Project Details\nSet the repository name Add a description Choose if you want it to be public or private Click “Create repository from template”\nClone Your New Repo\ngit clone https://github.com/your-username/your-connector-repo.git cd your-connector-repo Start Building\nModify the connector classes under src/main/java Update your pom.xml with any new dependencies Customize Docker configs and connector configs in /config Tip: Don’t forget to replace package names (com.example.kafka.connect) with your own!\nWhy Use This Template? Building a Kafka connector from scratch can take days or even weeks. This open-source template gives you a 5-minute head start:\n✅ Skip complex setup – Docker, Kafka (KRaft), and CI/CD are preconfigured\n✅ Focus on logic – Just implement your business-specific code\n✅ Test confidently – Comes with comprehensive integration tests using TestContainers\n✅ Go production – Includes Docker health checks, error handling patterns, and more\nThe project includes extensive integration tests using TestContainers that verify end-to-end functionality with real Kafka infrastructure. These tests are based on the testing framework from the Aiven JDBC Connector for Apache Kafka, providing a proven foundation for reliable connector testing.\nTech Stack \u0026amp; Project Structure This template includes:\nApache Kafka (KRaft mode) – No ZooKeeper needed Kafka Connect with both source and sink examples Docker for running Kafka, Connect, and ZooKeeper-free clusters Maven for dependency management GitHub Actions for automated testing Directory Layout:\nkafka-connector-template/ ├── .github/ # CI/CD workflows ├── config/ # Connector configs ├── docker/ # Docker setup (KRaft mode) ├── src/ # Java source and test code │ ├── main/ │ └── test/ ├── pom.xml # Maven config └── README.md Quick Start in 5 Minutes 1. Clone and Build the Project git clone https://github.com/Joel-hanson/kafka-connector-template.git cd kafka-connector-template mvn clean package 2. Start Kafka + Connect (KRaft Mode) cd docker docker-compose up -d 3. Deploy the Example Source Connector curl -X POST -H \u0026#34;Content-Type: application/json\u0026#34; \\ --data @config/source-connector.json \\ http://localhost:8083/connectors 4. Validate It’s Working docker exec -it kafka kafka-console-consumer.sh \\ --bootstrap-server localhost:9092 \\ --topic example-source-topic Building Your Own Connector Implementing a Source Connector A source connector pulls data from external systems into Kafka.\nEdit:\nsrc/main/java/com/example/kafka/connect/source/ExampleSourceConnector.java src/main/java/com/example/kafka/connect/source/ExampleSourceTask.java Sample:\npublic class MySourceConnector extends SourceConnector { @Override public void start(Map\u0026lt;String, String\u0026gt; props) { // Initialization logic } @Override public Class\u0026lt;? extends Task\u0026gt; taskClass() { return MySourceTask.class; } @Override public List\u0026lt;Map\u0026lt;String, String\u0026gt;\u0026gt; taskConfigs(int maxTasks) { // Configure tasks } } Implementing a Sink Connector A sink connector moves data from Kafka to an external system.\nEdit:\nsrc/main/java/com/example/kafka/connect/sink/ExampleSinkConnector.java src/main/java/com/example/kafka/connect/sink/ExampleSinkTask.java Sample:\npublic class MySinkTask extends SinkTask { @Override public void put(Collection\u0026lt;SinkRecord\u0026gt; records) { // Write to destination } } Docker Highlights Running Kafka with KRaft (no ZooKeeper) is easier than ever.\ndocker/docker-compose.yml includes:\nkafka: image: apache/kafka:3.6.0 environment: KAFKA_PROCESS_ROLES: controller,broker KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 ... Start it with:\ndocker-compose up -d Testing Connectors Run Unit Tests mvn test Run Integration Tests mvn verify The template includes:\nValidation of connector configs Sample source/sink connector tests Comprehensive integration tests with TestContainers End-to-end functionality verification with real Kafka clusters Deploying the Connector via REST API Once Kafka Connect is running, deploy your connector:\nPOST the Configuration curl -X POST -H \u0026#34;Content-Type: application/json\u0026#34; \\ --data \u0026#39;{ \u0026#34;name\u0026#34;: \u0026#34;source-connector\u0026#34;, \u0026#34;config\u0026#34;: { \u0026#34;connector.class\u0026#34;: \u0026#34;com.example.kafka.connect.source.SourceConnector\u0026#34;, \u0026#34;tasks.max\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;topic\u0026#34;: \u0026#34;test-topic\u0026#34;, \u0026#34;poll.interval.ms\u0026#34;: \u0026#34;5000\u0026#34; } }\u0026#39; \\ http://localhost:8083/connectors Check Connector Status curl http://localhost:8083/connectors/source-connector/status | jq Common Issues \u0026amp; Fixes Docker Volume Issues:\ndocker-compose down -v \u0026amp;\u0026amp; docker-compose up --build Connector Doesn’t Load?\nEnsure your .jar is in the right path Restart Kafka Connect: docker restart connect Ready for Production? Here’s how to take it to the next level:\n✅ Enable SSL and Auth in connect-distributed.properties\n✅ Add Prometheus metrics for monitoring\n✅ Use dead-letter queues and retry logic for error handling\n✅ Set config.storage.replication.factor=3 for HA setups\nContributing Want to improve the template or report a bug?\nFork the repo Make your changes Open a PR! Further Reading Kafka Connect Documentation Star the template → github.com/Joel-hanson/kafka-connector-template\nWhat\u0026rsquo;s Next? Level Up Your Testing Game Follow-up Reading: I\u0026rsquo;ve written a comprehensive follow-up post on how to automate integration testing for Kafka connectors, which complements this template perfectly.\nMastering Integration Testing for Kafka Connectors: A Complete Guide\nLearn how to build robust test suites that catch bugs before they hit production!\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/13-build-custom-kafka-connectors-fast-with-this-open-source-template/","summary":"Apache Kafka is a powerful distributed event streaming platform, and \u003cstrong\u003eKafka Connect\u003c/strong\u003e makes it easy to integrate Kafka with external systems. While many pre-built connectors exist, real-world applications often need \u003cstrong\u003ecustom connectors\u003c/strong\u003e tailored for proprietary systems, custom logic, or advanced error handling.\nThat’s where this \u003cstrong\u003eproduction-ready template\u003c/strong\u003e comes in—it removes the boilerplate and gives you everything you need to build, test, and deploy connectors with ease.","title":"Build Custom Kafka Connectors Fast with This Open-Source Template"},{"content":"Kafka Connect provides a flexible way to process streaming data using Single Message Transforms (SMTs). If you need to filter out tombstone records (records with null values), you should use the generic Filter transformation along with the RecordIsTombstone predicate.\nHere’s the correct configuration:\n# Define the predicate to detect tombstone records (i.e., records with null values) predicates=dropTombstone predicates.dropTombstone.type=org.apache.kafka.connect.transforms.predicates.RecordIsTombstone # Configure the Filter transformation to drop records that match the predicate transforms=dropTombstone transforms.dropTombstone.type=org.apache.kafka.connect.transforms.Filter transforms.dropTombstone.predicate=dropTombstone Explanation What is a Predicate? A predicate in Kafka Connect is a condition that evaluates whether a given record meets certain criteria. It returns either true or false. If true, the transformation (such as filtering) is applied. In this case, the predicate named dropTombstone uses the built-in class RecordIsTombstone, which evaluates to true when a record’s value is null.\nPredicate Setup The predicate named dropTombstone uses the built-in class RecordIsTombstone, which evaluates to true when a record’s value is null.\nWhat is the Filter Transformation? The Filter transformation allows records to be selectively included or excluded based on a predicate. If a record satisfies the specified predicate (i.e., evaluates to true), the Filter transformation removes it from the stream. This is useful for cleaning up unnecessary records before they reach downstream systems.\nFilter Transformation The Filter SMT is configured to drop any record for which the predicate evaluates to true, ensuring that tombstone records are filtered out before they reach downstream systems.\nHow It Works in This Configuration The dropTombstone predicate detects tombstone records (null values) in Kafka topics. The Filter SMT is configured to drop any record where the predicate evaluates to true, effectively filtering out tombstone records. This setup aligns with the Kafka Connect documentation on predicates and transformations, allowing you to handle tombstone records effectively in your data pipeline.\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/12-filtering-tombstone-records-in-kafka-connect/","summary":"\u003cp\u003eKafka Connect provides a flexible way to process streaming data using Single Message Transforms (SMTs). If you need to filter out tombstone records (records with \u003ccode\u003enull\u003c/code\u003e values), you should use the generic \u003cstrong\u003eFilter\u003c/strong\u003e transformation along with the \u003cstrong\u003eRecordIsTombstone\u003c/strong\u003e predicate.\u003c/p\u003e\n\u003cp\u003eHere’s the correct configuration:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-properties\" data-lang=\"properties\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Define the predicate to detect tombstone records (i.e., records with null values)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003epredicates\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003edropTombstone\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003epredicates.dropTombstone.type\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003eorg.apache.kafka.connect.transforms.predicates.RecordIsTombstone\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Configure the Filter transformation to drop records that match the predicate\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003etransforms\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003edropTombstone\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003etransforms.dropTombstone.type\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003eorg.apache.kafka.connect.transforms.Filter\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003etransforms.dropTombstone.predicate\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s\"\u003edropTombstone\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"explanation\"\u003eExplanation\u003c/h2\u003e\n\u003ch3 id=\"what-is-a-predicate\"\u003eWhat is a Predicate?\u003c/h3\u003e\n\u003cp\u003eA \u003cstrong\u003epredicate\u003c/strong\u003e in Kafka Connect is a condition that evaluates whether a given record meets certain criteria. It returns either \u003ccode\u003etrue\u003c/code\u003e or \u003ccode\u003efalse\u003c/code\u003e. If \u003ccode\u003etrue\u003c/code\u003e, the transformation (such as filtering) is applied. In this case, the predicate named \u003ccode\u003edropTombstone\u003c/code\u003e uses the built-in class \u003ccode\u003eRecordIsTombstone\u003c/code\u003e, which evaluates to \u003ccode\u003etrue\u003c/code\u003e when a record’s value is \u003ccode\u003enull\u003c/code\u003e.\u003c/p\u003e","title":"Filtering Tombstone Records in Kafka Connect"},{"content":"In this technical deep-dive, we unravel the power of Server-Sent Events (SSE), a game-changing web technology that simplifies real-time communication. Developers often struggle with complex, resource-intensive methods for creating live updates, but SSE offers an elegant solution that\u0026rsquo;s both lightweight and powerful.\nUnlocking Real-Time Updates with Minimal Complexity Ever wondered how to create a real-time web application without the heavyweight complexity of WebSockets? Let me introduce you to the unsung hero of web communication: Server-Sent Events (SSE).\nWhat are Server-Sent Events? SSE is a lightweight, browser-supported technology that allows a web page to receive automatic updates from a server. Unlike traditional polling or complex WebSocket setups, SSE provides a straightforward way to stream data from the server to the client.\nWhy SSE? Simplicity: Just a few lines of code One-way communication: Perfect for live feeds, notifications, and real-time dashboards Native browser support: No additional libraries required Lightweight: Less overhead compared to WebSockets A Practical Example: System Resource Monitor I\u0026rsquo;ve created a demonstration repository that shows just how easy it is to implement SSE. In this project, we build a real-time system resource monitor that:\nStreams CPU and RAM usage data Updates the dashboard in real-time Uses less than 50 lines of Python code Key Components Flask for the backend Server-Sent Events for real-time streaming Simple HTML/JavaScript for the frontend Code Snapshot: The Magic of SSE def generate_usage(): while True: cpu = psutil.cpu_percent(interval=1) ram = psutil.virtual_memory().percent yield f\u0026#39;data: {{\u0026#34;cpu\u0026#34;: {cpu}, \u0026#34;ram\u0026#34;: {ram}}}\\n\\n\u0026#39; time.sleep(1) # Push updates every second @app.route(\u0026#34;/stream\u0026#34;) def stream(): return Response(generate_usage(), content_type=\u0026#34;text/event-stream\u0026#34;) That\u0026rsquo;s it! Just a few lines to create a real-time data stream.\nWant to See It in Action? Check out the full repository: https://github.com/Joel-hanson/sse-stream\nDive into the code, experiment, and discover how simple real-time web applications can be!\nKey Takeaways SSE is incredibly straightforward Real-time updates don\u0026rsquo;t need to be complicated A few lines of code can create powerful, live applications Happy Coding!\nPro Tip: SSE is perfect for scenarios like live feeds, stock tickers, system monitors, and more!\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/11-build-real-time-web-apps-with-minimal-code/","summary":"In this technical deep-dive, we unravel the power of Server-Sent Events (SSE), a game-changing web technology that simplifies real-time communication. Developers often struggle with complex, resource-intensive methods for creating live updates, but SSE offers an elegant solution that\u0026rsquo;s both lightweight and powerful.","title":"Server-Sent Events: Build Real-Time Web Apps with Minimal Code"},{"content":"In this tutorial, we\u0026rsquo;ll walk through the process of setting up a Kafka connector using Docker and docker-compose.yml. We\u0026rsquo;ll focus on configuring a file connector, which is useful for reading data from files and writing data to files using Kafka.\nLink to repository Prerequisites Docker and Docker Compose installed on your system Basic understanding of Kafka and Docker concepts Project Structure Before we begin, let\u0026rsquo;s look at the files we\u0026rsquo;ll be working with:\ndocker-compose.yml: Defines our services (Kafka and Kafka Connect) Dockerfile: Custom image for Kafka Connect entrypoint.sh: Script to configure and start Kafka Connect plugins.sh: Script to set up the necessary connector plugins custom-plugins/: Directory for storing custom/external connector plugins Step 1: Setting up the Docker Compose File Our docker-compose.yml file defines two services: kafka and connect. Here\u0026rsquo;s a breakdown of the important configurations:\nversion: \u0026#34;3\u0026#34; services: kafka: image: apache/kafka:latest # ... (Kafka configuration) connect: build: context: . dockerfile: Dockerfile depends_on: kafka: condition: service_healthy ports: - 8083:8083 environment: BOOTSTRAP_SERVERS: kafka:9092 REST_ADVERTISED_HOST_NAME: connect REST_PORT: 8083 GROUP_ID: compose-connect-group CONFIG_STORAGE_TOPIC: docker-connect-configs OFFSET_STORAGE_TOPIC: docker-connect-offsets STATUS_STORAGE_TOPIC: docker-connect-status KEY_CONVERTER: \u0026#34;org.apache.kafka.connect.storage.StringConverter\u0026#34; VALUE_CONVERTER: \u0026#34;org.apache.kafka.connect.json.JsonConverter\u0026#34; CONFIG_STORAGE_REPLICATION_FACTOR: 1 OFFSET_STORAGE_REPLICATION_FACTOR: 1 STATUS_STORAGE_REPLICATION_FACTOR: 1 CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY: All CONNECT_PLUGIN_PATH: \u0026#34;/opt/kafka/plugins\u0026#34; volumes: - ./data:/data - ./custom-plugins:/opt/kafka/custom-plugin The connect service is built from our custom Dockerfile and depends on the kafka service. We\u0026rsquo;ve set up environment variables to configure Kafka Connect, including the bootstrap servers, storage topics, and converters.\nStep 2: Creating the Dockerfile Our Dockerfile sets up the Kafka Connect image:\nFROM apache/kafka:latest WORKDIR /opt/kafka EXPOSE 8083 RUN mkdir /opt/kafka/plugins COPY plugins.sh . RUN sh plugins.sh COPY entrypoint.sh . ENTRYPOINT [\u0026#34;./entrypoint.sh\u0026#34;] CMD [\u0026#34;start\u0026#34;] This Dockerfile:\nUses the latest Apache Kafka image as the base Creates a plugins directory Copies and runs the plugins.sh script to set up connectors Copies the entrypoint.sh script Sets the entrypoint to our custom script Step 3: Configuring the Entrypoint Script The entrypoint.sh script is responsible for configuring and starting Kafka Connect. It:\nSets default environment variables Configures logging Processes Kafka Connect properties Starts the Kafka Connect distributed worker Step 4: Setting up Connector Plugins The plugins.sh script copies the necessary connector JARs to the plugins directory:\n#!/bin/usr/env bash set -eo pipefail KAFKA_HOME=${KAFKA_HOME:-/opt/kafka} KAFKA_CONNECT_PLUGINS_DIR=${KAFKA_CONNECT_PLUGINS_DIR:-/opt/kafka/plugins} # Copy the necessary connector JARs cp \u0026#34;$KAFKA_HOME\u0026#34;/libs/connect-file-*.jar \u0026#34;$KAFKA_CONNECT_PLUGINS_DIR\u0026#34; cp \u0026#34;$KAFKA_HOME\u0026#34;/libs/kafka-clients-*.jar \u0026#34;$KAFKA_CONNECT_PLUGINS_DIR\u0026#34; cp \u0026#34;$KAFKA_HOME\u0026#34;/libs/connect-api-*.jar \u0026#34;$KAFKA_CONNECT_PLUGINS_DIR\u0026#34; cp \u0026#34;$KAFKA_HOME\u0026#34;/libs/connect-transforms-*.jar \u0026#34;$KAFKA_CONNECT_PLUGINS_DIR\u0026#34; echo \u0026#34;Kafka File Source and Sink connectors have been set up in $KAFKA_CONNECT_PLUGINS_DIR\u0026#34; # List the installed connector JARs echo \u0026#34;Installed connector JARs:\u0026#34; ls -1 \u0026#34;$KAFKA_CONNECT_PLUGINS_DIR\u0026#34; echo \u0026#34;Done!\u0026#34; This script ensures that the file connector and necessary dependencies are available in the plugins directory.\nStep 5: Running the Setup To start the Kafka and Kafka Connect services:\nMake sure all files (docker-compose.yml, Dockerfile, entrypoint.sh, and plugins.sh) are in the same directory. Create a data directory in the same location for file connector data. Run the following command: docker-compose up -d This command will build the custom Kafka Connect image and start both the Kafka and Kafka Connect services.\nStep 6: Configuring a File Connector Now that our services are running, we can configure a file connector. Here\u0026rsquo;s an example of how to create a file source connector:\nCreate a file named source.txt in the data directory with some sample content.\nUse the following curl command to create the connector:\ncurl -X POST http://localhost:8083/connectors -H \u0026#34;Content-Type: application/json\u0026#34; -d \u0026#39;{ \u0026#34;name\u0026#34;: \u0026#34;file-source\u0026#34;, \u0026#34;config\u0026#34;: { \u0026#34;connector.class\u0026#34;: \u0026#34;org.apache.kafka.connect.file.FileStreamSourceConnector\u0026#34;, \u0026#34;file\u0026#34;: \u0026#34;/data/source.txt\u0026#34;, \u0026#34;topic\u0026#34;: \u0026#34;file-topic\u0026#34; } }\u0026#39; This will create a source connector that reads from /data/source.txt and writes to the file-topic Kafka topic.\nTo create a file sink connector, use:\ncurl -X POST http://localhost:8083/connectors -H \u0026#34;Content-Type: application/json\u0026#34; -d \u0026#39;{ \u0026#34;name\u0026#34;: \u0026#34;file-sink\u0026#34;, \u0026#34;config\u0026#34;: { \u0026#34;connector.class\u0026#34;: \u0026#34;org.apache.kafka.connect.file.FileStreamSinkConnector\u0026#34;, \u0026#34;file\u0026#34;: \u0026#34;/data/sink.txt\u0026#34;, \u0026#34;topics\u0026#34;: \u0026#34;file-topic\u0026#34; } }\u0026#39; This will create a sink connector that reads from the file-topic Kafka topic and writes to /data/sink.txt.\nStep 7: Verifying the Setup After setting up your connectors, it\u0026rsquo;s important to verify that everything is working correctly. Here are some ways to do that:\nChecking Topics To verify if the topic is created, you can list all topics:\ndocker-compose exec kafka ./opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092 You should see file-topic in the list of topics.\nConsuming Messages To verify that messages are being produced to the topic, you can use the Kafka console consumer:\ndocker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic file-topic --from-beginning This command will display all messages in the file-topic from the beginning.\nChecking Connector Status You can use the Kafka Connect REST API to check the status of your connectors. Here are some useful curl commands:\nList all connectors:\ncurl http://localhost:8083/connectors Check the status of a specific connector (e.g., file-source):\ncurl http://localhost:8083/connectors/file-source/status Check the offsets of a specific connector:\ncurl http://localhost:8083/connectors/file-source/offsets These commands will help you verify that your connectors are running correctly and processing data as expected.\nAdding Custom Connector Plugins While our setup includes the file connector by default, you might want to add other custom connector plugins. You can do this easily using Docker volumes. Here\u0026rsquo;s how:\nCreate a directory for your custom plugins:\nmkdir -p custom-plugins Add your custom connector JAR files to this directory.\nModify the docker-compose.yml file to add a volume for the custom plugins. Add the following under the volumes section of the connect service:\nvolumes: - ./data:/data - ./custom-plugins:/opt/kafka/custom-plugins Update the CONNECT_PLUGIN_PATH environment variable in the docker-compose.yml file:\nenvironment: # ... other environment variables ... CONNECT_PLUGIN_PATH: \u0026#34;/opt/kafka/plugins,/opt/kafka/custom-plugins\u0026#34; Restart your services:\ndocker-compose down docker-compose up -d Verify the plugins:\ncurl http://localhost:8083/connector-plugins/ Now, Kafka Connect will load any connector plugins found in your custom-plugins directory.\nExample: Adding the Aiven JDBC connector:\nDownload the connector JAR from the Aiven JDBC connector releases. Place the JAR file in your custom-plugins directory. Restart the services as described above. You should now be able to use the JDBC connector in your Kafka Connect setup. Remember to check the documentation of any custom connectors you add, as they may require additional configuration or dependencies.\nConclusion In this tutorial, we\u0026rsquo;ve walked through the process of setting up Kafka and Kafka Connect using Docker and docker-compose.yml. We\u0026rsquo;ve also demonstrated how to configure file source and sink connectors. This setup provides a flexible foundation for building data pipelines with Kafka, allowing you to easily connect various data sources and sinks to your Kafka cluster.\nFor a more comprehensive approach to Kafka connector development, including extensive integration testing with TestContainers, check out our Kafka Connector Template which includes a complete testing framework based on proven patterns from production connectors.\nRemember to monitor your connectors and Kafka cluster for performance and adjust configurations as needed for your specific use case.\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/09-kafka-connector-docker-setup/","summary":"\u003cp\u003eIn this tutorial, we\u0026rsquo;ll walk through the process of setting up a Kafka connector using Docker and docker-compose.yml. We\u0026rsquo;ll focus on configuring a file connector, which is useful for reading data from files and writing data to files using Kafka.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003ch4 id=\"link-to-repository\"\u003e\u003cem\u003e\u003ca href=\"https://github.com/Joel-hanson/kafka-connector-docker-setup\"\u003eLink to repository\u003c/a\u003e\u003c/em\u003e\u003c/h4\u003e\u003c/blockquote\u003e\n\u003ch2 id=\"prerequisites\"\u003ePrerequisites\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003eDocker and Docker Compose installed on your system\u003c/li\u003e\n\u003cli\u003eBasic understanding of Kafka and Docker concepts\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"project-structure\"\u003eProject Structure\u003c/h2\u003e\n\u003cp\u003eBefore we begin, let\u0026rsquo;s look at the files we\u0026rsquo;ll be working with:\u003c/p\u003e","title":"Configuring Kafka Connector with Docker: A Step-by-Step Guide"},{"content":"Introduction In this post, we will walk through the process of setting up an Apache Kafka cluster without using Zookeeper, leveraging Kafka\u0026rsquo;s KRaft mode for metadata management. Kafka no longer requires a Zookeeper instance, simplifying the cluster setup. We will use Docker to deploy this Kafka cluster with the latest version of Kafka.\nPrerequisites Ensure that you have the following:\nDocker installed Docker compose installed Basic understanding of Kafka concepts (e.g., brokers, partitions) Step 1: Kafka Cluster Setup Let’s use the following Docker Compose configuration to set up the Kafka broker in KRaft mode:\nversion: \u0026#39;3\u0026#39; services: broker: image: apache/kafka:latest container_name: broker environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller # The node acts as both broker and controller KAFKA_LISTENERS: PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093 # Ports for broker and controller KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 # External listener for clients KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER # Listener name for inter-controller communication KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT # Security protocol for listeners KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093 # Quorum voters for the controller in KRaft mode KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 # Replication factor for the offsets topic KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 # Replication factor for transaction logs KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 # Minimum in-sync replicas for transactional logs KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 # Initial delay before consumer group rebalancing KAFKA_NUM_PARTITIONS: 3 # Default number of partitions for new topics ports: - \u0026#34;9092:9092\u0026#34; # Port for Kafka broker healthcheck: test: [ \u0026#34;CMD-SHELL\u0026#34;, \u0026#34;./opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 \u0026gt; /dev/null 2\u0026gt;\u0026amp;1\u0026#34;, ] interval: 10s timeout: 10s retries: 5 Step 2: Breakdown of Kafka Configuration Let’s break down each configuration setting:\nKAFKA_NODE_ID: Specifies the unique ID of the node in the Kafka cluster. In KRaft mode, this ID identifies both brokers and controllers.\nKAFKA_PROCESS_ROLES: Defines the roles of the node. In this setup, the node will act as both a broker and a controller.\nKAFKA_LISTENERS: Specifies the endpoints that Kafka will bind to for communication. Here, we use port 9092 for the broker and 9093 for the controller.\nKAFKA_ADVERTISED_LISTENERS: This is the address that Kafka will advertise to clients for connecting to the broker. External clients use this address to connect.\nKAFKA_CONTROLLER_LISTENER_NAMES: Defines the listener name used for controller-to-controller communication. This is set to CONTROLLER for KRaft mode.\nKAFKA_LISTENER_SECURITY_PROTOCOL_MAP: Maps the listeners to their respective security protocols. In this case, both the controller and broker listeners use PLAINTEXT (i.e., no encryption).\nKAFKA_CONTROLLER_QUORUM_VOTERS: Specifies the quorum voters in the KRaft mode, which manage the metadata. Here, we define one controller voter on localhost:9093.\nKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: Sets the replication factor for the Kafka consumer offsets topic, which tracks consumer progress. For simplicity, it\u0026rsquo;s set to 1 in this single-node setup.\nKAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: Defines the replication factor for Kafka\u0026rsquo;s transactional state logs. Since we are running a single broker, this is set to 1.\nKAFKA_TRANSACTION_STATE_LOG_MIN_ISR: Specifies the minimum number of in-sync replicas required for transactional state logs. This is also set to 1 in a single-node cluster.\nKAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: Defines the initial delay (in milliseconds) before rebalancing consumer groups. Setting it to 0 minimizes the delay in group rebalancing.\nKAFKA_NUM_PARTITIONS: Defines the default number of partitions for new topics. We set this to 3, but you can adjust it based on your needs.\nStep 3: Running the Kafka Cluster To bring up the Kafka broker, follow these steps:\nSave the above configuration into a docker-compose.yml file.\nRun the following command in the same directory:\ndocker-compose up -d To verify that the Kafka broker is running, you can check the logs:\ndocker logs broker Step 4: Interacting with Kafka You can now interact with your Kafka cluster. For example, to create a topic or send messages, you can use Kafka CLI tools. Here’s how you can produce and consume messages.\nProducing a message docker exec -it broker ./opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic Consuming messages docker exec -it broker ./opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning Conclusion In this blog post, we demonstrated how to set up a Kafka cluster without Zookeeper using KRaft mode and Docker. KRaft simplifies the architecture by eliminating the need for an external Zookeeper service, making it easier to manage and scale Kafka clusters.\nPlease refer to the official docker page for more detailed information.\nFeel free to experiment with this setup and adjust the configuration for your specific use case!\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/08-setting-up-a-kafka-cluster-without-zookeeper-using-docker/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eIn this post, we will walk through the process of setting up an Apache Kafka cluster \u003cstrong\u003ewithout using Zookeeper\u003c/strong\u003e, leveraging Kafka\u0026rsquo;s \u003cstrong\u003eKRaft mode\u003c/strong\u003e for metadata management. Kafka no longer requires a Zookeeper instance, simplifying the cluster setup. We will use \u003cstrong\u003eDocker\u003c/strong\u003e to deploy this Kafka cluster with the latest version of Kafka.\u003c/p\u003e\n\u003ch3 id=\"prerequisites\"\u003ePrerequisites\u003c/h3\u003e\n\u003cp\u003eEnsure that you have the following:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDocker installed\u003c/li\u003e\n\u003cli\u003eDocker compose installed\u003c/li\u003e\n\u003cli\u003eBasic understanding of Kafka concepts (e.g., brokers, partitions)\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"step-1-kafka-cluster-setup\"\u003eStep 1: Kafka Cluster Setup\u003c/h3\u003e\n\u003cp\u003eLet’s use the following \u003cstrong\u003eDocker Compose\u003c/strong\u003e configuration to set up the Kafka broker in \u003cstrong\u003eKRaft mode\u003c/strong\u003e:\u003c/p\u003e","title":"Setting Up a Kafka Cluster Without Zookeeper Using Docker"},{"content":"In this blog post, we\u0026rsquo;ll explore the concept of online machine learning using a real-world example: classifying images from the CIFAR10 dataset in real-time using Apache Kafka for data streaming. We\u0026rsquo;ll walk through the entire process, from training the initial model to setting up a streaming pipeline for continuous learning and prediction.\nWhat is Online Machine Learning? Online machine learning is an approach where the model learns incrementally as new data becomes available, rather than training on a fixed dataset. This method is particularly useful for:\nHandling large-scale datasets that don\u0026rsquo;t fit in memory Adapting to changing patterns in the data over time Making predictions in real-time as new data arrives Our example will demonstrate a simplified version of online learning, focusing on the real-time prediction aspect.\nProject Overview Our project consists of three main components:\nModel Training: We\u0026rsquo;ll train a Convolutional Neural Network (CNN) on the CIFAR10 dataset. Data Producer: A script that sends CIFAR10 images to a Kafka topic. Stream Processor: A consumer that reads images from Kafka and makes real-time predictions. Let\u0026rsquo;s dive into each component.\n1. Model Training First, we\u0026rsquo;ll train our CNN model on the CIFAR10 dataset. Here\u0026rsquo;s a breakdown of the key parts of our main.py script:\nimport torch import torch.nn as nn import torch.optim as optim import torchvision import torchvision.transforms as transforms # ... (setup code omitted for brevity) class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.conv1 = nn.Conv2d(3, 32, 3, padding=1) self.pool = nn.MaxPool2d(2, 2) self.conv2 = nn.Conv2d(32, 64, 3, padding=1) self.conv3 = nn.Conv2d(64, 64, 3, padding=1) self.fc1 = nn.Linear(64 * 4 * 4, 64) self.fc2 = nn.Linear(64, 10) def forward(self, x): # ... (forward pass implementation) # ... (training loop and testing code) if __name__ == \u0026#34;__main__\u0026#34;: net = Net().to(device) criterion = nn.CrossEntropyLoss() optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE) train(device, trainloader, net, criterion, optimizer) testing(device, testloader, net) This script defines a CNN architecture, trains it on the CIFAR10 dataset, and saves the trained model to a file. The model achieves decent accuracy on the test set, which we\u0026rsquo;ll use for our online prediction system.\n2. Data Producer The producer.py script simulates a stream of incoming images by sending CIFAR10 images to a Kafka topic:\nfrom kafka import KafkaProducer from torchvision.datasets import CIFAR10 from torchvision.transforms import Compose, Normalize, ToTensor # ... (setup code omitted for brevity) def send_image(image, label): image_bytes = image.numpy().tobytes() image_base64 = base64.b64encode(image_bytes).decode(\u0026#34;utf-8\u0026#34;) message = {\u0026#34;image\u0026#34;: image_base64, \u0026#34;label\u0026#34;: label.item(), \u0026#34;id\u0026#34;: str(uuid4())} producer.send(TOPIC, json.dumps(message).encode(\u0026#34;utf-8\u0026#34;)) producer.flush() logger.info(\u0026#34;Produced image\u0026#34;) if __name__ == \u0026#34;__main__\u0026#34;: for i, data in enumerate(trainloader, 0): images, labels = data send_image(images[0], labels[0]) time.sleep(5) # Send messages with 5 second delay This script loads the CIFAR10 dataset and sends each image to a Kafka topic, simulating a stream of incoming data.\n3. Stream Processor (prediction) The consumer.py script consumes images from the Kafka topic and makes real-time predictions:\nfrom kafka import KafkaConsumer import torch import torch.nn as nn # ... (setup code omitted for brevity) model = Net() model.load_state_dict(torch.load(MODEL_PATH)) model.eval() for message in consumer: data = json.loads(message.value.decode(\u0026#34;utf-8\u0026#34;)) image_bytes = base64.b64decode(data[\u0026#34;image\u0026#34;]) image = np.frombuffer(image_bytes, dtype=np.float32).reshape(3, 32, 32) image = torch.from_numpy(image) img_tensor = preprocess(image).unsqueeze(0) with torch.no_grad(): output = model(img_tensor) _, predicted = torch.max(output, 1) logger.info(f\u0026#34;Classified as: {class_names[predicted.item()]}\u0026#34;) This script loads the pre-trained model, consumes images from the Kafka topic, and makes predictions in real-time.\nRunning the Online Learning System To run this online learning system:\nStart your Kafka broker. Run the consumer.py script to start the consumer and start doing prediction. Run the producer.py script to start sending images to Kafka. You\u0026rsquo;ll see real-time predictions being made as images are consumed from the Kafka topic.\nExtending to True Online Learning While our initial setup demonstrates real-time prediction, it doesn\u0026rsquo;t update the model with new data. Let\u0026rsquo;s extend this to true online learning by implementing the following steps:\nAccumulate a batch of new images and their true labels. Periodically retrain the model on this new data. Update the model in the stream processor. This would allow the model to adapt to new patterns in the data over time.\nHere\u0026rsquo;s how we can modify our consumer.py script to achieve this:\nimport base64 import json import logging import sys import types from collections import deque from threading import Thread import time import torch import torch.nn as nn import torch.optim as optim from kafka import KafkaConsumer from torchvision import transforms # Assuming the Net class is defined as before from main import Net # ... (setup code omitted for brevity) # Data accumulator data_buffer = deque(maxlen=BATCH_SIZE) def accumulate_data(image, label): \u0026#34;\u0026#34;\u0026#34;Accumulate new data for retraining\u0026#34;\u0026#34;\u0026#34; data_buffer.append((image, label)) def retrain_model(): \u0026#34;\u0026#34;\u0026#34;Retrain the model on accumulated data\u0026#34;\u0026#34;\u0026#34; global model if len(data_buffer) \u0026lt; BATCH_SIZE: logger.info(\u0026#34;Not enough data for retraining\u0026#34;) return logger.info(\u0026#34;Retraining model...\u0026#34;) criterion = nn.CrossEntropyLoss() optimizer = optim.Adam(model.parameters(), lr=0.001) model.train() for epoch in range(5): # Do a few epochs of training running_loss = 0.0 for i, (image, label) in enumerate(data_buffer): inputs = preprocess(image).unsqueeze(0) labels = torch.tensor([label]) optimizer.zero_grad() outputs = model(inputs) loss = criterion(outputs, labels) loss.backward() optimizer.step() running_loss += loss.item() logger.info(f\u0026#34;Epoch {epoch+1}, Loss: {running_loss/len(data_buffer):.3f}\u0026#34;) model.eval() logger.info(\u0026#34;Model updated\u0026#34;) # Save the updated model torch.save(model.state_dict(), MODEL_PATH) def update_model_periodically(): \u0026#34;\u0026#34;\u0026#34;Periodically retrain the model\u0026#34;\u0026#34;\u0026#34; while True: time.sleep(UPDATE_INTERVAL) retrain_model() # Start a thread for periodic model updates update_thread = Thread(target=update_model_periodically) update_thread.daemon = True update_thread.start() # Main loop for processing incoming data for message in consumer: # ... (message processing code) # Accumulate new data accumulate_data(image, label) # Make prediction # ... (prediction code) # Check prediction accuracy if predicted.item() == label: logger.info(\u0026#34;Correct prediction!\u0026#34;) else: logger.info(\u0026#34;Incorrect prediction.\u0026#34;) logger.info(f\u0026#34;Current buffer size: {len(data_buffer)}\u0026#34;) Let\u0026rsquo;s break down the key components of this true online learning implementation:\n1. Accumulating New Data We use a deque with a fixed maximum length to store incoming images and their labels:\ndata_buffer = deque(maxlen=BATCH_SIZE) def accumulate_data(image, label): data_buffer.append((image, label)) This ensures we always have the most recent data for retraining.\n2. Periodic Retraining We implement a retrain_model function that performs a few epochs of training on the accumulated data:\ndef retrain_model(): # ... (function implementation) This function is called periodically by a separate thread:\ndef update_model_periodically(): while True: time.sleep(UPDATE_INTERVAL) retrain_model() update_thread = Thread(target=update_model_periodically) update_thread.daemon = True update_thread.start() 3. Updating the Model After retraining, we update the global model and save it to disk:\nmodel.eval() logger.info(\u0026#34;Model updated\u0026#34;) torch.save(model.state_dict(), MODEL_PATH) This ensures that future predictions use the most up-to-date model.\nBenefits of True Online Learning This implementation of true online learning offers several advantages:\nAdaptability: The model can adapt to changes in the data distribution over time. Continuous Improvement: As more data is processed, the model\u0026rsquo;s performance can potentially improve. Efficient Resource Use: By training on batches of recent data, we avoid the need to store and retrain on the entire dataset. Considerations and Potential Improvements While this implementation demonstrates the basics of online learning, there are several ways to enhance and optimize it:\nValidation: Implement a separate validation set to monitor the model\u0026rsquo;s performance over time. Adaptive Learning Rate: Adjust the learning rate based on the model\u0026rsquo;s performance or the amount of data processed. Concept Drift Detection: Implement mechanisms to detect when the underlying data distribution changes significantly. Incremental Learning Algorithms: Explore more sophisticated online learning algorithms designed for streaming data. Model Versioning: Implement a system to track different versions of the model and potentially roll back if performance degrades. Conclusion This project demonstrates the basics of setting up an online machine learning system using Kafka for data streaming and PyTorch for image classification. While our example focuses on real-time prediction, the same principles can be applied to create a fully online learning system that continuously updates and improves its model.\nBy extending our initial real-time prediction system to include true online learning capabilities, we\u0026rsquo;ve created a more dynamic and adaptive machine learning pipeline. This system can continuously improve its performance as it processes new data, making it well-suited for real-world applications where data patterns may change over time.\nRemember that implementing online learning in production environments comes with its own set of challenges, including ensuring data quality, managing computational resources, and monitoring model performance. Always thoroughly test and validate your online learning system before deploying it in critical applications.\nAs you continue to explore online machine learning, consider diving deeper into topics like concept drift, incremental learning algorithms, and techniques for efficiently managing and updating models in distributed systems\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/07-online-machine-learning-real-time-image-classification-with-cifar10-dataset-and-kafka/","summary":"This blog post explores online machine learning through a practical example using the CIFAR10 dataset and Apache Kafka.","title":"Online Machine Learning: Real-time Image Classification with CIFAR10 Dataset and Kafka"},{"content":"Introduction Imagine having a conversation with an AI that not only understands your questions but also draws upon a vast repository of knowledge to provide accurate, contextually relevant answers. This isn\u0026rsquo;t science fiction—it\u0026rsquo;s the power of Retrieval-Augmented Generation (RAG) systems. In this blog post, we\u0026rsquo;ll dive into the exciting world of RAG and show you how to harness this technology using PostgreSQL and pgvector.\nWhat is RAG, and Why Should You Care? RAG is like giving your AI a photographic memory and the ability to instantly recall relevant information. It combines the power of large language models with a knowledge base, allowing the AI to provide more accurate, up-to-date, and contextually relevant responses. Whether you\u0026rsquo;re building a chatbot, a question-answering system, or any AI application that requires access to specific knowledge, RAG is your secret weapon.\nPrerequisites Before we embark on this RAG adventure, make sure you have:\nPostgreSQL 13 or later Python 3.7 or later pip (Python package manager) Docker (for building our custom PostgreSQL image) Step 1: Building a Custom PostgreSQL Image with pgvector Let\u0026rsquo;s start by creating a supercharged PostgreSQL image with the pgvector extension pre-installed. This will save us time and ensure consistency across different environments.\nCreate a new directory for your Dockerfile: mkdir postgres-pgvector \u0026amp;\u0026amp; cd postgres-pgvector Create a file named Dockerfile with the following content: FROM postgres:14 ENV POSTGRES_DB=vectordb ENV POSTGRES_USER=vectoruser ENV POSTGRES_PASSWORD=vectorpass RUN apt-get update \u0026amp;\u0026amp; apt-get install -y \\ build-essential \\ git \\ postgresql-server-dev-14 RUN git clone https://github.com/pgvector/pgvector.git RUN cd pgvector \u0026amp;\u0026amp; \\ make \u0026amp;\u0026amp; \\ make install RUN apt-get remove -y build-essential git postgresql-server-dev-14 \u0026amp;\u0026amp; \\ apt-get autoremove -y \u0026amp;\u0026amp; \\ apt-get clean \u0026amp;\u0026amp; \\ rm -rf /var/lib/apt/lists/* /pgvector RUN echo \u0026#34;CREATE EXTENSION vector;\u0026#34; \u0026gt; /docker-entrypoint-initdb.d/init.sql Build the Docker image: docker build -t postgres-pgvector . Run the container: docker run -d --name postgres-vector -p 5432:5432 postgres-pgvector You now have a PostgreSQL instance running with pgvector ready to go.\nStep 2: Install Required Python Packages Let\u0026rsquo;s set up our Python environment:\npip install psycopg2-binary pgvector sentence-transformers torch transformers Step 3: Create a Table for Vector Storage Connect to your PostgreSQL database and run:\nCREATE TABLE documents ( id SERIAL PRIMARY KEY, content TEXT, embedding vector(384) ); Step 4: Inserting Documents and Embeddings Now, let\u0026rsquo;s populate our database with some interesting facts:\nimport psycopg2 from sentence_transformers import SentenceTransformer import numpy as np model = SentenceTransformer(\u0026#39;all-MiniLM-L6-v2\u0026#39;) # Connect to the PostgreSQL database conn = psycopg2.connect(\u0026#34;dbname=vectordb user=vectoruser password=vectorpass host=localhost\u0026#34;) cur = conn.cursor() # Sample documents documents = [ \u0026#34;The Great Wall of China is visible from space, but it\u0026#39;s a common myth that it can be seen from the moon.\u0026#34;, \u0026#34;Honey never spoils. Archaeologists have found pots of honey in ancient Egyptian tombs that are over 3,000 years old and still perfectly edible.\u0026#34;, \u0026#34;The world\u0026#39;s oldest known living tree is a Great Basin Bristlecone Pine that is over 4,800 years old.\u0026#34;, \u0026#34;Octopuses have three hearts: two pump blood through each of the two gills, while the third pumps blood through the body.\u0026#34;, \u0026#34;The shortest war in history lasted only 38 minutes. It was between Britain and Zanzibar on August 27, 1896.\u0026#34; ] # Insert documents and their embeddings for doc in documents: embedding = model.encode(doc) cur.execute(\u0026#34;INSERT INTO documents (content, embedding) VALUES (%s, %s)\u0026#34;, (doc, embedding.tolist())) conn.commit() cur.close() conn.close() print(\u0026#34;Documents inserted successfully!\u0026#34;) Step 5: Implementing RAG Query Now for the exciting part—let\u0026rsquo;s implement our RAG system:\nimport psycopg2 from sentence_transformers import SentenceTransformer from transformers import AutoModelForCausalLM, AutoTokenizer import torch # Initialize models embedding_model = SentenceTransformer(\u0026#39;all-MiniLM-L6-v2\u0026#39;) llm_model = AutoModelForCausalLM.from_pretrained(\u0026#34;gpt2\u0026#34;) tokenizer = AutoTokenizer.from_pretrained(\u0026#34;gpt2\u0026#34;) # Connect to the PostgreSQL database conn = psycopg2.connect(\u0026#34;dbname=vectordb user=vectoruser password=vectorpass host=localhost\u0026#34;) cur = conn.cursor() def query_similar_documents(query, top_k=1): query_embedding = embedding_model.encode(query) cur.execute(\u0026#34;\u0026#34;\u0026#34; SELECT content, 1 - (embedding \u0026lt;=\u0026gt; %s) AS similarity FROM documents ORDER BY similarity DESC LIMIT %s \u0026#34;\u0026#34;\u0026#34;, (query_embedding.tolist(), top_k)) return cur.fetchall() def generate_response(query, context): prompt = f\u0026#34;Context: {context}\\n\\nQuestion: {query}\\n\\nAnswer:\u0026#34; input_ids = tokenizer.encode(prompt, return_tensors=\u0026#34;pt\u0026#34;) with torch.no_grad(): output = llm_model.generate(input_ids, max_length=100, num_return_sequences=1, no_repeat_ngram_size=2) return tokenizer.decode(output[0], skip_special_tokens=True) # Example usage user_query = \u0026#34;Tell me an interesting fact about nature.\u0026#34; similar_docs = query_similar_documents(user_query) context = similar_docs[0][0] # Use the most relevant document as context response = generate_response(user_query, context) print(f\u0026#34;User Query: {user_query}\u0026#34;) print(f\u0026#34;Context: {context}\u0026#34;) print(f\u0026#34;AI Response: {response}\u0026#34;) cur.close() conn.close() Bringing It All Together Congratulations! You\u0026rsquo;ve just built a RAG system that can:\nStore and retrieve vector embeddings of documents efficiently using PostgreSQL and pgvector. Find the most relevant context for a given query. Generate human-like responses using a language model, augmented with retrieved information. This system forms the foundation for creating intelligent chatbots, question-answering systems, and other AI applications that can tap into specific knowledge bases.\nHappy coding!\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/06-setting-up-rag-with-postgre-sql-and-pgvector/","summary":"Learn how to implement a powerful Retrieval-Augmented Generation (RAG) system using PostgreSQL and pgvector. This comprehensive guide covers everything from setting up a custom PostgreSQL Docker image to creating a fully functional RAG query system with vector embeddings and language model inference.","title":"Supercharging Your AI: Setting Up RAG with PostgreSQL and pgvector"},{"content":"Using git as a code versioning tool is a day-to-day activity for developers, and some of you may be practicing your git workflow through the command line. Preferably with a dark theme too, right?\nAlthough a GUI for git might come in handy for an integrated development environment (IDE) such as IntelliJ, or VS Code, you may at times need to resort to the command line interface (CLI).\nIn this article, we’re have gathered git aliases that can help with a faster and more productive git workflow as an individual, or within a team.\nHow to set a git alias? Your git aliases are often stored per your user’s configuration at ~/.gitconfig. You can also manually set aliases using, for example, the command git config alias.s 'status -s'.\nNext, we’ll cover the git aliases, which you should add to your ~/.gitconfig file in a specific [alias] section, where all aliases and their commands are stored. Check out below some examples of git aliases configuration in that file.\nHow to list git aliases? The git executable has a command line option to list all of git configuration and where they originate from (system, user, or local). The following git command will list all git configuration verbosely with their origins: git config --list --show-origin.\nGit Aliases Git status Do you often run a git status command? I do! Use the following git alias to have a shortcut for it:\n[alias] s = status -s ss = status -s Here it is in action:\n$ git s On branch main Your branch is up to date with \u0026#39;origin/main. nothing to commit, working tree clean Git checkout Moving back and forth between git branches is also something we often do. Let’s shorten that!\nAdd the alias:\n[alias] co = checkout And then give it a try:\ngit co feat/add-popup In fact, a git branch checkout can be done using a shortcut, such as git checkout - which checks out the previous branch you were on. It is actually a shorthand of git checkout @{-1} so this alias can use it as well using git co -.\nCreate a new branch and switch to it If you find yourself often creating new git branches from the CLI, here’s one to save you some keystrokes:\n[alias] cob = checkout -b Use it as follows:\n$ git cob feat/add-popup Switched to a new branch \u0026#39;feat/add-popup\u0026#39; Delete a branch Removing a branch perhaps isn’t something you often do, but when you have to, it might be time-consuming to Google through git commands to find the command argument and flag it. Instead, we can create a shorter and more memorable git branch checkout-related alias:\n[alias] del = branch -D Use it as follows:\n$ git del feat/add-popup Deleted branch featureB (was b5cbv113). List all branches When we work on multiple features, our git workflow usually ends up with multiple git branches as well. Finding where we last left-off and which branch we used could be challenging.\nThe following git alias will list all branches and sort them by commit date, showing the most recent git branch first, based on commits made to it.\n[alias] br = branch --format=\u0026#39;%(HEAD) %(color:yellow)%(refname:short)%(color:reset) - %(contents:subject) %(color:green)(%(committerdate:relative)) [%(authorname)]\u0026#39; --sort=-committerdate Here is how to use it and what the output might look like:\n$ git br wordpress-sync/blog-git-branch-code As you can see, it provides more context, such as the name of the git commit author on each of the last commits, with colorful highlights.\nGit commit Committing always and early is one of git’s strengths. It is fast and adds no overhead due to the way git internals work. So, why not do it often?\nIt really depends on your specific git workflow—however, if you work on git workflows often and want to save your commits just as often (so you don’t lose track of work being done), the commit message text may be less of a priority, especially if you’re working within a pull request workflow.\nTo allow for a faster and more productive git workflow of saving commit status points, the following git alias is a quick one to add all git unstaged files into the staging area with a generic commit message.\n[alias] save = !git add -A \u0026amp;\u0026amp; git commit -m \u0026#39;chore: commit save point\u0026#39; Here is how it looks like in action:\n$ git save [feat/add-popup 98cba110acb] chore: commit save point 1 file changed, 1 insertion(+) Rollback changes If you find yourself, at times, needing to commit changes but then do a rollback, the following alias will roll all of your commit changes back into the local unstaged area, so you can make modifications and add them to the staging area (often referred to as the index).\n[alias] undo = reset HEAD~1 --mixed Once you invoke this git alias, use git undo to reset the previous commit on this branch, and check out all the previous committed changes as uncommitted, so you can resume work on them.\nUse it as follows:\ngit undo Clean all changes We all wish for a fresh start, don’t we? When in need of a quick clean up of all of your local changes which you wish to avoid committing to the repository at this stage, even locally, use the following alias:\n[alias] res = !git reset --hard It resets all staged changes (those that you added with the git add command). Here is how it looks like in action:\n$ git res HEAD is now at 6e7879bc81a chore: commit save point Push changes to upstream By the time you finish working on the code, you send commits upstream to the git repository like all of us, right? We often do that using a lengthy command: git push origin or git push origin main.\nHowever, if you’re working on the same branch name as the one you wish to push changes to, then we can alias the remote branch name as HEAD. Push your commits from the local git repository to the origin or upstream remotes with a shortcut as simple as git done using this alias:\n[alias] done = !git push origin HEAD Using this alias and the shortcut of HEAD is often useful when the branch names are long. Here is how it is easily used:\ngit done Git log To wrap up, we’re going to feature one of the most useful git commands: git log. There are many different ways of using git aliases to construct log output but my suggestion is to use the following customized git alias:\n[alias] lg = !git log --pretty=format:\\\u0026#34;%C(magenta)%h%Creset -%C(red)%d%Creset %s %C(dim green)(%cr) [%an]\\\u0026#34; --abbrev-commit -30 This git alias will make git log print an output that is considerably more readable. Try it out:\n$ git log wordpress-sync/blog-git-log Certainly! Here\u0026rsquo;s the updated Markdown document with explanations for each Git alias:\nOne-line Log [alias] l = log --pretty=format:\u0026#34;%C(yellow)%h\\\\ %ad%Cred%d\\\\ %Creset%s%Cblue\\\\ [%cn]\u0026#34; --decorate --date=short This alias provides a concise one-line log output showing commit hash, date, branch, commit message, and author name.\nAdd [alias] a = add This alias is a shorthand for the git add command, allowing you to stage changes for commit more quickly.\nAdd Patch [alias] ap = add -p This alias is for adding changes interactively, allowing you to selectively stage portions of changed files.\nCommit [alias] c = commit --verbose This alias is a shorthand for committing changes with verbose output, providing more details about the commit.\nCommit All [alias] ca = commit -a --verbose This alias is for committing all changes, including untracked files, with verbose output.\nCommit Message [alias] cm = commit -m This alias allows you to commit changes with a specified message without opening an editor.\nCommit All with Message [alias] cam = commit -a -m This alias combines committing all changes with a specified message, useful for quick commits.\nAmend [alias] m = commit --amend --verbose This alias allows you to amend the previous commit with new changes while maintaining the same commit message.\nDiff [alias] d = diff This alias is a shorthand for the git diff command, showing changes between commits, commit and working tree, etc.\nDiff Stat [alias] ds = diff --stat This alias provides a summarized view of changes, showing which files were modified and how many lines were added or removed.\nDiff Cached [alias] dc = diff --cached This alias shows changes in the staging area compared to the last commit, useful for reviewing staged changes before committing.\nStatus Short [alias] s = status -s This alias provides a concise, short summary of the current repository status, showing which files are modified, staged, or untracked.\nCheckout [alias] co = checkout cob = checkout -b This alias is a shorthand for switching branches or restoring files from the repository.\nCheckout and Create Branch [alias] cob = checkout -b This alias combines creating a new branch and checking it out in one command.\nBranches Sorted by Last Modified [alias] b = \u0026#34;!git for-each-ref --sort=\u0026#39;-authordate\u0026#39; --format=\u0026#39;%(authordate)%09%(objectname:short)%09%(refname)\u0026#39; refs/heads | sed -e \u0026#39;s-refs/heads/--\u0026#39;\u0026#34; This alias lists all branches sorted by the last commit date, providing insight into the most recently modified branches.\nList Aliases [alias] la = \u0026#34;!git config -l | grep alias | cut -c 7-\u0026#34; Branch Management For working with branches, these aliases can save you a lot of time:\n[alias] br = branch --format=\u0026#39;%(HEAD) %(color:yellow)%(refname:short)%(color:reset) - %(contents:subject) %(color:green)(%(committerdate:relative)) [%(authorname)]\u0026#39; --sort=-committerdate The alias br in the given command is used to list branches in a Git repository with a customized format for displaying each branch\u0026rsquo;s information.\nCommitting frequently Committing frequently? These will help:\n[alias] save = !git add -A \u0026amp;\u0026amp; git commit -m \u0026#39;chore: savepoint\u0026#39; [alias] m = commit --amend --verbose m: allows you to modify the most recent commit with verbose output, letting you change the commit message and include any staged changes. Log A prettier and more readable log output:\n[alias] lg = !git log --pretty=format:\\\u0026#34;%C(magenta)%h%Creset -%C(red)%d%Creset %s %C(dim green)(%cr) [%an]\\\u0026#34; --abbrev-commit -30 l = log --pretty=format:\u0026#34;%C(yellow)%h\\\\ %ad%Cred%d\\\\ %Creset%s%Cblue\\\\ [%cn]\u0026#34; --decorate --date=short lg: displays the last 30 commits in a compact, colorful format, showing the commit hash, commit message, relative commit time, and author. l: displays a one-line log with the commit hash, short date, ref names, commit message, and committer name, all highlighted with colors for better readability. Push Push your changes with ease:\n[alias] done = !git push origin HEAD pushitgood = push -u origin --all po = !echo \u0026#39;Ah push it\u0026#39; \u0026amp;\u0026amp; git push origin \u0026amp;\u0026amp; echo \u0026#39;PUSH IT REAL GOOD\u0026#39; done: Pushes the current branch to the remote repository named origin. pushitgood: Pushes all local branches to the remote repository named origin and sets the upstream tracking for the branches. po: Pushes the current branch to the remote repository named origin, with playful messages displayed before and after the push. Reset Need a fresh start? Clean up with these:\n[alias] undo = reset HEAD~1 --mixed res = !git reset --hard undo-commit = reset --soft HEAD~1 undo: Reverts the last commit, keeping changes in the working directory unstaged. res: Resets the working directory and index to the last commit, discarding all changes. undo-commit: Reverts the last commit, keeping changes staged in the index. Remote and Submodules Manage remotes and submodules efficiently:\n[alias] rao = remote add origin sup = submodule update --init --recursive sobmodules = submodule update --init --recursive rao: Adds a remote repository named \u0026ldquo;origin.\u0026rdquo; sup: Initializes and updates all submodules recursively. sobmodules: Initializes and updates all submodules recursively (same as sup). Rebasing and Merging Streamline your rebasing and merging tasks:\n[alias] rb = \u0026#34;!f() { \\ echo fetching...; \\ git fetch; \\ if [ $? -eq 0 ]; then \\ last_status=$(git status --untracked-files=no --porcelain); \\ if [ \\\u0026#34;$last_status\\\u0026#34; != \\\u0026#34;\\\u0026#34; ]; then \\ echo stashing local changes...; \\ git stash; \\ else \\ echo nothing to stash...; \\ fi;\\ if [ $? -eq 0 ]; then \\ echo rebasing...;\\ git rebase;\\ if [ $? -eq 0 ]; then \\ if [ \\\u0026#34;$last_status\\\u0026#34; != \\\u0026#34;\\\u0026#34; ]; then\\ echo applying stashed changes...;\\ git stash pop;\\ if [ $? -ne 0 ]; then \\ echo STASH POP FAIL - you will need to resolve merge conflicts with git mergetool; \\ fi; \\ fi; \\ else \\ echo REBASE FAILED - you will need to manually run stash pop; \\ fi;\\ fi;\\ fi; \\ if [ $? -ne 0 ]; then \\ echo ERROR: Operation failed; \\ fi; \\ }; f\u0026#34; ria = \u0026#39;!git rebase -i `git merge-base HEAD master`\u0026#39; clean-merged = !git branch --merged | grep -v \\\u0026#34;\\\\*\\\u0026#34; | xargs -n 1 git branch -d rb: Fetches updates, stashes local changes if needed, rebases, then applies stashed changes. ria: Interactively rebases the current branch onto the master branch from their common ancestor. clean-merged: Deletes all branches that have been merged into the current branch. Extra Goodies A few more handy aliases for special tasks:\n[alias] git-current-branch = \u0026#34;!git rev-parse --abbrev-ref HEAD\u0026#34; ga = add -A gap = add -p gch = cherry-pick gpthis = !git push origin $(git_current_branch) gpthis! = !git push --set-upstream origin $(git_current_branch) fb = !sh -c \\\u0026#34;git branch -a | grep -v remotes | grep $1\\\u0026#34; cb = !sh -c \\\u0026#34;git branch -a | grep -v remotes | grep $1 | head -n 1 | xargs git checkout\\\u0026#34; sco = !sh -c \u0026#34;git branch -a | grep -v remotes | grep $1 | xargs git checkout\u0026#34; fc = \u0026#34;!f() { git branch -a | grep -m1 -e ${1}.*${2} | sed \\\u0026#34;s/remotes\\\\/origin\\\\///\\\u0026#34; | xargs git checkout; }; f\u0026#34; git-current-branch: Retrieves the name of the current branch. ga: Stages all changes in the working directory for commit. gap: Interactively stages changes in the working directory for commit. gch: Applies changes from a specific commit to the current branch. gpthis: Pushes the current branch to the remote repository named \u0026ldquo;origin\u0026rdquo;. gpthis!: Pushes the current branch to the remote repository named \u0026ldquo;origin\u0026rdquo; and sets it as the fb: Searches for branches matching a specific pattern. cb: Checks out the first branch matching a specific pattern. sco: Checks out branches matching a specific pattern. fc: Checks out the first branch matching two specific patterns. Wrapping Up Git aliases can save you a ton of time and keystrokes, making your workflow smoother and more enjoyable. Try adding these to your ~/.gitconfig and watch your productivity soar! 🌟\nHappy coding, and may your commits be clean and your merges conflict-free! 💻✨\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/05-useful-git-aliases-for-a-productive-workflow/","summary":"Git aliases are a powerful workflow tool that create shortcuts to frequently used Git commands.","title":"Git Aliases to Supercharge Your Workflow"},{"content":"Intro In the vast landscape of Unix commands, few wield as much power and versatility as \u0026lsquo;find\u0026rsquo;. Whether you\u0026rsquo;re a seasoned system administrator, a curious developer, or a novice Linux user, understanding how to effectively utilize \u0026lsquo;find\u0026rsquo; can greatly enhance your productivity and efficiency.\nAt its core, \u0026lsquo;find\u0026rsquo; is a command-line utility designed to search for files and directories within a specified directory hierarchy. However, its capabilities extend far beyond simple file retrieval. With the right combination of options and predicates, \u0026lsquo;find\u0026rsquo; becomes a Swiss Army knife for navigating and manipulating file systems.\nHow to Use the Find Command In Linux Let\u0026rsquo;s dive into some practical examples to illustrate the diverse functionality of \u0026lsquo;find\u0026rsquo;:\nBasic File Search:\nfind /path/to/search -name \u0026#34;filename\u0026#34; This command searches for a file named \u0026ldquo;filename\u0026rdquo; within the specified directory (/path/to/search) and its subdirectories.\nSearch by File Type:\nfind /path/to/search -type f Restricts the search to only files, excluding directories and other types of files like symbolic links.\nSearch by File Size:\nfind /path/to/search -size +10M Finds files larger than 10 megabytes within the specified directory.\nCombining Criteria:\nfind /path/to/search -name \u0026#34;*.txt\u0026#34; -size +1M Searches for text files larger than 1 megabyte.\nExecute Commands on Found Files:\nfind /path/to/search -type f -exec chmod 644 {} \\; Changes the permission of all files within the specified directory to read/write for the owner and read-only for others.\nSearch by Modification Time:\nfind /path/to/search -mtime -7 Locates files modified within the last 7 days.\nSearch by Ownership:\nfind /path/to/search -user username Finds files owned by a specific user.\nSearch by Permissions:\nfind /path/to/search -perm 644 Locates files with specific permissions set.\nSearch and Delete:\nfind /path/to/search -type f -name \u0026#34;*.tmp\u0026#34; -delete Deletes all files with a .tmp extension within the specified directory.\nSearch and Archive:\nfind /path/to/search -type f -name \u0026#34;*.log\u0026#34; -exec tar -czvf logs_archive.tar.gz {} + Archives all log files within the specified directory into a single compressed tarball.\nSearch and Count Files:\nfind /path/to/search -type f | wc -l Counts the number of files within the specified directory and its subdirectories.\nSearch for Empty Files or Directories:\nfind /path/to/search -empty Finds empty files or directories within the specified directory.\nSearch and Copy Files:\nfind /path/to/search -name \u0026#34;*.txt\u0026#34; -exec cp {} /destination/path \\; Copies all text files within the specified directory to another location.\nSearch for Setuid/Setgid Files:\nfind /path/to/search -type f \\( -perm -4000 -o -perm -2000 \\) Locates files with the setuid or setgid bit set, which can pose security risks.\nSearch and Execute Commands Interactively:\nfind /path/to/search -type f -execdir vi {} \\; Opens each file found by \u0026lsquo;find\u0026rsquo; in the vi text editor for interactive editing.\nTricks One neat trick with the \u0026lsquo;find\u0026rsquo; command is to use it in conjunction with the \u0026lsquo;xargs\u0026rsquo; command to perform operations on the files found. This can be particularly useful when you want to execute a command on multiple files matching certain criteria.\nFor example, let\u0026rsquo;s say you have a directory with a bunch of text files and you want to search for a specific string within all these files. You can achieve this with \u0026lsquo;find\u0026rsquo; and \u0026lsquo;xargs\u0026rsquo; like this:\nfind /path/to/search -type f -name \u0026#34;*.txt\u0026#34; -print0 | xargs -0 grep \u0026#34;search_string\u0026#34; Here\u0026rsquo;s what this command does:\nfind /path/to/search -type f -name \u0026quot;*.txt\u0026quot;: Searches for all files with a \u0026ldquo;.txt\u0026rdquo; extension within the specified directory and its subdirectories. -print0: Prints the file names with a null character at the end of each name. This ensures compatibility with filenames containing spaces or special characters. xargs -0 grep \u0026quot;search_string\u0026quot;: Takes the list of file names produced by \u0026lsquo;find\u0026rsquo; and passes them as arguments to the \u0026lsquo;grep\u0026rsquo; command, which then searches for the specified string (\u0026ldquo;search_string\u0026rdquo;) within each file. This trick allows you to efficiently search for a string across multiple files without having to manually iterate through each file. It\u0026rsquo;s a handy way to perform batch operations on files found by the \u0026lsquo;find\u0026rsquo; command.\nAnother useful trick with the \u0026lsquo;find\u0026rsquo; command is to locate and delete files that match certain criteria. This is particularly handy for cleaning up your file system by removing unwanted or obsolete files in bulk.\nFor example, let\u0026rsquo;s say you want to delete all temporary files (files with a \u0026ldquo;.tmp\u0026rdquo; extension) within a directory and its subdirectories. You can accomplish this with \u0026lsquo;find\u0026rsquo; and the \u0026lsquo;-delete\u0026rsquo; option:\nfind /path/to/search -type f -name \u0026#34;*.tmp\u0026#34; -delete Here\u0026rsquo;s what this command does:\nfind /path/to/search -type f -name \u0026quot;*.tmp\u0026quot;: Searches for all files with a \u0026ldquo;.tmp\u0026rdquo; extension within the specified directory and its subdirectories. -delete: Deletes each file found by the \u0026lsquo;find\u0026rsquo; command. This command efficiently deletes all temporary files without needing to manually locate and remove each one individually. It\u0026rsquo;s a great way to streamline file cleanup tasks and free up disk space. However, use it with caution, especially when dealing with important files, as it permanently removes them from your system. Always double-check your \u0026lsquo;find\u0026rsquo; command to ensure it targets the correct files before executing it.\nAliases Here are some useful aliases for the \u0026lsquo;find\u0026rsquo; command:\nff: Short for \u0026ldquo;find files\u0026rdquo;, this alias can be used for basic file searches.\nalias ff=\u0026#39;find . -type f\u0026#39; With this alias, you can simply type \u0026lsquo;ff\u0026rsquo; followed by any additional options or predicates you want to use.\nffr: Short for \u0026ldquo;find files recursively\u0026rdquo;, this alias can be used to search for files within the current directory and its subdirectories.\nalias ffr=\u0026#39;find . -type f -name\u0026#39; Similar to the previous alias, you can add any additional options or predicates after \u0026lsquo;ffr\u0026rsquo;.\nfd: Short for \u0026ldquo;find directories\u0026rdquo;, this alias can be used to search for directories within the current directory.\nalias fd=\u0026#39;find . -type d\u0026#39; This alias is particularly useful when you only want to locate directories.\nfext: Short for \u0026ldquo;find by extension\u0026rdquo;, this alias can be used to search for files with a specific file extension.\nalias fext=\u0026#39;find . -type f -name \u0026#34;*.$1\u0026#34;\u0026#39; You can use this alias followed by the desired file extension as an argument, e.g., \u0026lsquo;fext txt\u0026rsquo; to find all text files.\nfsize: Short for \u0026ldquo;find by size\u0026rdquo;, this alias can be used to search for files of a specific size.\nalias fsize=\u0026#39;find . -type f -size +$1M\u0026#39; You can use this alias followed by the desired file size in megabytes as an argument, e.g., \u0026lsquo;fsize 10\u0026rsquo; to find files larger than 10MB.\nThese aliases provide shortcuts for common \u0026lsquo;find\u0026rsquo; command tasks, making it easier and quicker to perform file searches and manipulations in the terminal. Adjust them to suit your specific needs and preferences.\nConclusion The \u0026lsquo;find\u0026rsquo; command offers a plethora of options and predicates, allowing users to tailor their searches with precision. By mastering \u0026lsquo;find\u0026rsquo;, you unlock a powerful tool for navigating, searching, and managing files in Unix-based systems. Experimentation and practice are key to fully harnessing its capabilities, but once you do, you\u0026rsquo;ll wonder how you ever managed without it. Happy searching!\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/04-mastering-the-find-command-unleashing-unix-file-management-power/","summary":"Discover the unparalleled capabilities of the \u0026lsquo;find\u0026rsquo; command in Unix systems. From basic file searches to advanced manipulations, mastering \u0026lsquo;find\u0026rsquo; empowers users to efficiently navigate, search, and manage files. With practical examples and powerful techniques, this blog unveils the full potential of \u0026lsquo;find\u0026rsquo;, transforming your Unix experience into one of seamless productivity and control. Unlock the power of Unix file management with the mastery of the \u0026lsquo;find\u0026rsquo; command.","title":"Mastering the Find Command: Unleashing Unix File Management Power"},{"content":"Introduction In the heart of the city lies the 100-story Fabergé Egg Museum, presenting a distinctive challenge to the world\u0026rsquo;s slickest jewel thief. The mission: procure the most valuable egg without causing a mess. Armed with only two replicas, our stealthy thief concocts a plan.\nThe Challenge The task at hand: identify the highest floor from which an egg survives a fall without breaking. Each egg\u0026rsquo;s value increases with each floor, making the decision all the more crucial.\nThe Simple Scenario Let\u0026rsquo;s start with a scenario involving just one replica egg. The thief would drop it from the first floor and work her way up incrementally. This process could take up to 100 attempts. However, the introduction of a second replica egg allows for a more strategic approach.\nStrategic Egg Drops Armed with two replicas, the thief optimizes her strategy. Rather than dropping the first egg from every floor, she strategically chooses larger intervals, efficiently narrowing down the range of possible critical floors.\nFor instance, starting with drops every 10 floors, the thief minimizes the number of attempts with the second egg. If the first egg breaks at a certain interval, the second egg helps refine the search floor by floor within that interval.\nThe Math Unleashed The equation to solve the problem involves finding the smallest value of $( n )$ such that the sum of the first $( n )$ positive integers is greater than or equal to $(100)$. The sum of the first $( n )$ positive integers is given by the formula:\n$[ \\frac{n \\cdot (n + 1)}{2} ]$\nSo, the equation to solve is:\n$[ \\frac{n \\cdot (n + 1)}{2} \\geq 100 ]$\nTo solve this inequality, you can multiply both sides by $(2)$ to get rid of the fraction:\n$[ n \\cdot (n + 1) \\geq 200 ]$\nExpand and rearrange:\n$[ n^2 + n \\geq 200 ]$\nNow, set the inequality to zero:\n$[ n^2 + n - 200 \\geq 0 ]$\nSolving this quadratic inequality reveals that the smallest positive integer solution for $( n )$ is $(14)$. Thus, the equation $( \\frac{n \\cdot (n + 1)}{2} \\geq 100 )$ is solved by $( n \\geq 14 )$. This means the jewel thief should start dropping the replica eggs from intervals of $(14)$ floors to minimize the number of attempts required to find the critical floor.\nWhy $( n(n + 1) / 2 )$? The formula $(\\frac{n \\cdot (n + 1)}{2})$ comes from the sum of the first $(n)$ positive integers, and it is derived through a process known as the arithmetic series formula.\nThe sum of the first $(n)$ positive integers, denoted by $(S_n)$, can be expressed as:\n$[S_n = 1 + 2 + 3 + \\ldots + n.]$\nThis is an arithmetic series with a common difference of 1. Now, there is a neat trick to find the sum. You can pair the first and last terms, the second and second-to-last terms, and so on:\n$[S_n = 1 + n + 2 + (n - 1) + 3 + (n - 2) + \\ldots.]$\nEach pair sums up to $(n + 1)$, and there are $(n)$ such pairs because there are $(n)$ terms in the series. Therefore, the sum is given by:\n$[S_n = n \\cdot (n + 1).]$\nHowever, this sum includes each number twice (since we\u0026rsquo;re pairing them up), so we need to divide by 2 to get the actual sum:\n$[S_n = \\frac{n \\cdot (n + 1)}{2}.]$\nThis formula is a convenient way to express the sum of the first $(n)$ positive integers and is widely used in mathematics and computer science. In the context of the two-egg problem, this formula is used to determine the minimum value of $(n)$ such that $(S_n \\geq N)$, where $(N)$ is the total number of floors in the building.\nTime Complexity The time complexity of the solution to the two-egg problem is $(O(\\sqrt{N}))$, where $(N)$ is the number of floors in the building. The process involves finding the sum of the first $(N)$ positive integers and solving a quadratic inequality. These operations have a time complexity of $(O(1))$, making the overall time complexity $(O(\\sqrt{N}))$.\nSolution for Finding the Minimum Number of Steps import math class Solution: def twoEggDrop(self, total_floors: int) -\u0026gt; int: # Calculate the initial interval using the O(sqrt(N)) solution initial_interval = self.calculate_initial_interval(total_floors) # Determine the number of drops needed drops = self.calculate_drops(initial_interval, total_floors) return drops def calculate_initial_interval(self, n: int) -\u0026gt; int: # Calculate the initial interval using the O(sqrt(N)) solution return math.ceil(math.sqrt(2 * n)) def calculate_drops(self, initial_interval: int, n: int) -\u0026gt; int: # Use the quadratic formula to calculate the number of drops return math.ceil((-1 + math.sqrt(1 + 8 * n)) / 2) # Example usage sol = Solution() result = sol.twoEggDrop(100) print(\u0026#34;The breaking point is at or below floor:\u0026#34;, result) Real-World Question Question: Given two crystal balls and a list indicating the floors where the ball breaks ([false, false, true, true, true]), how do you adapt this strategy to find the breaking point with minimal drops?\nAnswer:\nimport math def find_breaking_point(breaks): min_step = math.floor(math.sqrt(len(breaks))) i = min_step while i \u0026lt; len(breaks): if breaks[i]: # If the breaking point is found, refine the search # Ensure that the end index does not go beyond the array bounds return refine_search(breaks, i - min_step, min(i, len(breaks))) i += min_step return -1 def refine_search(breaks, start, end): for i in range(start, end): if breaks[i]: # Return the index where the breaking point is found return i return -1 # Example usage breaks = [False, False, True, True, True] result = find_breaking_point(breaks) print(\u0026#34;The breaking point is at or below floor:\u0026#34;, result) References https://leetcode.com/problems/egg-drop-with-2-eggs-and-n-floors/description/ https://www.youtube.com/watch?v=NGtt7GJ1uiM\u0026amp;t=1s\u0026amp;pp=ygUPdHdvIGVnZyBwcm9ibGVt https://frontendmasters.com/courses/algorithms/ For more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/03-solve-egg-drop-with-2-eggs-and-n-floors/","summary":"Mastering the egg drop involves a careful blend of strategy, mathematics, and optimization. From crafting a plan with two replicas to understanding the arithmetic behind the sum formula, the jewel thief\u0026rsquo;s mission highlights the fascinating intersection of math and real-world problem-solving. As we unravel the complexities, we find that breaking a few eggs may require a dash of strategic brilliance and a pinch of mathematical finesse.","title":"Mastering the Egg Drop: A Math Heist Unveiled"},{"content":"Introduction Ever wondered how to handle command-line arguments like a pro in your Bash scripts? Here are some nifty techniques to make your scripts more versatile:\n💡 Positional Parameters: Access arguments directly using $1, $2, \u0026hellip;, $n. Simple and straightforward!\n#!/usr/bin/env bash echo \u0026#34;First argument: $1\u0026#34; echo \u0026#34;Second argument: $2\u0026#34; 🌀 Special Variables ($@, $*): $@ represents all parameters as separate words, while $* combines them into one. Flexibility at its best!\n#!/usr/bin/env bash for arg in \u0026#34;$@\u0026#34;; do echo \u0026#34;$arg\u0026#34; done 🔀 Using shift to Shift Arguments: Shift positional parameters to the left, discarding the first one. Great for iterative processing!\n#!/usr/bin/env bash while [ \u0026#34;$#\u0026#34; -gt 0 ]; do echo \u0026#34;Argument: $1\u0026#34; shift done 🔍 getopts for Option Parsing: Perfect for handling options and their arguments with finesse!\n#!/usr/bin/env bash while getopts \u0026#34;:a:b:\u0026#34; opt; do case $opt in a) arg_a=\u0026#34;$OPTARG\u0026#34;;; b) arg_b=\u0026#34;$OPTARG\u0026#34;;; \\?) echo \u0026#34;Invalid option: -$OPTARG\u0026#34; \u0026gt;\u0026amp;2; exit 1;; esac done echo \u0026#34;Option a: $arg_a\u0026#34; echo \u0026#34;Option b: $arg_b\u0026#34; 🔄 Using shift with getopts: Combine shift and getopts for advanced option handling and processing of remaining arguments.\n#!/usr/bin/env bash while getopts \u0026#34;:a:b:\u0026#34; opt; do case $opt in a) arg_a=\u0026#34;$OPTARG\u0026#34;;; b) arg_b=\u0026#34;$OPTARG\u0026#34;;; \\?) echo \u0026#34;Invalid option: -$OPTARG\u0026#34; \u0026gt;\u0026amp;2; exit 1;; esac done shift $((OPTIND-1)) echo \u0026#34;Option a: $arg_a\u0026#34; echo \u0026#34;Option b: $arg_b\u0026#34; echo \u0026#34;Remaining arguments: $@\u0026#34; 📝 Read from Command Line: Use the read command to read input directly from the command line. Simple user interaction!\n#!/usr/bin/env bash echo -n \u0026#34;Enter your name: \u0026#34; read name echo \u0026#34;Hello, $name!\u0026#34; 🚀 Example: Here\u0026rsquo;s a production-ready and clean script to calculate square of a number.\n📝 Script: Calculate Square\n#!/usr/bin/env bash ############################################################ # Help Function: Display information about the script ############################################################ Help() { echo \u0026#34;Usage: $0 -n \u0026lt;number\u0026gt;\u0026#34; echo \u0026#34;Calculate the square of a number.\u0026#34; echo echo \u0026#34;Options:\u0026#34; echo \u0026#34;-n Specify the number for which to calculate the square.\u0026#34; echo \u0026#34;-h Display this help message.\u0026#34; echo exit } ############################################################ # Main Program Section: Placeholder for script logic ############################################################ # Set default value Number=\u0026#34;\u0026#34; ############################################################ # Process the input options. Add options as needed. ############################################################ while getopts \u0026#34;:hn:\u0026#34; option; do case $option in h) # display Help Help;; n) # Specify a number Number=$OPTARG;; \\?) # Invalid option echo \u0026#34;Error: Invalid option\u0026#34; exit 1;; esac done ############################################################ # Validate and perform the task ############################################################ # Check if the number is provided if [ -z \u0026#34;$Number\u0026#34; ]; then echo \u0026#34;Error: Please provide a number using the -n option.\u0026#34; exit 1 fi # Check if the provided input is a valid number if ! [[ \u0026#34;$Number\u0026#34; =~ ^[0-9]+$ ]]; then echo \u0026#34;Error: \u0026#39;$Number\u0026#39; is not a valid number.\u0026#34; exit 1 fi # Calculate the square Square=$((Number * Number)) # Display the result echo \u0026#34;The square of $Number is: $Square\u0026#34; # Additional script logic goes here... 💡 How to Use:\nRun the script with -n to specify the number for calculating the square. Use -h to display the help message. ./square_calculator.sh -n 5 For more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/02-bash-cmd-line-argument/","summary":"Different ways to accept arguments in a Bash script","title":"Crafting Command-Line Magic with Ease"},{"content":"Introduction CSV (Comma-Separated Values) files are a common format for storing tabular data. In this blog post, we\u0026rsquo;ll explore the process of creating a CSV reader in C and integrating it with Python using the CPython API. Our goal is to develop a robust and efficient CSV reader that can handle various scenarios, including file errors and memory allocation issues.\nPrerequisites Before diving into the code, make sure you have the following:\nBasic knowledge of the C programming language. A C compiler installed on your system. Python installed, including the Python interpreter and header files. Code Structure Let\u0026rsquo;s begin by defining the structure of our CSV reader module. We\u0026rsquo;ll use the CPython API to create a Python extension module. Here\u0026rsquo;s an overview of the main components:\nMethod Definitions We\u0026rsquo;ll have a method named read_csv_file that takes a file path as an argument and returns a Python list of lists representing the CSV data. This method will handle errors gracefully.\nstatic PyObject *read_csv_file(PyObject *self, PyObject *args); Module Definition We define the methods and module name in the module definition structure.\nstatic PyMethodDef customcsv_methods[] = { {\u0026#34;read_csv_file\u0026#34;, read_csv_file, METH_VARARGS, \u0026#34;Read a CSV file.\u0026#34;}, {NULL, NULL, 0, NULL} // Sentinel }; static struct PyModuleDef customcsvmodule = { PyModuleDef_HEAD_INIT, \u0026#34;customcsv\u0026#34;, // Module name NULL, -1, customcsv_methods // Method definitions }; Module Initialization Finally, we create the customcsv module and return it to the Python interpreter.\nPyMODINIT_FUNC PyInit_customcsv(void) { return PyModule_Create(\u0026amp;customcsvmodule); } Reading CSV with Error Handling Now, let\u0026rsquo;s delve into the implementation of the read_csv_file method. We\u0026rsquo;ll handle various error scenarios, including file not found, memory allocation failures, and decoding errors.\n#include \u0026lt;Python.h\u0026gt; static PyObject *read_csv_file(PyObject *self, PyObject *args) { // Parse the Python arguments: a single string representing the file path const char *file_path; if (!PyArg_ParseTuple(args, \u0026#34;s\u0026#34;, \u0026amp;file_path)) { PyErr_SetString(PyExc_TypeError, \u0026#34;Invalid argument. Expected a string.\u0026#34;); return NULL; } // Open the CSV file for reading FILE *file = fopen(file_path, \u0026#34;r\u0026#34;); if (!file) { PyErr_SetFromErrnoWithFilenameObject(PyExc_FileNotFoundError, PyUnicode_DecodeFSDefault(file_path)); return NULL; } // Create a Python list to store the rows from the CSV file PyObject *result = PyList_New(0); if (!result) { PyErr_NoMemory(); fclose(file); return NULL; } // Buffer to store each line read from the CSV file char line[4096]; // Read each line from the CSV file while (fgets(line, sizeof(line), file)) { // Create a Python list to store the elements of the current row PyObject *row = PyList_New(0); if (!row) { PyErr_NoMemory(); fclose(file); Py_XDECREF(result); return NULL; } // Tokenize the line using \u0026#39;,\u0026#39; as the delimiter char *token = strtok(line, \u0026#34;,\u0026#34;); while (token != NULL) { // Calculate the length of the token size_t token_len = strlen(token); // Strip newline characters from the end of the token while (token_len \u0026gt; 0 \u0026amp;\u0026amp; (token[token_len - 1] == \u0026#39;\\n\u0026#39; || token[token_len - 1] == \u0026#39;\\r\u0026#39;)) { token[--token_len] = \u0026#39;\\0\u0026#39;; } // Decode the token into a Python Unicode string and append it to the row list PyObject *item = PyUnicode_DecodeUTF8(token, token_len, \u0026#34;strict\u0026#34;); if (!item) { PyErr_SetString(PyExc_UnicodeError, \u0026#34;Failed to decode token\u0026#34;); fclose(file); Py_XDECREF(result); Py_XDECREF(row); return NULL; } // Append the item to the row list if (PyList_Append(row, item) == -1) { PyErr_SetString(PyExc_RuntimeError, \u0026#34;Failed to append item to row\u0026#34;); fclose(file); Py_XDECREF(result); Py_XDECREF(row); return NULL; } Py_XDECREF(item); // Move to the next token token = strtok(NULL, \u0026#34;,\u0026#34;); } // Append the row list to the result list if (PyList_Append(result, row) == -1) { PyErr_SetString(PyExc_RuntimeError, \u0026#34;Failed to append row to result\u0026#34;); fclose(file); Py_XDECREF(result); Py_XDECREF(row); return NULL; } Py_XDECREF(row); } // Close the CSV file fclose(file); // Return the final result, which is a Python list of lists representing the CSV data return result; } This implementation ensures that the CSV reader is resilient to different error conditions, providing informative error messages and handling memory allocation issues gracefully.\nPython Setup and Running the Code Now, let\u0026rsquo;s go through the steps to set up your Python environment and run the CSV reader module:\nA setup.py file is typically used in Python projects to provide metadata about the project and to specify the packaging details. For a Python extension module written in C, the setuptools library can be used to simplify the packaging process.\nHere\u0026rsquo;s a basic setup.py file for the CSV reader module:\nfrom setuptools import setup, Extension # Define the extension module customcsv_module = Extension( \u0026#39;customcsv\u0026#39;, # Name of the module sources=[\u0026#39;customcsv.c\u0026#39;], # C source file ) # Setup information setup( name=\u0026#39;customcsv\u0026#39;, version=\u0026#39;1.0\u0026#39;, description=\u0026#39;Custom CSV reader in C with Python integration\u0026#39;, ext_modules=[customcsv_module], ) Save this file as setup.py in the same directory as your C code (customcsv.c). This setup.py file specifies the name of the extension module (customcsv) and the source file (customcsv.c). The setuptools.setup function is then used to provide metadata about the project.\nTo build and install the module, follow these steps:\nOpen a terminal and navigate to the directory containing setup.py and customcsv.c.\nRun the following command to build the extension module:\npython setup.py build_ext --inplace This command uses setuptools to build the extension module and places the compiled customcsv.so (or customcsv.pyd on Windows) in the current directory.\nAfter a successful build, you can import the module in Python scripts as shown in the previous example.\n# reader.py import customcsv file_path = \u0026#34;example.csv\u0026#34; # Replace with your CSV file path result = customcsv.read_csv_file(file_path) print(result) Run your Python script:\npython reader.py System-wide Installation (Optional) If you want to install the module system-wide, use the following command:\npython setup.py install After running this command, the compiled module will be placed in the site-packages directory of your Python installation, making it accessible to all Python scripts on your system.\nThis setup allows for a more standardized and manageable packaging and distribution of your C extension module with Python. Adjust the setup.py file according to your project\u0026rsquo;s specific details and requirements.\nConclusion In this blog post, we\u0026rsquo;ve walked through the process of creating a CSV reader in C, integrated with Python using the CPython API. The reader is equipped with robust error handling to handle various scenarios, making it a reliable tool for working with CSV files.\nFeel free to experiment with the code, and explore further improvements and optimizations based on your specific use cases. The complete source code is available on GitHub.\nFor more Kafka Connect tips and open-source tools, follow the blog series\n","permalink":"https://joel-hanson.github.io/posts/01-how-to-load-csv-using-cpython/","summary":"CSV (Comma-Separated Values) files are a common format for storing tabular data. In this blog post, we\u0026rsquo;ll explore the process of creating a simple CSV reader in C using CPython standard library.","title":"Building a CSV Reader in C with Python Integration"},{"content":"Introduction 👋🏼 Welcome to my blog.\nHello World ","permalink":"https://joel-hanson.github.io/posts/my-first-post/","summary":"This my first blog post on my personal website.","title":"My First Post"},{"content":"About This Blog Welcome to my personal blog! Here, I share my experiences, insights, and knowledge in the field of technology, software development, and machine learning. Dive into my articles to explore the exciting world of coding, innovation, and continuous learning.\nAbout Me I\u0026rsquo;m Joel Hanson, a seasoned software engineer with a passion for creating cutting-edge solutions. My journey in the tech industry has been marked by diverse experiences and a relentless pursuit of excellence. Let me give you a glimpse into my professional journey: Resume\nContact Feel free to reach out to me through the following channels:\nEmail: \u0026lt;joelhanson025[at]gmail.com\u0026gt; LinkedIn: Joel Hanson Portfolio: GitHub Portfolio ","permalink":"https://joel-hanson.github.io/about/","summary":"\u003ch2 id=\"about-this-blog\"\u003eAbout This Blog\u003c/h2\u003e\n\u003cp\u003eWelcome to my personal blog! Here, I share my experiences, insights, and knowledge in the field of technology, software development, and machine learning. Dive into my articles to explore the exciting world of coding, innovation, and continuous learning.\u003c/p\u003e\n\u003ch2 id=\"about-me\"\u003eAbout Me\u003c/h2\u003e\n\u003cp\u003eI\u0026rsquo;m Joel Hanson, a seasoned software engineer with a passion for creating cutting-edge solutions. My journey in the tech industry has been marked by diverse experiences and a relentless pursuit of excellence. Let me give you a glimpse into my professional journey: \u003ca href=\"/files/Resume.pdf\"\u003eResume\u003c/a\u003e\u003c/p\u003e","title":"About"}]