DZone Spotlight

Over the last two years, the industry has moved from simple chatbots and retrieval-augmented generation pipelines to something fundamentally more powerful: agentic systems. These systems do not just answer questions; they plan, act, observe the consequences of their actions, and keep iterating until a goal is achieved. Whether it is Cursor writing and debugging code, Perplexity performing multi-step research, Manus executing complex tasks through code, or Gemini Deep Research producing long-form investigative reports, the underlying architecture is agentic. At the heart of every agentic system lie 2 critical design decisions. The first is the overall topology: should the system be a single agent that owns the entire problem, or a multi-agent system where specialized agents collaborate under an orchestrator? The second is the choice of internal design patterns that govern how the agent reasons, selects tools, handles errors, and improves its own output. Get these decisions right, and the system becomes reliable, scalable, and genuinely useful. Get them wrong, and you end up with brittle loops, runaway costs, or agents that hallucinate tool calls. This article provides a clear, production-oriented map of both layers. Let us examine single-agent versus multi-agent architectures, then walk through the six design patterns that dominate real-world systems today. Each pattern is illustrated with a diagram and explained with concrete examples drawn from products already in production. 1. Agentic Systems Topology Every agentic system falls into one of two broad categories. Single Agent System In a single-agent architecture, one agent owns the complete loop. It receives the user query, maintains both short-term context and long-term memory, decides which tools or MCP servers to call, observes the results, and eventually produces the final output. This design is simple to implement, easy to debug, and has lower latency for focused tasks. It is the natural starting point for most teams. The main limitations appear when the problem becomes long-horizon or requires genuinely different skills. Context windows fill up, specialized knowledge is hard to isolate, and a single failure mode can bring the entire system down. Multi-Agent System A multi-agent system introduces a meta-agent (or orchestrator) that decomposes the high-level goal and delegates work to specialized agents, for example, a data-retrieval agent, a search agent, a coding agent, or a critic. Each specialist may have its own tools and memory. Results flow back to an aggregator LLM that synthesizes the final answer. Shared or private memory and MCP servers support the collaboration. The advantages are specialization, parallelism, and higher reliability through division of labor. The costs are coordination overhead, higher token consumption, and more complex failure modes (handoff errors, inconsistent state, cascading retries). Most production systems begin as single-agent and only move to multi-agent when the benefits clearly outweigh the complexity. 2. Core Design Patterns Topology decides how agents are organized. Design patterns decide how each agent thinks and acts. The six patterns below appear, often in combination, in virtually every serious agentic product shipping today. ReACT Agent (Reason + Act) The foundational pattern used by the majority of tool-using agents. The agent interleaves three explicit steps in a tight loop: Thought: verbalized reasoning about the current state, what is known, what is still missing, and which single action would close the biggest gap.Action: a concrete tool call with specific arguments.Observation: the tool result is injected back into the context so the next thought is grounded in reality. This loop continues until the agent decides the task is complete. Pure chain-of-thought reasoning is blind to the external world; pure tool calling is planless and reactive. ReAct fuses the two and remains the default architecture in LangGraph, LlamaIndex, the OpenAI Agents SDK, Claude’s tool-use agents, and most coding assistants. The main engineering concerns are infinite loops (always set a hard iteration limit) and context growth (every thought and observation consumes tokens). CodeAct Agent Used by Manus, OpenHands, and an increasing number of advanced coding and automation systems. Instead of emitting one discrete tool call per turn, the agent writes executable code (usually Python) as its primary action space. That code can contain control flow, multiple tool invocations, data transformations, filtering, and error handling — all in a single execution step. The sandbox runs the code and returns stdout, stderr, or artifacts. If something fails, the agent can observe the error and revise the code, achieving a form of self-debugging. CodeAct collapses what would have been many ReAct steps into one more expressive action. It also leverages the full power of existing software libraries. The trade-off is the need for a secure, well-instrumented execution environment. Self-Reflection Critical for reliability and high-stakes outputs. After generating a first draft (a plan, a piece of code, or an answer), the agent or a dedicated critic LLM evaluates the output against explicit criteria: correctness, completeness, safety, style, and grounding. If the critique fails, the agent revises using the feedback. The loop continues until the critique passes. Lessons can optionally be written into long-term memory, so the same mistake is less likely in the future. Self-reflection is the agentic analog of System-2 deliberative thinking. Andrew Ng has repeatedly listed it as one of the four fundamental building blocks of agentic systems (alongside planning, tool use, and multi-agent collaboration). It is especially valuable when the cost of a wrong answer is high. Tool Use and Agentic RAG Tool use is the substrate of nearly every modern agent. Cursor is a canonical example: the agent is given a rich set of tools (file system, terminal, search, browser, cloud APIs, MCP servers) and decides at each step which tools to call and with what arguments. The quality of the tool schemas and the system prompt that teaches the model when and how to use them is often as important as the underlying model itself. Agentic RAG elevates classic retrieval-augmented generation. In a traditional RAG pipeline, the retrieval step is fixed: embed the query, fetch top-k chunks, generate. In agentic RAG, retrieval itself becomes a tool the agent can invoke repeatedly. The agent can rewrite queries, decide which sources to consult (vector database, web search, structured APIs), evaluate intermediate results, and continue retrieving until it judges the context sufficient. Only then does it generate the final grounded answer, usually with citations. Perplexity’s more advanced research modes and modern enterprise agentic RAG frameworks follow this pattern. Multi-Agent Workflow A planner or meta-agent receives the high-level goal and decomposes it into sub-tasks. Specialized agents execute those sub-tasks, often in parallel, reading from and writing to shared memory and using tools as needed. An aggregator or synthesizer LLM collects the partial results and produces the final coherent output. Optional critique or reflection loops can be added on top. This pattern shines on long-horizon, multi-skill problems such as deep research, complex software engineering workflows, or multi-document analysis. Gemini Deep Research is a prominent production example. 3. Choosing and Combining Patterns PatternBest suited forLatency / CostReliabilityComplexityReActAdaptive multi-step tool useMediumHighLow–MedCodeActComplex logic & data pipelinesLower (fewer turns)HighMediumSelf-ReflectionHigh-stakes accuracyHigherVery HighMediumTool UseAny grounded actionLowBaselineLowAgentic RAGKnowledge-intensive questionsMedium–HighHighMediumMulti-AgentSpecialized + parallel long-horizon workHigherHighHigh In practice, these patterns are almost never used in isolation. Cursor combines heavy tool use with a ReAct-style loop and test-driven reflection. Manus leans on CodeAct. Deep research systems blend multi-agent orchestration, agentic RAG, and reflection. The patterns are composable building blocks rather than mutually exclusive choices. 4. Engineering Principles That Matter Most Regardless of which patterns you choose, several engineering practices determine whether the system is production-ready: Explicit termination conditions and hard iteration limits prevent runaway loops and cost explosions.High-quality tool schemas and descriptions are often more important than the choice of model. Clear parameter types, realistic examples, and guidance on when to use each tool dramatically improve reliability.Careful memory management decides what stays in the context window versus what is externalized to a vector store or key-value memory.Full observability: Every thought, action, and observation must be logged and inspectable. Without traces, debugging agentic systems is nearly impossible.Cost, safety, and permission guardrails must be enforced at the harness level, not left to the model.End-to-end evaluation that measures real task success (not just next-token accuracy) is essential for continuous improvement. Closing Thoughts Agentic systems represent a genuine architectural shift. We are moving from models that merely generate text to systems that can plan, act in the world, observe the consequences, and improve their own behavior. The difference between a fragile demo and a reliable product usually comes down to the quality of the topology and the design patterns chosen. We have discussed in this article, the following: Single-agent versus multi-agent structure, ReAct as the workhorse reasoning-and-acting loop, CodeAct for expressive actions, self-reflection for reliability, tool use as the universal substrate, agentic RAG for dynamic knowledge retrieval, and multi-agent workflows for complex collaboration give practitioners a practical decision framework. Start simple. Begin with a single agent using the ReAct pattern and solid tools. Add self-reflection when accuracy matters. Introduce CodeAct when the logic becomes complex. Move to multi-agent architectures only when specialization and parallelism clearly pay off. Instrument everything. Measure real task success. Iterate. The systems that follow these principles are already delivering meaningful value in coding assistants, research tools, enterprise automation, and personal AI agents. The patterns themselves will continue to evolve, but the underlying ideas — interleaving reasoning with action, grounding decisions in observation, and composing specialized capabilities — are likely to remain foundational for years to come.

By Ram Ghadiyaram CORE

The Index Can Be Fast and Still Be Wrong Vector search teams usually define performance with query latency, recall, and throughput. Those measures matter, but they can all look healthy while the system returns a stale version of a document that changed minutes ago. The index is fast. The answer is still wrong. This failure is easy to miss because a vector index is normally downstream from the source of truth. Between a database write and a searchable embedding sit event capture, transport, chunking, model inference, index mutation, and cache invalidation. Freshness is the end-to-end property produced by that entire chain. Freshness Needs a Contract Saying that updates are processed quickly is not a contract. A useful freshness SLO states which source version must be searchable, how long the pipeline may lag, and what the query path should do when that guarantee cannot be met. For example, a service might require 99 percent of committed updates to become searchable within 60 seconds, while deletes must disappear within 10 seconds. The distinction matters because showing old text is inconvenient, but returning deleted or access-revoked content can become a security incident. Capture the Write Without a Dual-Write Gap The first failure appears when application code writes the business row and publishes an indexing event as two separate operations. If the database commit succeeds and the publish fails, the source changes without any durable instruction to update the index. Retrying the request does not reliably repair that gap. A transactional outbox avoids the split. The application updates the entity and inserts an outbox record in the same database transaction. A change-data-capture process then publishes the outbox record asynchronously. SQL BEGIN; UPDATE documents SET body = :body, version = version + 1 WHERE id = :id; INSERT INTO embedding_outbox (event_id, entity_id, source_version, operation) SELECT :event_id, id, version, 'UPSERT' FROM documents WHERE id = :id; COMMIT; Make Every Event Versioned and Idempotent Delivery systems retry. Partitions rebalance, workers crash after writing but before acknowledging, and older events can arrive after newer ones. The index consumer must therefore treat duplicate and out-of-order delivery as normal behavior, not an edge case. Each event should carry an immutable event identifier, entity identifier, monotonic source version, operation, and payload reference or hash. The consumer applies a mutation only when the incoming version is newer than the indexed version. That compare-and-set must be atomic in the index or in a strongly consistent metadata store beside it. Python def apply(event, index, embed): current = index.metadata(event.entity_id) if current and current.source_version >= event.source_version: return "already_applied" if event.operation == "DELETE": index.delete_if_newer(event.entity_id, event.source_version) return "deleted" vector = embed(event.content) index.upsert_if_newer( id=event.entity_id, vector=vector, metadata={"source_version": event.source_version} ) return "updated" Deletes Are First-Class Data Upserts get most of the design attention because they create embeddings. Deletes are more dangerous because there is no new content to process. A delete event must survive the same durable path and carry a version that prevents an older upsert from resurrecting the record later. Keep tombstones long enough to cover the maximum replay and recovery window. If a full rebuild reads a snapshot taken before a delete, the rebuild process must also consume the change stream from the snapshot position forward. Otherwise, the old record can quietly return when the new index is promoted. Model Versions Belong in the Index Schema Fresh source data can still be semantically stale when query and document vectors were created by different embedding models. Store the embedding model identifier, chunking configuration version, and normalization settings with every indexed item. Treat those fields as part of the index schema. A model upgrade should normally create a new physical or logical index generation. Dual-write new updates, backfill historical content, validate retrieval quality, then switch query traffic. Mixing vectors from incompatible spaces in one collection creates a failure that looks like weak relevance but cannot be tuned away. Use Watermarks to Measure What the Pipeline Has Proven A queue-depth metric shows workload, not freshness. The more useful signal is a source-position watermark: the highest committed database position or entity version that the searchable index has fully applied. Compare that watermark with the source head to measure version lag and event-time lag. Parallel consumers complicate this because one partition can race ahead while another is stuck. The global searchable watermark is bounded by the slowest required partition. Reporting the fastest worker hides exactly the stale slice users are likely to hit. Guard Queries When Freshness Matters Some requests know the minimum version they require. A write API can return the committed source version, and a later search request can send that version as a read-your-writes token. The query layer then checks whether the relevant index watermark has caught up. The fallback depends on the product. The service can wait briefly, route to a fresher generation, perform a source-of-truth lookup, or return a clear retryable status. Serving an older result without saying so should not be the default. Python def search(query, minimum_version=None): if minimum_version is not None: if index_watermark() < minimum_version: raise RetryableFreshnessError( "search index has not reached the required version" ) return vector_index.search(query) Rebuild Without Creating a Freshness Blackout Large indexes eventually need rebuilding because schemas, models, or partition layouts change. A safe rebuild uses a snapshot plus a change-stream handoff. Record the snapshot position, bulk-load the snapshot into a new generation, replay every later event, and promote only after its watermark reaches the live index. The promotion itself should be an atomic alias or routing change. Keep the previous generation available for rollback until both correctness and latency checks pass. A rebuild is not complete when bulk loading ends. It is complete when the new generation proves that no committed change was skipped. Operate Freshness Like Availability The dashboard should track source-to-index lag percentiles, oldest unapplied event age, consumer retry rate, dead-letter volume, version conflicts, delete lag, model-version distribution, and watermark gaps by partition. Alerts should be tied to the freshness SLO rather than to queue depth alone. Periodic reconciliation closes the final gap. Sample source entities, compare their versions and hashes with indexed metadata, and repair mismatches through the normal event path. The goal is not to pretend delivery is perfect. The goal is to make drift observable, bounded, and repairable. Test Failure Modes Before Launch Also test partial degradation. If one embedding worker pool is unavailable, confirm that lag remains visible and query guards behave as designed. If the dead-letter path fills, verify that alerts fire before the SLO is exhausted. These exercises turn recovery assumptions into executable evidence and reveal whether the pipeline can repair itself without manual database edits. Freshness behavior deserves fault-injection tests, not only happy-path integration tests. Pause one consumer partition, duplicate a batch, deliver versions out of order, fail an embedding call after the index write, and replay a snapshot across a recent delete. Each test should assert the final indexed version, not merely that the worker returned success. The Missing SLO Production vector search is a replicated data system with expensive transformation in the middle. Once that is clear, familiar distributed-systems rules apply: capture changes durably, version every mutation, make consumers idempotent, preserve deletes, expose watermarks, and rebuild from a known log position. Latency tells you how quickly the index answered. Freshness tells you whether it answered from the world that exists now. A production search system needs both guarantees, because a fast answer from yesterday is still a failure. References Debezium Outbox Event RouterPostgreSQL Logical Decoding ConceptsApache Flink Timely Stream Processing and Watermarks

This article was originally published on my blog. For the latest version and future updates, please visit the original post: https://jaketao.com/language/en/kv-cache-vs-prompt-cache/. Every time a large language model generates a token, it draws on the content that came before it. If it had to compute everything from scratch at every step, responses would be much slower. When building an agent, the same set of system prompts, tool definitions, and conversation history is used over and over again. If these were reprocessed each time, latency and computational costs would continually increase. These two types of redundant computation correspond to two concepts that are often confused: the KV cache and the prompt cache. A model’s processing of a single request is usually split into two stages: prefill and decode. The KV cache stops the system from redoing the work on historical token K/V pairs during decoding, and the prompt cache lets later requests reuse the same prefix. In short, the KV cache is the underlying state and inference mechanism. The prompt cache is a strategy or product capability that reuses preprocessing results across requests. Many prompt cache implementations rely on reusing precomputed K/V states. Tip for reading: This text is going to talk about Q, K, V, prefill, decode, prefix matching, and cache breakpoints (also called cache boundaries). You don’t need to know anything about math or APIs to understand this article. When you’re reading, first think of Q, K, and V as “intermediate vectors” in attention calculations. Then, follow along with the two examples: “Beijing weather” and “product manual.” KV Cache: “Intermediate Results” During Model Generation Large models generate content token by token. Whenever a new token is generated, the model has to consider the tokens that have already appeared. For example, in a standard Transformer, each token makes three sets of vectors  —  Q, K, and V  —  at every layer. You can think of Q as “what I’m looking for,” K as “what I have here,” and V as “what information I should extract if I’m selected.” In autoregressive decoding, the Q values of historical tokens aren’t reused in subsequent steps. However, their K and V values are repeatedly queried by tokens generated later. So, the model stores these K and V values  —  this is the KV cache. For example, if you were to ask, “What’s the weather like in Beijing?” During the prefill phase, the model processes the whole question and stores the K and V values for each token at every layer. Once the decoding phase starts, new Q, K, and V values are calculated only for the token just added to the sequence at each step. The model combines the current K and V with the cached history, then performs attention calculations using the current Q on both the historical and current K and V to predict the next token. This way, you won’t have to keep recalculating the K and V of historical tokens. But that doesn’t mean long context is free. For standard full-attention models, the longer the context, the more video memory the KV cache uses, and the more historical K/V pairs usually need to be read at each step. So, long conversations might still feel slow. Attention structures like sliding windows limit the history that can be seen. The KV cache is usually managed by the inference engine, and application developers rarely interact with it directly. It’s mostly used for incremental decoding within a single generation, but the cached K/V state can also be used by the inference framework for cross-request prefix reuse. The latter is often called a prompt cache or prefix cache. Prompt Cache: Eliminating Redundant Processing of Identical Prefixes When people hear the term “cache,” many immediately think of an “output cache,” where a previously answered question is simply returned. But the prompt cache isn’t the same kind of output cache. Even if there’s a cache hit, the model will still regenerate the response. The prompt cache reuses intermediate results from the prefill phase for prompt prefixes, such as K/V states or other similar preprocessing results. A cache hit reduces redundant prefill computations and shortens the delay for the first token. If the API provider charges for cached inputs, it can also lower the cost of repeated inputs. For example, let’s say you give the model a 50-page product manual and ask: Plain Text Product manual → What's the warranty period? A bit later, you ask another question based on the same manual: Plain Text Product manual → What are the requirements for returning an item? The product manual used in both requests is identical, except for the last question. If this common prefix is cached, the second request can reuse the preprocessed results associated with the manual and process only the new question that follows. On the other hand, if you only use this manual once, prompt cache might not be that helpful. For prompt caches that use automatic matching or caching based on breakpoints, the reusable portion should typically consist of a continuous, identical prefix starting from the beginning of the prompt. So, content that changes slowly and can be reused in many ways should go at the beginning, while content that changes frequently should go at the end. For example: Plain Text Long-term stable content: system prompt, tool definitions → Periodically stable content: user configuration, reference documentation, task background → Session content: conversation history, task status → Current request: current time, temporary information, user question This isn’t a fixed classification. The key is to arrange content by stability, but this shouldn’t alter message roles, command priorities, or business semantics. Different service providers may use automatic matching, explicit cache breakpoints, or independent cache objects. Also, keep in mind that minimum length, expiration periods, and billing rules can differ depending on the model. When integrating, it’s a good idea to check the latest model documentation and cache statistics in the response. Two Common Bad Cases: These Approaches Can Quietly Break Cache Reuse Here are two common examples. The code uses Anthropic’s cache_control as an example, but other service providers may use automatic matching, different cache markers, or independent cache objects. So, you can’t simply copy these fields across providers. 1. Dynamic Content Can Mess With Prefix Stability When you’re counting on prefix matching, if the content changes at a certain point, the old prefix following that point usually can’t be reused. So, including a timestamp  —  which changes with every request — in the cache prefix will affect the fixed rules that follow it. JavaScript // ❌ Timestamp is included in the cached prefix and changes every request const system = [{ type: "text", text: `Current time: ${new Date().toISOString()} You are a code assistant. Here are the fixed behavior rules...`, cache_control: { type: "ephemeral" } }] A better approach is to put long-term, stable content at the beginning and set a cache breakpoint at the end of the stable prefix. Dynamic information, like timestamps, should be placed after the cache breakpoint. JavaScript // ✅ Stable content first; dynamic content after the cache breakpoint const system = [ { type: "text", text: "You are a code assistant. Here are the fixed behavior rules..." }, { type: "text", text: "Here are the fixed tool usage instructions...", cache_control: { type: "ephemeral" } }, { type: "text", text: `Current time: ${new Date().toISOString()}` } ] Content like project configurations and reference materials might be somewhere between “long-term stable” and “subject to frequent changes.” You can sort them by stability. If there are multiple cache breakpoints, set breakpoints for stable prefixes of different lengths. You also need consistency beyond just the text: the order of tool definitions, image parameters, and other elements may also participate in prefix matching. 2. Only Caching the System Prompt in Multi-Round Conversations Multi-round conversations usually include the conversation history in every request. If you set the cache breakpoint only at the end of the system prompt and don’t enable automatic caching, the growing conversation history will still need to be processed repeatedly. JavaScript // ❌ Only caches the system prompt; conversation history is outside the cache boundary const request = { system: [{ type: "text", text: "You are a code assistant...", cache_control: { type: "ephemeral" } }], messages: history } You can use Anthropic’s current auto-caching as an example. Enable cache_control at the top level of the request to automatically move the cache boundary forward as the conversation grows: JavaScript // ✅ Automatically cache the prefix of an ever-growing conversation const request = { cache_control: { type: "ephemeral" }, system: [ { type: "text", text: "You are a code assistant..." } ], messages: history } Once enabled, the next round of requests can use the prefix that was cached from the previous round. It processes only the responses, tool calls, tool results, and current question that were added, and writes a new cache prefix for later requests. So, this reduces unnecessary processing of the conversation history. It does not mean that only the last message results in a cache miss. If the service provider doesn’t support automatic caching, you need to follow its rules and place an explicit cache breakpoint at a stable position near the end of the conversation. Setting a cache doesn’t guarantee a hit. When a prefix is encountered for the first time, the system typically has to finish the computation and write it to the cache first. A cache miss may occur if the cache has expired, doesn’t meet the minimum length, the historical prefix has changed, or the cache entry isn’t yet available. As of August 2026, each cache breakpoint in Anthropic will only search for previously written cache entries within the most recent 20 content blocks. A miss may also occur if too many blocks are added during a single Agent cycle. You can’t just look at whether cache configuration is present in the request to determine whether caching really provides benefits. You should check the cache read, write, and hit metrics that the API returns. If latency is a concern, you should also log first-token latency on the application side and evaluate cache effectiveness together with actual costs. Finally, How to Tell the Two Apart ConceptCore FunctionKV cache (Inference Mechanism)Reuses the K/V pairs of historical tokens during generation to avoid redundant calculations at each step.Prompt cache/prefix cache (Cross-Request Reuse)Reuses prefill results with the same prompt prefix across different requests. So, the two are not equivalent, nor are they entirely unrelated. The KV cache is the underlying state and inference mechanism. The prompt cache reuses the pre-computed prefix state for other requests. For application developers, the best approach is to keep the common prefix stable and put timestamps, temporary information, and the current question as far toward the end as possible. References OpenAI: Prompt cachingAnthropic: Prompt cachingHugging Face: CachingvLLM: Automatic Prefix CachingDeepSeek: Context Caching

Everybody wants to build agents these days  —  the internet is bombarded with stories of developers who built their own productivity agents in a single day. Can enterprises do the same? Maybe, but not at scale. Yes, agents bring unprecedented speed of operation, but they also bring the potential for sometimes hidden technical debt. Personalized using AI with an Adobe Stock image reference as baseline Everybody wants to build agents these days  —  the internet is bombarded with stories of developers who built their own productivity agents in a single day. Can enterprises do the same? Maybe, but not at scale. Yes, agents bring unprecedented speed of operation, but they also bring the potential for sometimes hidden technical debt. This reminds me of a chart my professors showed us in our machine learning class almost a decade ago. Source: Hidden Technical Debt in Machine Learning Systems, D. Sculley et. al, 2015 Only a small fraction of real-world ML systems is composed of the ML code, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex. — Hidden Technical Debt in Machine Learning Systems, D. Sculley et. al, 2015 Replace ML with agents, and a lot of the truth in that statement remains the same 10 years later. Let’s dissect these blocks and delve deeper into how each of these concepts could apply to agentic systems and introduce technical debt. Configuration → Identity and Agent Configuration Identity An agent, at its heart, is a large language model capable of reasoning to call the right tools for a user task. Marketing spiels promise quick return on investment across business functions like HR, Procurement, and IT, among others. The promise of an agent autonomously performing tasks like applying for leave or applying for an employment letter for an end user. But it’s not as simple as adding a few Workday APIs behind an LLM, is it? The configuration — especially setting up proper authorization and role-based access  —  becomes critical. Authentication (AuthN) How does the tool in the backend know that an agent has authorization from an end user to invoke that action? Through concepts such as on-behalf-of, or OBO, flows. The chart below shows what that flow looks like. Ensuring that this is configured correctly is critical. Although most identity providers rely on standard protocols like OAuth 2.0, each has its own implementation details and configuration requirements. Likewise, enterprise APIs and backend tools may expect different token types, audiences, or authentication mechanisms. A mismatch at either end can prevent the agent from successfully acting on the user’s behalf. Authorization (AuthZ): Another consideration at this stage is the authorization and access that go beyond the establishment of identity. Does every user get access to every tool, or are the tools available to an agent determined by the user’s access rights? Some general best practices in this area are enforcing least-privileged access and limiting scope based on user session and/or role. AuthN and AuthZ work together to enforce end-to-end context: can this agent act on behalf of this user, and can this user perform this task in this application? [2][3] Agent Configuration This is where we think about the agent itself. The first decision point is the model to be used. Are the tasks simple enough to be handled by a smaller model, or are there complex decisions that need a larger model? Do we need multiple models in a multi-agent architecture? Large language models also come with their own set of parameters. Does model reasoning need to be set to high, leading to higher latency, or can it be set to low? How much flexibility in responses are we willing to handle with the knob set to temperature? How long or short do we want our responses to be? What’s the prompt going into the system? There is so much configuration that sits on top of the base LLM that finding the right balance of parameters can take some trial and error. However, out of all the technical debts we’ll discuss in this article, this one is probably the most apparent to all developers. Data Collection → Context Curation Just because you don’t need data to train an LLM does not mean data is not needed to leverage an LLM. An agent is only as good as the context it receives to support its internal knowledge, especially for enterprise applications where you cannot rely on its internal knowledge to respond to user queries. You need it to respond to user queries using your in-house ground truth. Between LLMs and agents, there was a time when everyone was promoting retrieval-augmented generation (RAG) as the solution to that challenge. And it is still a prevalent part of almost every agentic system being built by large enterprises. In fact, RAG has become so pervasive that it is now being applied across a wide range of knowledge sources  —  from traditional relational databases and document stores to graph databases and other structured knowledge representations. But context curation doesn’t stop at the context fetched through RAG. The design of the agent memory plays a big role in how the user and conversation context are carried from one query to another. LLMs have limited context memory, so additional methods can be applied to the conversation history to efficiently compress and summarize it, and to extract and store valuable user and conversation facts into context variables. There’s also a fine balance that must be maintained between too much and too little context. Too much context can confuse the model, leading to issues such as hallucinations, latency, and increased costs. Too little context can also lead to hallucinations and poor response quality and decision-making for the agent. Imagine a scenario where you have 100s of tools in your backend systems. Would you load all those tools into your model context? That’s what many naive implementations can do if the number of tools is small, but it becomes risky in an enterprise setting. These are also the kinds of issues that good context curation can help address through features like progressive discovery [1]. Feature discovery provides your agent with tool information on demand, instead of overloading the context window upfront. Feature Extraction → Tool Cataloging and Discovery Similar to the data exploration phase of a machine learning modeling process, there should be a tool cataloging phase where the enterprise lists all the tools needed in their agentic system. This exercise influences the agentic system design more than meets the eye. In an ideal agentic system architecture process, the architect and/or lead developer already know which tools they need in their system, and that knowledge influences the number of agents in the system, the overall architecture design of the system (sequential, parallel, human-in-the-loop, among others), and the design of the agent payload. If these things are not outlined before the development process begins, the risk of rearchitecting the agentic system as new tools are onboarded increases. Setting up the Model Context Protocol (MCP) layer would also fall under this bucket. MCP is an open-source standard for connecting agentic (or AI) applications to external systems. Using MCP, these applications can connect to data sources, tools, and workflows in a standardized way, enabling them to access key information and perform tasks. Rather than building custom integrations for every backend system, developers can expose capabilities through MCP servers, allowing agents to discover available tools and their schemas dynamically. This reduces the implementation complexity and the development cycles required to connect agents with backend systems. Other components that fall into this area would be general best practices for tool design and exposure. Things like using a consistent naming convention, applying the right versioning policies, converting an existing OpenAPI spec into a tool only after an audit and cleanup, and ensuring schemas are defined and documented. Inconsistent tool descriptions, poor schema design, versioning changes, and the proliferation of overlapping or redundant tools can still lead to technical debt. Data Verification → Guardrails and Guidelines There are many stories online about people misusing online AI-based chatbots for non-malicious purposes. [4] However, there are bigger concerns surrounding the idea that nefarious user prompts can override a system prompt in an LLM-based application. Data verification is still a critical building block of an agentic system, even if the data validation looks different from a traditional ML model. Guardrails need to be put in place to detect and counter any input containing hate, abuse, or profanity, flag and escalate input containing signs of jailbreak or prompt injection, and block inappropriate questions. PII redaction may need to be implemented before the agent responds to a user, considering deploying agents in a healthcare organization operating under HIPAA regulations, for instance. There may also be guidelines built around the agent to ensure it adheres to business policies. In a customer refund scenario, for instance, you do not want the agent to issue a refund without checking the rules defined in its guidelines. The ideal workflow here would be for the agent to take the request, first verify the customer’s eligibility, then process the information according to its guidelines, request additional input from the customer as needed, and finally either approve the refund for simple cases or pass the request to a human for complex cases. Guardrails also need to be established for the output of the agent. The agent needs to be protected against responding with any hateful language. Guidelines may be defined to validate the arguments returned from a tool, and conversion to structured outputs might need to happen before a response is generated for the end user. Just as data verification ensures an ML model operates on trusted and well-formed inputs, guardrails and guidelines ensure an agent acts after verifying the user request (guardrails), gathering sufficient information, and following the organization’s prescribed decision-making process (guidelines). Machine Resource Management → Model & Resource Optimization Some of the concepts that fall under this overlap with the previous sections, but now we examine them through a slightly different lens. These are only a handful of the top highlights; there may be other concepts that also fall under this umbrella. Model Selection We briefly mentioned choosing between a large and a small model according to the needs of the use case. However, developers often gravitate toward larger models because they are more forgiving of ambiguity in prompts and edge cases. But larger models also come with their large price tag. Then there is a choice between reasoning vs. non-reasoning models. And a choice between high and low reasoning for models. Adaptive model routing can be another way to address this challenge  —  route the query to a small model for simple tasks and to a larger model for more complex tasks. One size never fits all, and excessive computation for simple tasks introduces unnecessary cost and latency, while underpowered models can create reliability and performance issues. Running experiments is critical at this time to ensure stable, cost-effective model selection and to avoid performance- or cost-related repercussions in production. Prompt Caching Many agentic applications repeatedly send the same instructions, system prompts, tool descriptions, and context to the LLM. Prompt caching avoids reprocessing these unchanged prompt prefixes, reducing both inference cost and latency. Prompt caching is not available by default; it depends on both the model architecture and the deployment platform. Before deploying your agentic application to production, verify whether prompt caching is available. It could save you both time and money. Parallel Tool Execution This could be either an architectural decision or an implementation choice. And this is one of those things that we get better at with experience. Let’s say you’re building a travel agent that helps an end user book travel. It needs to check the weather and respond to the user with flight and hotel recommendations before being prompted to make certain reservations. A naive approach could run these three in sequence  —  first check the weather, then book the flight, then book the hotel. That’s what feels most natural because that’s how we’d do it as humans. But an agent does not have to be limited to the parallel-processing capabilities of humans (or the lack thereof). The workflow can trigger the three tool calls in parallel, process the results, and respond to the user with recommendations. Inference Budgets Each interaction with an LLM consumes input and output tokens, making token usage a shared resource that must be actively managed. Without clear token budgets, costs can scale unpredictably as the solution gets looser. Use of larger models where smaller models could’ve sufficed, lengthy prompts that should be steps in a workflow, ten passages retrieved for a RAG solution that only needs three, or LLMs trying to call the right tool ten times before returning an error, can all lead to the exponential growth of the token usage for a solution that did not need to be so bloated in the first place. Token limits encourage smarter design choices by creating a constrained environment, and help balance response quality with latency and cost while preventing runaway execution in production. Analysis Tools → Observability & Tracing In ML, analysis tools include scripts and/or notebooks that can help developers troubleshoot and debug the ML pipelines. In agents, debugging happens by investigating the agent traces. OpenTelemetry is a common term that you’ll hear when discussing agentic evaluation. OpenTelemetry is a vendor-neutral standard for collecting and exporting traces, metrics, and logs from agentic workflows, using OpenTelemetry SDKs, the OpenTelemetry Protocol (OTLP), and compatible monitoring backends. LangFuse is a popular open-source framework for tracing and evaluating agentic workflows. If your agentic workflows were built using LangChain or LangGraph, you can also use LangSmith to evaluate your agents using native methods such as execution graphs, conversation replay, prompt inspection, and tool call visualization. [5] If you thought you were flying blind without the ability to troubleshoot your ML pipelines, you’d be flying blind in the dark in an undefined subspace without the ability to investigate the trace logs for your agentic systems. While ML pipelines were deterministic to some extent, agentic pipelines are much more autonomous, and the need to understand and debug the decision-making is even higher. Serving Infrastructure → Inference Providers Many infrastructure providers give developers access to the same models, and many enterprises have access to multiple providers. The question then becomes  —  how do the developers choose? This can be broken down into a few areas from an inference standpoint. Latency Some providers offer specialized inference hardware designed to minimize latency for supported open-weight models, while others optimize for broader model availability or advanced reasoning capabilities. Providers may also employ techniques such as model quantization to accelerate inference with minimal impact on quality. Finally, deployment topology matters: network distance, regional availability, and additional network hops can all contribute to end-to-end response latency. Cost Providers have different pricing models and per-token costs. Regional Compliance Enterprises may need to choose providers based on data residency or regulatory requirements. Scalability & Reliability Service availability, rate limits, concurrency support, request throughput, load balancing, and autoscaling all influence how well a provider performs under production workloads. While these differences may not be apparent during development, they become increasingly important as agent adoption grows, determining whether a system can maintain consistent performance and reliability under sustained or burst traffic. Deployment Choices Enterprises also need to choose between options such as Software-as-a-Service, on-premises deployment, or a hybrid deployment architecture. There are other considerations — provider routing (similar to model routing but for providers) and failover strategies — that would also fall under this umbrella. Monitoring → Evaluation This is related to the concepts covered under Observability and Tracing, but unpacks more concepts in that area, especially as they relate to monitoring agents in production. Setting up a solid foundation for observability during the development period is critical to ensure success with evaluation during production. The first step is to determine the metrics you need to evaluate once your agent is in production. In agentic workflows, you’re not just monitoring the LLM; you’re monitoring prompts, tool calls, latency, costs, journey failures, journey successes, and some form of user feedback. If you’ve implemented a RAG tool, you’d also need to evaluate specific RAG metrics like retrieval quality, generation quality, and faithfulness. If you want to capture user feedback, you’d want to build in implicit metrics — user journey completion analytics, tool usage pattern analytics, goal achievement metrics, escalation triggers, error and confusion signals, complexity indicators, and explicit metrics — “thumbs up” and “thumbs down” feedback responses. Process Management Tools → Workflow Orchestration Last but definitely not least (probably the first in order of execution as it relates to agentic system design) — this is where we take a step back and ask ourselves what really needs to be agentic and how we best design our multi-agent orchestration system. This is probably one of the loosest mappings to the original paper’s concept, but it’s an important topic to discuss nonetheless. Unlike traditional ML systems, agentic systems don’t just make predictions; they execute multi-step workflows, introducing an entirely new class of technical debt around coordination, state management, and recovery that the original paper didn’t have to consider. The first key decision point is what needs to be an agent and what can be a deterministic flow for an agent to leverage. The second key decision is how the state is maintained between different agents and tools. The third decision is the flow of information and branching of business processes and decisions. Then there are decisions around retries, checkpoints, timeouts, error handling, and the points for human escalation that need to be made at this stage. In summary: The new agentic system technical debt paradigm (image generated using AI) Let’s continue to adopt agentic design best practices so that this quote does not become a forewarning for agentic systems as well. Acknowledgment: All the opinions in this article are my own, not those of my employer. I leveraged AI tools for my research and to generate some of the graphics in the article.

For a while now, an idea has been gaining traction: with artificial intelligence, anyone can build an app without knowing how to code. The promise is incredibly seductive: with just a few prompts, we can generate code and instantly turn an idea into a product. It’s no coincidence that this vision took hold so quickly and gave rise to services like Lovable.dev, Blot.new, v0, and others. Every new technological evolution that narrows the gap between an idea and software tends to make developers' work look like an arcane ritual waiting to be dismantled by a simpler formula. There is something deeply familiar about all of this. Something that reminds me of a line from a song many of us grew up with, with its slightly childish enthusiasm: everybody wants to be a cat! Today, it seems like everybody wants to be a dev. The real question is whether everybody can be a dev. Joking aside, the attempt to make programming accessible to everyone is an old story, one that certainly didn't start with the advent of AI. A World Without Developers The idea that technological evolution can democratize programming is a recurring theme in the history of computer science. Every time a new abstraction emerges, someone proclaims that the job of writing software is about to become obsolete. Sometimes the promise is alluring; other times, it's just a clever way to sell a new tool. Yet, the core premise remains the same: if computers get closer and closer to understanding human language, then perhaps those seemingly indispensable technical skills are no longer needed. I’ve seen this pattern repeat itself multiple times. A demo takes half an hour to build, a prototype seems to work, and suddenly, the idea of building an app feels within anyone's reach. It’s fascinating, but the problem is that what you see at the beginning is often just the surface-level work: the interface, the screens, the user flow. What remains hidden is the hardest part, the work that determines whether the application will actually hold up when it goes live in production. Promises of the Past Looking back, the history of computing is full of waves that announced the end of developers. These waves didn't eliminate the profession; they transformed it. And that transformation should serve as a lesson to help us understand exactly what is happening today with AI. COBOL and Quasi-Natural Language In the 1950s and '60s, when programming meant working directly with hardware, assembly, and mathematical logic, COBOL was born. Its goal was clear: to bring programming closer to everyday language so that business managers could express rules more naturally. The idea was that a manager could describe a process's logic in English, and the computer would handle the rest. That promise didn't pan out the way people imagined. We didn't end up in a world where everyone wrote software the way they wrote letters. Instead, a vast ecosystem of specialists emerged who knew how to use that language rigorously, efficiently, and sustainably. In other words, the barrier to computer programming didn't disappear; it shifted. SQL and Fourth-Generation Languages In the 1970s and '80s, with the rise of databases, fourth-generation languages (4GLs) and SQL arrived. The concept was simple: instead of explaining every procedural step to the computer, you just had to declare what you wanted to achieve. In theory, a non-technical user could query a database and get a result. In practice, however, writing correct queries, managing complex schemas, and understanding how data connects requires a much deeper level of reasoning than it appears at first glance. As a result, the language became more accessible, but the need for expertise didn't vanish. If anything, it became more specialized. New roles, new professionals, and new problems to manage emerged. The computer kept doing its part, but the ability to think in a structured and precise way remained essential. HyperCard and the Dream of Democratic Programming In the 1980s, HyperCard truly felt like a revolution. With a simple card-based metaphor and a highly readable language, it promised to put software creation into the hands of anyone. Teachers, artists, students, everyday people: everyone could build interactive apps, games, or educational tools without a deep background in computer science. It was a captivating dream, and it partially worked. HyperCard became a massive tool for creativity, inspiring the evolution of the Web and early forms of digital collaboration. But when it came to building something truly robust, scalable, or professional, the system hit technical and organizational walls. The democratization of programming remained a promise that looked much better on paper than in industrial reality. CASE Tools and the Dream of Guided Software Between the 1980s and '90s, another promise attempted to make development more accessible: CASE (Computer-Aided Software Engineering) tools. The idea was simple: if a system could help map out an application's flow, generate pieces of code, and guide the design process, then even non-experts could build software in a more structured way. In practice, however, CASE tools didn't eliminate the need for expertise. They simplified certain steps, especially during the analysis and design phases, but they didn't replace the work of someone who could see the bigger picture. Visual Basic and the Drag-and-Drop Era In the 1990s, Visual Basic turned creating desktop applications into a near drag-and-drop affair. It was the modern version of the dream: just draw a window, drop a button, and tell the computer what to do when that button was clicked. To many, it felt like the moment the barrier between user and developer would dissolve once and for all. To an extent, it did. But it also opened up a different narrative. Many applications built this way were fast to construct but incredibly fragile without a solid architecture backing them. As a system grows, knowing how to make a window pop up isn't enough anymore. You need to know how to define architecture, manipulate state, handle errors, maintain code, and ensure quality. The initial simplicity didn't eliminate the need for technical skills; it just pushed the problem down the road to a later stage of the product lifecycle. No-Code and the Myth of the Citizen Developer In the 2010s, with the explosion of the web and APIs, no-code and low-code carried the torch of a new promise. Platforms like Bubble, Webflow, or Zapier suggested that even those who couldn't code could build personal tools, automations, or full-fledged applications. This birthed the idea of the "citizen developer", a business professional who creates their own solution without going through IT. Here too, reality proved more nuanced. These platforms are phenomenal for prototyping, automating minor processes, and creating straightforward experiences. But the moment a project requires complex integrations, security, scalability, or nontrivial logic, you hit a wall. People can navigate the system, but they don't always have full control over it. AI Is Not the End of Programming Today, AI is making it easier to build the first version of an application, but it isn't eliminating the developer's job. What's changing is how they work, as I mentioned in another article: less time spent writing code, and more time dedicated to understanding the problem, defining requirements, guiding the tools, and verifying the output. The skills that matter now aren't just about syntax; they are about choosing the right solution for the context, anticipating errors, and knowing if a system will truly hold up. Anyone using AI can churn out software faster, but they can't always tell if the result is correct, secure, or sustainable. And that's exactly where the difference lies. A beginner can make something simple work. An experienced developer also knows how to explain why the system holds together, where it might break, and how to prevent it. The history of computing has already taught us that while every new technology takes a step forward in making software development more accessible, it never eliminates the need for specific expertise. The type of skill required changes, but its importance never does.

The biggest shift in local AI is no longer benchmark leadership but the ability to deliver language intelligence directly inside desktop and mobile applications without depending on cloud services. Apple Silicon is particularly well suited for this because its hardware and software stack is optimized for on-device inference. Core ML executes models across the CPU, GPU, and Neural Engine, while MLX leverages Apple's unified memory architecture, allowing computation on CPU or GPU without explicit memory transfers. Together, they enable responsive, private, and offline-first AI experiences by aligning software with Apple Silicon's architecture rather than treating local inference as a secondary deployment target. Latency Shapes User Experience For interactive applications, latency is more important than raw model size. Apple's Foundation Models framework reflects this philosophy by exposing the on-device language model behind Apple Intelligence with streaming support, offline execution, and on-device processing. Apple positions the model for focused tasks such as summarization, extraction, refinement, and short conversations rather than unrestricted chatbot interactions. Small models become highly effective when solving well-defined application problems with structured outputs instead of broad general knowledge. Apple's 2025 Foundation Models report reinforces this design. Its approximately 3-billion-parameter model is optimized specifically for Apple Silicon using techniques such as quantization-aware training and KV-cache sharing. Responsiveness is therefore engineered into the model itself. The same principle applies to open models running locally. Families such as Llama 3.2 provide lightweight 1B and 3B variants for edge deployment, while Gemma 4 includes compact E2B and E4B models spanning phones, laptops, and servers. In practice, the best model is usually the smallest one that consistently solves the intended task under realistic prompt and context conditions. Memory Determines Practical Deployment Selecting a model on Apple Silicon is largely a memory decision. Quantization reduces weight precision, and Core ML Tools supports both 8-bit and 4-bit quantization. MLX LM similarly treats quantization as a primary workflow, enabling Hugging Face models to be converted into reduced-precision variants with minimal effort. This matters because interactive applications usually become memory-bound before they become compute-bound. Model weights, KV cache, and prompt length collectively determine time-to-first-token. As a result, 1B–4B models generally provide a better balance between responsiveness and resource usage than significantly larger alternatives. Memory pressure also comes from autoregressive decoding. Apple's recent Core ML improvements support stateful models, allowing KV caches to remain as persistent model state instead of repeatedly passing them through input and output tensors. This reduces overhead while improving inference efficiency. Apple's Neural Engine research reaches a similar conclusion, showing that many transformer workloads become memory-bandwidth limited when repeatedly fetching large parameter tensors. Choosing the Right Runtime Runtime selection should follow deployment goals rather than framework preference. Core ML remains the preferred option for applications deeply integrated with Apple's platforms because it converts models into optimized Core ML formats while dispatching inference across CPU, GPU, and Neural Engine. When application requirements match Apple's built-in language model, the Foundation Models framework further simplifies development through guided generation, streaming, tool calling, and stateful sessions. Since the model is included within the operating system, applications avoid bundling large model files altogether. When open-weight models are required, MLX and llama.cpp become practical alternatives. MLX provides lazy computation, dynamic graphs, shared-memory arrays, and seamless CPU-GPU execution without explicit transfers. MLX LM extends these capabilities with model loading, streaming generation, prompt caching, rotating KV caches, and quantization support. llama. Meanwhile, llama. cpp emphasizes broad compatibility, using ARM NEON, Accelerate, and Metal to optimize Apple Silicon while supporting aggressive integer quantization and OpenAI-compatible HTTP serving. PyTorch MPS remains valuable for experimentation and training, although Apple primarily positions it as a Metal-accelerated development backend rather than a production inference runtime. Streaming Improves Responsiveness Perceived responsiveness depends less on maximum throughput than on how quickly useful information reaches users. Apple's Foundation Models framework addresses this through guided generation and streaming. Swift types define the expected response structure, allowing constrained decoding to produce strongly typed output without additional parsing. Swift @Generable struct Suggestions { @Guide(description: "Four concise search terms", .count(4)) var terms: [String] } let session = LanguageModelSession() let stream = session.streamResponse( to: "Generate search suggestions for CI build failures", generating: Suggestions.self ) for try await partial in stream { render(partial.terms) } This approach removes post-processing while allowing partial results to appear immediately. Rather than waiting for complete responses, applications progressively update the interface, reducing perceived latency and minimizing formatting-related errors. Streaming Beyond Apple's Foundation Models The same streaming principles apply to open models. MLX LM streams generated tokens immediately while supporting prompt caching and rotating KV caches for long contexts. These capabilities are especially valuable because many desktop applications repeatedly reuse the same background context. Caching that context once and appending only the latest query reduces repeated prefill work and improves responsiveness. Python repo = "mlx-community/Llama-3.2-3B-Instruct-4bit" model, tokenizer = load(repo) messages = [{"role": "user", "content": "Summarize this release note in three sentences."}] prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True) for chunk in stream_generate(model, tokenizer, prompt, max_tokens=160): append_token(chunk.text) Streaming allows interfaces to render useful information as soon as the first tokens arrive instead of waiting for an entire response. Even when overall throughput remains unchanged, incremental rendering creates a noticeably faster experience. Prompt caching further improves responsiveness by eliminating repeated computation for shared context across successive requests. Compression Should Be Part of Architecture Responsive AI applications are rarely the result of runtime selection alone. Model compression should be considered during application design rather than as a final optimization step. Core ML Tools supports post-training weight quantization, pruning, palettization, and joint compression techniques that combine multiple optimizations. Apple recommends these methods to reduce storage, memory usage, energy consumption, and inference latency while improving Neural Engine efficiency. Apple's transformer optimization research reinforces this recommendation by demonstrating that optimized implementations achieve significantly better performance and memory efficiency than default deployments. As a result, compression directly influences application responsiveness and should be integrated into the build pipeline instead of treated as packaging work. Specialization Outperforms Scale Specialization is equally important. Apple recommends beginning with prompting and tool calling before introducing custom adapters or fine-tuning. The same principle applies to open models. A compact language model connected to application tools, retrieval systems, or domain-specific data frequently outperforms much larger standalone models because it no longer relies on memorizing live information or generating rigid formats from unrestricted prompts. Smaller prompts and narrowly scoped tasks also reduce both prefill and decoding costs, improving latency while lowering memory pressure. Instead of selecting the largest available model, developers should identify the smallest model that consistently solves the intended workflow under realistic conditions. This approach improves responsiveness, reduces resource consumption, and simplifies deployment across Apple Silicon devices. Conclusion Small language models on Apple Silicon are valuable because they enable a different class of software rather than attempting to replicate cloud-scale AI. Apple's unified memory architecture, Core ML execution across dedicated compute engines, MLX's Apple-native runtime, and the Foundation Models framework all point toward the same engineering principle: responsive AI depends on keeping models compact, tasks narrowly scoped, context carefully managed, and outputs structured. When these principles are followed, local AI becomes a native application capability instead of a cloud-dependent feature. Combined with streaming, quantization, prompt caching, and workflow specialization, small language models deliver fast, private, offline-first experiences while maintaining the responsiveness users expect from modern Apple applications. Rather than treating local inference as a compromise, Apple Silicon enables developers to build AI experiences that feel immediate, efficient, and fully integrated into the operating system.

By Uthej Mopathi CORE

When you are pushing terabytes of weights to hundreds of GPU nodes, unicast stops being a solution. Here is what actually works — and where multicast still struggles. The Problem Engineers Hit at Scale If you have ever watched a 70-billion-parameter model take 20 minutes to load across a 200-node inference cluster, you have felt this problem in practice. The culprit is almost always the same: the model server opens a separate TCP stream to each receiver, saturating its own NIC before the first node finishes loading. This is not a configuration issue. It is the fundamental geometry of unicast in a one-to-many scenario. For every additional receiver you add, the sender's bandwidth demand grows linearly. Distribute a 1 TB model to 100 nodes, and you are generating roughly 100 TB of traffic — all of it originating from the same host, all of it transiting the same top-of-rack switch. The math is simple: unicast sends N copies of your model. Multicast sends one copy and lets the network replicate it. For large clusters, the difference is orders of magnitude. Network-layer multicast solves this at the right abstraction level. Instead of the application managing individual connections, the network itself handles replication — copying packets only where distribution paths diverge. The sender transmits once; every receiver gets it. The practical upside: distribution time stops scaling with receiver count and becomes approximately constant. That said, multicast is not a drop-in replacement for your current distribution stack. The tradeoffs are real, and understanding them determines whether multicast belongs in your architecture. How Network Multicast Actually Works The mechanics are worth understanding before you evaluate whether to deploy this. When a node wants to receive multicast traffic, it joins a group address (e.g., 239.1.1.1 in the administratively scoped IPv4 range) by sending an IGMP membership report to its local router. The router records that interest and propagates it upstream. The network builds a distribution tree — typically using PIM-SM (Protocol Independent Multicast – Sparse Mode) or PIM-SSM (Source-Specific Multicast) for most data-center deployments. Packets enter the tree at the source and are replicated at each branch point as they flow toward receivers. No link carries duplicate traffic unless required by topology. The Distribution Workflow for Model Loading 1. Nodes scheduled to receive the model join a multicast group, typically identified by model version or checkpoint hash. 2. The model server segments the weights file into fixed-size chunks (commonly 64 KB–1 MB depending on MTU and FEC overhead) and begins transmitting to the group address. 3. Switches and routers replicate packets along the multicast tree. No receiver is privileged — all get the same stream simultaneously. 4. Each receiver tracks which chunks it has received, reassembles the model in shared memory, and loads it into accelerator memory once complete. 5. Missing chunks trigger repair requests. How those are handled is where implementation complexity lives. The result is synchronized parallel delivery. In a well-engineered deployment, you can go from "model server starts transmitting" to "all 200 nodes ready for inference" in roughly the same time it would take to deliver to one node over unicast. Unicast vs. Multicast: Side-by-Side Here is a direct comparison for a 1 TB model to 100 nodes: dimensionunicastnetwork multicast Traffic at sender N × model size 1 × model size Scales with receivers? Linearly worse Near-constant Congestion risk High (sender ToR) Distributed Reliability TCP guarantees Must be engineered Ops complexity Low Medium–High Best fit Small clusters, <20 nodes Large clusters, HPC, bootstrapping The bandwidth story is unambiguous. The reliability and operational story is where the real engineering work lives. The Reliability Problem (and How to Engineer Around It) Standard IP multicast runs over UDP. There is no acknowledgment, no retransmission, no ordering guarantee, and no congestion control. Drop a packet, and the network does not notice. For distributing cat videos, this is fine. For distributing model weights, it is not — a single missing chunk means every receiver that lost it cannot reconstruct the model. In practice, this is solvable, but it requires deliberate engineering. The approaches that work in production: 1. Application-Layer Reliability This is the most common approach for custom implementations. The sender assigns a sequence number to every chunk. Receivers track which sequences arrived. After a transmission window completes, receivers that missed chunks broadcast a NACK (Negative Acknowledgment). The sender retransmits missing chunks — typically via unicast to the specific requester to avoid generating duplicate traffic on the multicast tree. Practical tip: Use a NACK aggregation window (50–100 ms is a reasonable starting point) to avoid NACK implosion when many receivers miss the same chunk simultaneously. Collate NACKs server-side before deciding what to retransmit. 2. Forward Error Correction FEC (Raptor codes or Reed-Solomon are common choices) adds redundant encoded symbols to the stream. Receivers can reconstruct the original data from any sufficiently large subset of received symbols, even without a retransmission round-trip. This trades increased bandwidth (~5–10% overhead) for near-zero retransmission latency — useful when the network has predictable, bounded loss rates. Practical tip: FEC works best when loss is random and bounded. If you are seeing burst loss from switch buffer overruns, fix the congestion first — FEC will not save you from a sustained drop rate above its recovery threshold. 3. Hybrid Multicast/Unicast A pragmatic middle ground: use multicast for the initial bulk transfer (which has the highest bandwidth leverage) and fall back to unicast for repairs. Most receivers get 100% of chunks from the multicast stream. Stragglers use point-to-point retransmission to fill gaps. This avoids the complexity of pure reliable multicast while capturing most of the bandwidth benefit. 4. RDMA Multicast in HPC Fabrics If your cluster runs InfiniBand or RoCEv2, you have access to reliable RDMA multicast (UD multicast with software reliability layers, or IB reliable multicast extensions). This is not available in standard Ethernet fabrics but is worth noting for HPC and specialized AI hardware deployments. Why Most Hyperscalers Do Not Use Native IP Multicast This is the part that surprises engineers who arrive at this problem from a networking background. The bandwidth math is obviously favorable. So why are hyperscale AI clusters not running multicast everywhere? Three reasons, in order of practical impact: Control-plane complexity at scale. PIM state grows with the number of active groups and sources. In a dynamic AI cluster where job scheduling creates and tears down groups constantly, multicast routing state can become a significant operational burden. Debugging a stuck join or a flapping tree in a 10,000-node fabric is not straightforward.Application-layer alternatives are mature and integrated. NCCL (NVIDIA Collective Communications Library) provides AllReduce, Broadcast, and Scatter operations that are already optimized for GPU-to-GPU communication patterns. They integrate directly with PyTorch and JAX, handle topology awareness, and have years of production hardening. Building reliable multicast transport is an engineering investment that competes with "just use NCCL."Unicast TCP is boring in the best way. It has known failure modes, well-understood debugging tools, and works without fabric-level multicast support. For clusters below roughly 50–100 nodes, the bandwidth overhead of unicast is often acceptable. The honest framing: multicast is not universally better. It is specifically better for large-cluster, one-to-many distribution where bandwidth is the binding constraint and you are willing to invest in the reliability layer. Where Multicast Fits in the Current AI Infrastructure Landscape Given those tradeoffs, here are the deployment contexts where network multicast genuinely earns its complexity cost: Large Inference Farms When you are starting up hundreds of replicas of the same model simultaneously — a common pattern in autoscaling inference serving — multicast collapses what would be a serialized loading queue into a single parallel delivery. The bandwidth savings at this scale are substantial, and the operational overhead of managing multicast groups is manageable because the topology is relatively static. Checkpoint Synchronization in Distributed Training During training, periodic checkpointing saves model state to distributed storage and sometimes requires re-broadcasting the latest checkpoint to restore a failed worker. This is a clear one-to-many pattern where the checkpoint (potentially hundreds of gigabytes) needs to reach a set of known receivers simultaneously. Multicast is well-suited here. Private AI Clusters and HPC Environments If you control the fabric end-to-end — your own switches, your own routing, predictable topology — the operational complexity of multicast is much lower than in a multi-tenant cloud environment. This is where reliable multicast protocols have historically seen the most traction, and it remains the most viable deployment context today. Model Bootstrapping at the Edge Edge inference deployments (think CDN-scale or industrial IoT) often need to push model updates to large numbers of geographically dispersed nodes. Application-layer multicast over IP overlay networks (similar to BitTorrent-style distribution) is common here, though it trades network-layer efficiency for deployment simplicity. Practical Implementation Guidance If you are evaluating multicast for a specific use case, here is a concrete starting framework: Step 1: Validate Your Fabric Supports Multicast Before writing any application code, confirm that your switches support IGMP snooping (for confinement within VLANs) and that PIM is enabled on your router interfaces. In cloud environments, check whether your VPC supports multicast — many do not by default, and overlay solutions (GRE tunnels, VXLAN with multicast underlay) add latency and complexity. Shell # Quick sanity check on a Linux node ip maddr show # View joined multicast groups netstat -gn # Group memberships with interface tcpdump -i eth0 'ip[16] >= 224' # Capture multicast traffic Step 2: Design Group Namespace Multicast group address assignment matters for operational clarity. A practical scheme for AI workloads: Use SSM (232.0.0.0/8) rather than ASM to avoid Rendezvous Point complexityEncode model version or checkpoint ID into the group address or use a lookup tablePlan for group lifecycle — join on job start, leave on completion, and ensure IGMP leave messages propagate promptly Step 3: Build the Reliability Layer Explicitly Do not assume UDP reliability. At minimum, implement: Chunk sequencing with 64-bit sequence numbersPer-receiver bitmap tracking of received chunksNACK aggregation and retransmission (unicast repair is usually simpler)End-to-end checksum validation before model load Performance note: For a 1 TB model with 1 MB chunks, you have ~1 million sequence numbers to track per receiver. Use a sparse bitmap, not an array, or memory overhead becomes significant. Step 4: Test Failure Modes Deliberately The failure modes that will bite you in production are not random packet loss — they are: Late joiners: a node that joins mid-transfer needs either a full retransmit or a catch-up mechanismReceiver asymmetry: nodes with different NIC speeds or CPU load will have different loss profilesSwitch buffer overruns during the initial burst: implement sender-side rate limiting (start at ~70% of available bandwidth, tune up) What Is Coming Next The gap between multicast's theoretical efficiency and its practical deployment complexity is narrowing. A few developments worth tracking: Smart NIC and DPU offloads are pushing reliability processing off the host CPU, making application-layer reliable multicast cheaper to implement and operate. NVIDIA BlueField DPUs, for example, can handle NACK processing and chunk reassembly in dedicated network processing cores.SDN-orchestrated multicast trees — where a controller computes and installs multicast forwarding state based on real-time cluster topology — remove much of the per-hop PIM complexity and enable faster group setup/teardown in dynamic job-scheduling environments.Hardware vendors are adding native multicast acceleration to AI fabric switches. NVSwitch (in NVLink domains) has supported hardware multicast for GPU collective operations; similar capabilities are appearing in Ethernet-based AI fabrics.The IETF RIFT working group has active proposals around multicast-aware link-state routing for AI data centers, including MoE (Mixture-of-Experts) multicast use cases where different model experts are selectively distributed to different nodes. For exascale training runs and inference farms in the tens-of-thousands-of-nodes range, the bandwidth economics of multicast become increasingly hard to ignore. The infrastructure to support it reliably is maturing to match. Bottom Line for Practitioners Network-layer multicast is not a silver bullet, but it is the right tool for a specific problem: one-to-many distribution of large, identical payloads to clusters large enough that unicast bandwidth becomes the binding constraint. That problem is increasingly common as AI model sizes grow and inference clusters scale. The implementation cost is real — you need a reliability layer, fabric support, and operational tooling. For clusters under ~50 nodes or in environments where application-layer solutions like NCCL already cover your communication patterns, the tradeoff may not be worth it. For large-scale inference serving, checkpoint broadcasting, or HPC-style model distribution, it is worth the engineering investment. If your model loading time scales linearly with cluster size, multicast is the architectural lever that can make it near-constant. That is the question to ask before committing to either path. Further Reading Abdous et al., "One to Many: Closing the Bandwidth Gap in AI Datacenters with Scalable Multicast" — HotNets 2025NVIDIA NCCL Documentation — developer.nvidia.com/ncclIETF RIFT WG: LLM MoE Multicast use case — datatracker.ietf.org

Almost every conversation about observability budgets I have been in ultimately arrives at the same conclusion: “we need to reduce our telemetry volume.” That sentence is usually followed by a number. Thirty percent. Half. Whatever the finance spreadsheet needs it to be. Then someone says the thing that makes everyone in the room relax. "Good news: most of it’s noise anyway. We can cut the volume and improve the signal at the same time." It is a comforting idea, because it turns an unpleasant budget cut into an engineering improvement. But it only gets you so far. It is true that some of your telemetry is noise, but it’s much less of it than "most." But it doesn’t follow that you can then simply cut volume and automatically improve signal. There is real noise in your telemetry, and I will get to where it lives. But "reduce volume by thirty percent" is not an instruction to remove noise. It is an instruction to remove bytes, and your noise and your signal are made of the same bytes. The target doesn’t differentiate, so what you end up removing is dictated by whatever is easiest to find. What is easy to find is a category. All INFO logs. All user agent strings. Everything below WARN. Categories are easy because your pipeline already knows them, and that is the whole of their appeal. Whether a category happens to be useful or not is a coincidence. So your telemetry is full of junk, but the problem isn't that there is too much of it. It is that by adopting a volume reduction target, you are not looking at whether the telemetry data you cut has any value. Once you hit the byte target, the exercise is seen as a success. Two Axes, Loosely Coupled When you change your telemetry pipeline, two things move. The first is easy: bytes through the pipeline, or active series if it is metrics, or whichever unit your contract happens to price. One number, on a chart, updated hourly. This is what we call volume. The second is what those bytes enable you to find out. Whether, six weeks from now, you can still answer the question in front of you. This is what we commonly call signal, and everything else is noise. It is measurable, but it is not measured in bytes, and it is probably not on any chart you are currently looking at. The two are related, obviously. Delete everything, and both go to zero. But across the range you actually operate in, they are only loosely coupled, because the bytes in your telemetry are not distributed anything like the value. The smallest fields often do the most work. A tenant identifier is a few dozen bytes, and it tells you whether something is impacting everyone or just one customer. A trace ID is thirty-two hex characters, but without it you are correlating your signals by hand, across three browser tabs. If the resource attributes naming the deployment are missing, good luck telling a bad release from a bad node. On the flipside, fields that take the most space frequently do the least. Meanwhile, the ten-thousandth identical stack trace in an hour is several kilobytes and tells you the same thing the first one did. So a lever that operates on bytes will spend most of its effect in the wrong place, and no exchange rate exists that would let you convert one axis into the other. Drawing them as two axes is a crude picture for that reason. But it is still worth doing, because it separates four moves that a byte count reports as only two. Let me walk through each one. Q1: The Free Lunch, Real But Limited This is the noise I promised at the top, and finding it feels great. Every tutorial on making your observability pipeline better has these prominent examples: Kubernetes liveness and readiness probes logging every few seconds, per pod, forever. A debug logger somebody enabled during an incident last quarter, and nobody turned off. The same records shipped twice because a node agent and an application-level exporter both picked them up. Most of this can go. But be careful even here, because a health check is not the same thing as a worthless record. Probe failures and probe latency are how you find a sick node before your users do. What you want to drop is the successful ones, the ninety-nine percent that only ever confirm that nothing is happening. The filter processor will do it: YAML processors: filter/healthchecks: log_conditions: - 'IsMatch(log.attributes["http.route"], "^/(healthz|readyz)$") and log.attributes["http.response.status_code"] == 200' This assumes http.route has been promoted onto the log record; it is a span attribute by default, so on the trace side the equivalent lives under trace_conditions, with a span. prefix instead of log.. That status code check is the difference between Q1 and Q2. Without it, you have removed probe observability rather than probe noise, and you will find that out the next time readiness starts flapping and nothing in the logs can tell you when it began. With it, volume goes down, and signal is untouched, or arguably goes up, because you are no longer scrolling past successful probe traffic to find a real request. Sounds like a good deal, right? This is the quadrant everybody is imagining when they say "most of it is noise anyway." The same trade is available on the retry storm that repeats one stack trace ten thousand times in an hour. The logdedup processor collapses each ten-second window into one record carrying the count, so the storm stops drowning the query you are running, and you can still see how big it was. Finding the rest of this kind of waste means clustering records by shape and looking at what dominates, which is a different class of tool than a filter, and it is the part most volume-reduction programs skip. The challenge is that this quadrant is finite. In my experience, it is somewhere in the range of 10-20%, depending on how neglected the pipeline has been. If your mandate was 30%, you exhaust Q1 in the first week, and then you keep going, because the mandate does not stop when the free lunch does. Q2: Paying With Data Instead of Money So the free lunch got you 15%, the middle of that range, and the mandate was 30%, so the next 15% has to come out of data that somebody might actually need. Which is a good moment to read the mandate again, because almost nobody means it literally. "We need to reduce our telemetry volume by thirty percent" is very rarely a statement about telemetry. It is a statement about an invoice. Does anybody in that meeting actually want fewer log lines? They want a smaller number at the bottom of a bill. Volume is simply the variable their contract happens to be calculated on. The distinction matters because volume reduction and reducing your bill have different solution spaces. Reducing volume by 30% has one family of answers, and every one of them involves deleting something. Reducing observability spend by 30%, has a different set of options, several, and deleting your data is the one with the worst terms. A logging config goes from INFO to WARN and ships with the next release. Retention drops from thirty days to seven. Traces get sampled at 5%: YAML processors: probabilistic_sampler: sampling_percentage: 5 None of these options is free. Each one of them is defensible in isolation, and what makes them defensible is that they have a big impact. INFO is most of your log volume, seven days covers most incidents, and 5% is a perfectly good sample if all you want is a latency distribution. You end up paying the bill twice, but only one of the payments shows up on the invoice. You are also settling the bill in a second currency: answers you will not have, because you didn’t store the data needed for them. Nobody counts that. Nothing fails and nothing alerts, because a trace that was never recorded does not raise anything. When a customer sends an order ID on Thursday, and the trace behind it was one of the ninety-five per cent, the investigation stalls; somebody says we do not have that, and nobody goes back to look at the config change from earlier in the year that caused it to be dropped. My position is that most of this work should not exist. The engineering is fine! The sampler is correct, the retention change is correct, and both do exactly what they say on the tin. It is just that the whole exercise is effort spent making a bad unit price easier to swallow. It's like an old fridge: defrost it, keep the door shut, put less in it, and yes, your bill really does go down every month. Somebody should still go and look at what a new fridge costs. Q3: The Enrichment Nobody Gets To There is a second way to improve signal-to-noise: instead of removing noise, you add signal. You make the data you are already paying for be more useful. Attaching Kubernetes and cloud metadata with the k8sattributes processor, so a log line knows which namespace, deployment, node, and pod produced it. Parsing an unstructured message body into named, queryable fields with OTTL. Making sure trace context actually propagates across the boundary where it currently drops, so your logs and traces can be correlated instead of merely coexisting. Carrying code.file.path and code.line.number on the records that warrant it, so a log line points at the statement that emitted it instead of leaving you to grep the repository for the format string. YAML processors: k8sattributes: extract: metadata: - k8s.namespace.name - k8s.deployment.name - k8s.pod.name - k8s.node.name transform/parse_access_log: log_statements: - context: log statements: - merge_maps(attributes, ExtractPatterns(body, "^(?P<method>\\w+) (?P<path>\\S+) (?P<status>\\d{3}) (?P<duration_ms>\\d+)$"), "insert") These changes make your telemetry substantially more valuable, but they also increase volume. But most of that is cheaper than you would guess. The Kubernetes metadata are resource attributes, written once per batch in OTLP and shared by every record from the same pod, so at the collector's egress they cost a fraction of a byte per record. The parsing is the real exception: you keep the original body alongside the extracted fields, so the record roughly doubles, and no amount of batching recovers that. A bytes-per-day chart shows you none of that. The enrichment that costs almost nothing and the one that doubles every record show up the same way: the budget line went up. So the work never really gets argued about. Nobody is blocking k8sattributes – it ships enabled in half the Helm charts you might install – and most teams already intend to do all of the above. They just do not do it now, because a volume program has a number in it, and programs with numbers in them end when the number is hit. Q1 gets you fifteen percent, Q2 grinds out the rest, somebody screenshots the graph for the quarterly review, and the work is closed. There is no step after "we reduced it by 30%," because reducing it by 30% was the entire brief. Whether your observability spend is value for money is unanswerable while the telemetry is unusable. You can't defend a bill for data nobody can query, and you can't really attack it either, so the argument settles on price – the only number anybody in the room actually has. Enriched telemetry gets used, and usage is evidence. Most of what produces it is unglamorous work: consistent structure, correlation IDs that survive a hop, log levels that mean the same thing across services. But a team that can name the investigations that resolved faster this quarter, and the correlation that did it, walks into the budget meeting with something to say. Q4: The Change You Were Sure About The framework logs the request. Then the middleware logs it, because the framework's version does not carry the tenant. Then the application logs it a third time with slightly different wording, because by that point nobody trusts the other two. Three records, one event, and no reliable way to say which is authoritative. Every one of those lines was added by somebody trying to improve matters, and each has a different team behind it. That is what Q4 actually is, and why I think of it as the backfire. It is not really the stuff that piles up while nobody is looking; that was the double-shipping back in Q1, where either copy is safe to delete because they are identical. These three records differ from one another, and none of them goes without a conversation. Logging whole request and response bodies for completeness is the same story: you add a great deal of data, and the four fields anybody queries end up inside a blob that nothing has parsed. The same thing happens with a processor from the previous section. Take the k8sattributes block from Q3, change nothing about it, and point it at a different pipeline: YAML service: pipelines: metrics: processors: [k8sattributes] On logs, that was enrichment. On metrics, as soon as the backend treats resource identity as series identity, it is a separate series for every pod – and a fresh set of them after every deploy, because pod names churn. That is how a well-meaning label addition takes out a Prometheus. The config did not change, and neither did the intention behind it. Underneath all three is an assumption that more data is the same thing as more signal, and that if the answer is not in there yet then adding should get you closer. It is the same mistake the volume mandate makes, pointed the other way, and I have watched one team make both inside about two years. The awkward thing is that Q3 and Q4 are not separable at the time, and not only on the chart. From the inside, they are the same act: somebody adds something to a pipeline because they are fairly confident it will help. The engineer putting a pod name on a metric is doing what the engineer putting it on a log did. One of them is right. Review will not catch it either, because the reviewer is working from the same information and the same instinct. You need something that checks whether a question actually got easier to answer. What to Govern Instead Put the four quadrants back together, and the problem shows up in one line. Q1 and Q2 both report as a reduction in volume, so dropping probe traffic and dropping the log lines that explain a failure show up in the quarterly review as the same green arrow. Q3 and Q4 both report as volume up, so the enrichment that made an incident tractable and the label that took out your metrics backend are reported as the same red arrow. A bytes-per-day number cannot separate any of that, but it is the number the entire program is steered by. None of which is an argument against governing telemetry. It grows without limit if nobody is watching, somebody has to own the bill, and a team that has never questioned its telemetry costs is not being principled, is just not looking. The argument is about which variable should be on the dashboard. The goal is to try and measure signal, and it is less work than it sounds. Take the ten questions your team actually asks during an incident. Can I segment this failure by tenant? Can I get from this alert to the trace that caused it? Can I tell which deployment introduced it? Write each one as a literal query, in a file, checked into the repository that holds your collector config, and run them in CI against a replay of real telemetry, once with the proposed change and once without. If any answer moves, the build fails. Not just if it comes back empty: sampling does not empty a result; it quietly changes it. That is the difference between Q3 and Q4 made mechanical. The engineer adding pod name to a metric finds out in the pull request instead of during the next incident. It works in reverse too, which is the part that matters for Q3: adding a question and watching it fail is how you justify an enrichment to somebody whose only other number is bytes per day. And if you would rather start with something off the shelf, the Instrumentation Score is an open specification for grading OTLP against semantic conventions and instrumentation best practice, which is a different cut at the same question. Either way: your observability pipeline is probably the only production system you own with no tests on it, and there is no particular reason for that. Changing the Constraints I want to end somewhere slightly uncomfortable, because I do not think this is really a discipline problem or an education problem. Which quadrants you can operate in is dictated by your observability platform's cost model, not by your engineers. If ingest cost scales linearly with bytes, and retention is tiered so that older data becomes slow or expensive or both, then the economics have already made your architectural decisions. Q3 is priced out of existence. Q2 becomes not just permitted but mandatory, because it is the only lever that moves the number anybody is measured on. Your telemetry strategy is a downstream consequence of a pricing page. Teams under that constraint are not making bad choices. They are making the only choices available, and then rationalizing them as noise reduction, because "we improved our signal-to-noise ratio" is a much better sentence than "we deleted data we may need." The interesting question is what changes when volume stops being the binding constraint. When enriching a log record does not require a budget conversation, the matrix opens up. You can attack Q4 aggressively and invest in Q3, which is the combination that actually improves the ratio. Until then, at minimum, name the quadrant. When somebody proposes a pipeline change, ask which of the four it is. It is a five-second question, and I have not yet seen it fail to change the conversation.

Data engineers managing batch SQL pipelines on Snowflake, BigQuery, and increasingly Databricks, and streaming pipelines on Apache Flink face a familiar problem: two toolchains, two skill sets, two CI/CD pipelines.dbt is now extending into stream processing. This post explains what that means in practice, why it matters for data engineering teams, and what a concrete implementation looks like with Apache Flink on Confluent Cloud. Data Streaming Meets the Lakehouse Data lakes promised to solve the enterprise data problem. The reality has been messier. Batch pipelines produce stale information, and analytical workloads run hours after the business event occurred. By the time a query runs, the window for action is often already closed. The lakehouse pattern has improved matters. Apache Iceberg has become the dominant open table format, supported across Snowflake, Databricks, BigQuery, and a growing number of query engines. Teams can run SQL analytics directly on data in object storage without duplicating it into a proprietary warehouse. But the lakehouse alone does not solve the real-time problem. Data still arrives as a batch, minutes or hours after the source event. That gap reflects a deeper architectural split. Data streaming with Apache Kafka and Flink is the operational layer: it handles critical SLAs, powers event-driven applications, and keeps business systems running in real time. The lakehouse is the analytical layer: it stores historical data for reporting, ML, and near real-time or batch analytics. These are two distinct workloads with different requirements regarding uptime, data loss, latency, and throughput. They need to coexist without forcing engineers to build and maintain two separate pipelines. How Kafka, Flink, and Iceberg Work Together That is what the combination of Apache Kafka, Apache Flink, and Apache Iceberg addresses. Kafka captures every event at the source and serves as the operational backbone for real-time systems. Flink processes and enriches data in motion, supporting both immediate operational decisions and the preparation of data for downstream analytics. Iceberg stores the result as a governed, queryable table for any analytical engine, whether that is Snowflake, BigQuery, or Databricks. A full treatment of this architecture, including schema evolution, compaction, and catalog integration, is covered here: Data Streaming Meets Lakehouse: Apache Iceberg for Unified Real-Time and Batch Analytics. The question is no longer whether streaming and lakehouse architectures can coexist. They already do. The question is how data engineering teams can work across both without maintaining separate toolchains. That is where dbt enters the picture. What Is dbt? dbt, the data build tool, is an open-source framework for SQL-based data transformation. A dbt model is a SQL SELECT statement saved as a file. dbt infers execution order from how models reference each other using ref(). The standard commands cover the full engineering workflow: dbt run executes the SQL against the target platform, dbt test validates data quality, and dbt docs generate produces a browsable documentation catalog. What made dbt successful is the discipline it brings to SQL work. Before dbt, transformation logic lived in scattered scripts and proprietary ETL tools. dbt replaced that with a code-first, version-controlled workflow with built-in lineage, testing, and documentation. Snowflake and BigQuery are where most dbt adoption lives today. Both are SQL-native and optimized for the ELT pattern dbt was built around. Redshift is a strong third platform in AWS environments. Databricks has seen growing dbt adoption more recently, driven by investments in serverless SQL Warehousing, but its roots are in Spark and Python, making it a newer entrant in the dbt ecosystem. dbt Labs crossed $100 million in ARR in early 2025, with over 5,000 paying customers. Around 90,000 dbt projects are running in production today. The Fivetran and dbt Labs merger, announced in October 2025, created a combined data infrastructure company with nearly $600 million in annual revenue — a clear signal that dbt has moved well beyond a popular open-source tool and into foundational enterprise data infrastructure. dbt Meets Apache Flink: One Workflow for Data Engineers Data engineering teams managing both batch and streaming today operate in two separate realities. Snowflake or BigQuery on one side: dbt models, version-controlled SQL, automated tests, generated docs. Apache Flink on the other: Terraform scripts, custom deployment code, or the Flink console. Skills and practices do not transfer between the two. That separation has a real cost. Streaming pipelines are harder to test, harder to document, and harder to hand over. Many teams compensate by keeping streaming logic minimal and pushing transformation work downstream into the warehouse, which reintroduces latency and undermines the point of streaming. The vision is straightforward: one SQL workflow for both. The engineer who builds dbt models on Snowflake or BigQuery should be able to apply the same approach to an Apache Flink streaming pipeline, without switching tools or rebuilding CI/CD from scratch. Two toolchains mean two testing strategies, two documentation systems, and two skill sets to hire and retain. Governance enforcement becomes inconsistent across the two environments. SQL is the shared foundation that makes this realistic. Flink SQL is mature and production-proven. Snowflake and BigQuery are SQL-native. Apache Iceberg tables are queryable via SQL across multiple engines. dbt wraps SQL with engineering discipline. The model files look the same. The ref() dependency resolution works the same way. Tests and documentation generation work through the same commands. Organizations do not need to hire separate Flink infrastructure specialists. The existing data engineering team can own both sides. Apache Iceberg connects the two worlds at the storage layer. A Flink pipeline writes structured, governed events into an Iceberg table in the organization's own S3 bucket. That same table is immediately readable by Snowflake, BigQuery, or Databricks without any additional ETL step. dbt can model data across the full pipeline: shaping it as it streams through Flink, and transforming it again when it lands in the warehouse for analytics. This is also a direct enabler of the Shift Left Architecture 2.0. The Shift Left approach moves data integration logic closer to the source, applying quality checks, enrichment, and governance in the streaming layer before data lands in the lakehouse. Until now, that required streaming-specific skills that most dbt-native teams did not have. dbt for Flink lowers that barrier considerably. The full architectural detail is covered here: The Shift Left Architecture 2.0: Operational, Analytical and AI Interfaces for Real-Time Data Products. Concrete Example: dbt on Confluent Cloud with Apache Flink The most concrete implementation available today is the dbt-confluent adapter, released by Confluent alongside the confluent-sql Python driver. Both are open source and available on PyPI and GitHub. Data engineers define streaming pipelines as dbt models and deploy them to Flink compute pools using the standard dbt run command. Getting started is a single step: pip install dbt-confluent Three materializations are supported: view for a virtual Flink SQL view over a Kafka topic, streaming_table for a continuous always-current result set, and streaming_source for defining a Kafka topic as a dbt source. Testing is deterministic, using Confluent Cloud's snapshot query capability to return bounded point-in-time results rather than silently passing on timeout. Documentation generation works through INFORMATION_SCHEMA integration, producing the same browsable catalog that Snowflake and BigQuery projects generate. The underlying confluent-sql driver is DB-API v2 compliant, meaning any compatible tool can connect directly to Confluent Cloud Flink: Airflow and Dagster for orchestration, Pandas for snapshot queries, Streamlit for live dashboards, and LangChain for AI agent workflows. For data engineers already working in dbt, this means the skills and practices built around Snowflake or BigQuery transfer directly to the streaming side of the architecture. The Data Engineer Owns Batch and Streaming with dbt The separation between batch and streaming engineering has always been more organizational than technical. Both worlds use SQL. Both require testing, documentation, and reliable deployment. The tools just never bridged the gap, so organizations staffed and operated two distinct engineering disciplines. dbt extending to Apache Flink changes that equation. The data engineer who runs dbt on Snowflake or BigQuery today can apply the same mental model, commands, and CI/CD pipeline to Flink streaming pipelines. No Flink infrastructure specialization required. They write SQL models, define tests, generate documentation, and deploy, exactly as they do for batch. The implication is straightforward. The investment in dbt skills and tooling now extends further into the architecture. Streaming can be adopted incrementally by the same data engineering teams already trusted for batch. One team, one tool, one governance standard, across both operational and analytical workloads. The Flink adapter for dbt is earlier in maturity compared to dbt on Snowflake or BigQuery, and teams should expect to work with an evolving ecosystem. But the foundation is solid, the direction is clear, and the core architectural components are already running in production at scale across multiple industries. The demand from data engineering teams is real and growing.

By Kai Wähner CORE

When you’re standing up a web app, developers are trying to build required functionality quickly and at a low cost: budgets are still tight, teams are small, and resources are thin. These teams often turn to open-source tools to add PDF viewing functionality, and these libraries work: they give you the basics with simple integration and zero cost. However, whether you’re a scrappy startup or an established organization building new functionality in a platform with a large existing user base, open-source libraries can become a bit of a monkey’s paw: they’ve granted your wish, but the pain comes later. In the case of document processing functionality, this pain comes in the form of integration hell, as you need to add more document capabilities one after another, and the capabilities and dependencies of all the open-source libraries you’ve integrated start to show some cracks. Maintenance grows, user experience suffers, and developers are gently resting their foreheads on the desk. Basic PDF Library vs. Document SDK: What to Choose? The best PDF library for your app depends on how far your document requirements are going to grow, not on how they start. While a basic library renders a document and handles one or two operations well, a document SDK covers the full range a growing application eventually needs: viewing, annotation, editing, redaction, security, and accessibility, from a single license. The table below breaks down the three tiers teams typically move through as document requirements expand. The Three Tiers of PDF and Document Tooling Document tooling generally falls into three tiers, and knowing which one you are in is the first step toward the right decision. Tier Option Best for Limitation1Basic PDF library (open-source, for example, PDF.js, PDFBox, MuPDF)Simple, single-purpose PDF manipulation: view, merge, splitLimited scale, support, and feature breadth. Your team owns maintenance and vulnerability patching.2Point API / cloud document API (for example, Adobe PDF Services, AWS Textract, Azure Document Intelligence, Google Document AI)One specific task like conversion or OCR, fast to prototypeDocuments leave your environment. Per-page costs compound at scale. Adding a second task means fragmented workflows across vendors.3Full document SDK (for example, Apryse)Embedded, scalable document workflows across web, server, and mobileRequires more upfront integration planning than dropping in a single-purpose library. Basic PDF Library (Open-Source) PDF.js, PDFBox, and MuPDF are free, source-available, and fine for a basic viewer. PDF.js is the default free web viewer, built into Firefox, and wins the zero-cost use case outright. MuPDF is a proven rendering engine with decades of use behind it. The limitation shows up once the requirement grows. PDF.js loses fidelity on complex documents, redaction, signatures, and compliance formats like PDF/A and PDF/UA. MuPDF ships as a C-level API with no viewer UI, annotation layer, or forms support, which raises integration cost for anything beyond rendering. All three are single-purpose by design, so a non-trivial workflow means stitching several libraries together and maintaining the glue code between them, with no vendor accountable when something breaks. For a closer look at the tradeoffs between the two models, check out the article open-source vs. proprietary PDF SDKs. Cloud Document API Adobe PDF Services, AWS Textract, Azure Document Intelligence, and Google Document AI get you to a working prototype fast. You call an endpoint, get a converted file or extracted text back, and the vendor manages the scaling behind it. For low or unpredictable volume, pay-as-you-go pricing can make sense. The tradeoff is what happens once you need more than one capability. Each task — conversion, OCR, extraction — tends to live behind a different vendor endpoint, and every one of those endpoints is a place your documents leave your environment before the workflow finishes. Per-page or per-call pricing compounds at production volume, and none of these four hyperscalers offer an air-gapped or offline option if your compliance posture requires it. Full Document SDK A full-document SDK puts extraction, redaction, conversion, and signing behind one engine, instead of several vendors glued together with different conditional code paths. The Apryse PDF SDK runs inside your own environment, whether that is your VPC, on-premises, or fully air-gapped. Document content does not route through a third party to get processed. While Apryse offers a full suite of document-processing capabilities, different tools are licensed as separate add-ons, so you’re not paying for capabilities such as digital signatures or secure redaction unless you actually need them. For example, Docaposte moved its document conversion pipeline to Apryse and saw conversions run 16 times faster than its prior setup. Apryse also runs production document workflows for Dropbox, with more than 700 million users, and Egnyte, across 17,000 businesses. How to Tell When You've Outgrown a Basic PDF Library For developers, it may be time to recognize that your basic PDF library is no longer enough when one or more of these shows up in your backlog: Rendering breaks or slows down on complex or large files your library was not built to handle.Your team is maintaining two or more separate libraries stitched together for one workflow.The roadmap now asks for annotations, redaction, or e-signatures your current library does not support.A compliance requirement shows up, such as SOC 2, ISO 27001, or a data residency rule your current stack cannot meet.Your product needs to render and edit documents consistently across web and mobile, not just one platform.Engineers are spending sprint time patching an open-source dependency instead of building product features. Any one of these on its own might be manageable, but dealing with more usually means the maintenance cost of the current setup has started to exceed the cost of moving to a document SDK. Best PDF Library for Enterprise Apps: What to Evaluate Enterprise-grade performance isn’t just for large organizations. When it’s time to migrate from free libraries to a document SDK, evaluate these criteria to get an enterprise-grade solution: Performance at scale: How does the solution handle concurrency and large, complex files?Feature breadth across the document lifecycle: Does the solution provide viewing, annotation, editing, redaction, and signing from one vendor instead of a different license for each?Security and compliance posture: Look for true content redaction, which permanently removes underlying text and image content rather than masking it visually, plus other document security features such as encryption. On the vendor side, look for independent certifications like SOC 2 and ISO 27001.Support and SLAs: Does the vendor offer a dedicated point of contact for open issues?Deployment control: Can the SDK run on-premises, in your VPC, or fully air-gapped, or does it require routing documents through a vendor's cloud?Licensing model: Does the vendor license cover the full feature set, instead of a separate product and a separate contract for each platform or capability? PDF SDK vs. API: Avoiding Fragmented Workflows An API service solves one task well, but problems can start when the second task arrives. Conversion from Adobe, OCR from AWS Textract, and extraction from Azure Document Intelligence means your application accumulates a different conditional code path for every provider, plus potentially a whole new data residency questionnaire to answer during procurement processes. Check out the article, A Developer’s Guide to Reducing Dependencies, to learn more about vendor consolidation. Apryse consolidates that surface area into a single solution. Office-to-PDF conversion, full-text search across a searchable PDF, redaction, and signing all come from the same engine and the same license, so adding a capability is a configuration change rather than a new vendor integration. That consolidation also keeps document content within your own infrastructure, rather than routing it through several third parties to complete a workflow. Migrating From a Library To an SDK: What It Actually Costs The concern teams raise most often is the cost of moving later, after the app has grown around the library's limitations. That cost is real, but so is the cost of staying on a basic library past the point it fits: slower rendering, an inconsistent user experience, and engineering time spent on patching instead of product work. Let’s look at a real example: Blue Voice built its first version on an open-source React PDF viewer. As the product scaled across police departments, maintaining that PDF functionality started consuming engineering time that the team wanted to spend on its core product instead. After moving to Apryse, according to CTO and co-founder Amit Patankar, "the product felt more polished, our users immediately noticed the difference, and our team could focus on building Blue Voice instead of maintaining a PDF viewer." For a closer look at what the maintenance side of that decision costs over time, read the article The Hidden Costs of Choosing the Wrong PDF Library. If you are ready to compare specific SDKs against your requirements, the Document SDK Buying Guide walks through how to evaluate and buy one. What’s Next for Your Team? Whether you use an open-source document processing library today, or are still planning your project, you can try all Apryse capabilities in a test environment instantly (without needing to talk to sales) by starting your trial. When it’s time to use Apryse in production, contact sales to get licensing that fits your needs. FAQ What is the best PDF library for an enterprise app? The best PDF library for an enterprise app is usually not a basic library at all. Enterprise apps typically need viewing, editing, redaction, and security together, which points toward a full document SDK, like Apryse, rather than a single-purpose library. When do I need a document SDK instead of a basic library? You need a document SDK once your app requires more than one document capability, needs those capabilities to share state, or needs document content to stay inside your own environment for compliance reasons. Apryse offers viewing, editing, redaction, and security together, along with premise-based deployment options. What is the difference between a PDF SDK and a PDF API? A PDF SDK is embedded directly in your application and runs in your own environment. A PDF API is typically a cloud endpoint you call for a single task, which means documents leave your environment, and multiple tasks mean multiple vendor integrations. Is an open-source PDF library good enough for production? An open-source PDF library works well for simple, single-purpose tasks like viewing or merging. It becomes harder to justify once you need broader features, vendor accountability for security patches, or support beyond a community forum. How much does it cost to migrate from a library to an SDK later? The migration cost depends on how much the application has grown around the library's limitations. Teams that wait until rendering issues, maintenance load, or compliance gaps are already affecting users typically face a larger migration than teams that move earlier.