DZone Spotlight

Business documents rarely keep their most useful information in convenient database records or JSON objects. Invoices hold line items in tables, financial reports organize figures by period, inspection forms group findings by category, and emailed statements often arrive as attachments or message files. Before our applications can validate, compare, search, or store that information, they have to somehow recover the relationships between rows, columns, headers, and values. At first glance, this may look like a routine text-extraction problem. If we can read the words on the page, surely we can rebuild the table... right? In practice, unfortunately, recognizing the text is only the first layer. We also need to determine which values belong to the same row, where columns begin and end, which headers describe which cells, and whether the document contains multiple distinct tables. Again, tables represent relationships, and if those relationships are mishandled, the data becomes useless. In this article, we’ll look at why table extraction becomes difficult across different document formats. We’ll then implement an API-based extraction workflow in C# and process the structured table data it returns. Why Table Extraction Is More Than OCR Optical character recognition (OCR) has been around forever, and by today's technological standards, it answers a relatively narrow question: which characters appear in an image, and where are they located? Table extraction has to answer a more contextual question: how are those characters related? Consider a scanned invoice with the headers Description, Quantity, Unit Price, and Total. An OCR engine might correctly identify all four headers and every value beneath them, but a stream of recognized text isn’t enough for downstream automation. We still need to establish that 3, $12.00, and $36.00 belong to the same line item, and that $36.00 represents the total rather than the unit price. Visible borders can certainly help, but we can’t count on them; many modern tables use whitespace, shading, or alignment instead of grid lines. Cells may span multiple apparent columns, descriptions may run into character constraints and wrap onto additional lines, and a table may continue onto the next page with repeated or missing headers. Poor document image quality has also been the bane of OCR solutions for decades; it introduces another unwelcome layer of ambiguity. A recognition error can change both an individual data value and the entire interpretation of its surrounding structure. All of this is to say that a document can contain perfectly readable text while still producing a poor table. Reliable table extraction requires both character recognition and layout analysis. Different Formats Hide Tables in Different Ways The word "document" itself covers several very different internal structures. A DOCX file, for example, is a package of XML parts that can contain explicit table elements. An XLSX workbook stores cells, values, formulas, and worksheet relationships in another Office Open XML structure. PPTX files can combine true table objects with independently positioned text boxes that only look like tables to a human reader. PDF files are significantly more complicated. A PDF may describe text by drawing individual characters at specific page coordinates on the page without preserving any semantic concept of a row, column, or table for extraction services to key in on. Two values that appear next to each other on the page may or may not be related within the underlying PDF structure; context is required to know for sure. Images push us fully into visual interpretation. For example, JPG, PNG, and WEBP inputs don’t contain native text or table objects, so OCR and layout analysis are required for automated data extraction. Email containers such as EML and MSG introduce another variation. We might not think about these file types quite as frequently as the others mentioned in this section, but they introduce an interesting challenge worth considering in the pursuit of solving this problem for a wide range of inputs. These formats allow for a useful table to appear in the message body, a rendered representation of that body, or an attached document. As a result, a broadly focused table-extraction system can’t simply run the same parser against every extension. It first needs to identify and decode the format, obtain a useful page or layout representation, recognize text when necessary, and map the discovered structures into a consistent table model. Once every input produces the same hierarchy (tables containing rows, rows containing cells, and cells containing headers and values), the rest of our application doesn't need format-specific extraction logic anymore. Building the Workflow With Open-Source .NET Libraries The .NET ecosystem gives us several useful open-source building blocks for structured data extraction, but the right library depends heavily on the source document. For Office documents, the Open XML SDK is your first stop: it can inspect the native structures inside DOCX, XLSX, and PPTX files directly. Format-specific libraries such as ClosedXML can provide a more approachable layer for working with Excel worksheets. These options are a strong fit when our tables exist as real Office table or cell structures, which is fairly often. PDF documents often require a different path. A library such as PdfPig (more than 28 million downloads on GitHub) can extract text and positional information from text-based PDFs, but our application may still need custom logic to group those positioned words into rows and columns. If the PDF contains scanned pages, we first need to render those pages into images and send them through an OCR engine such as Tesseract (another widely used & loved package). If reading from email files is a must, a bit more routing work is required. MimeKit can parse MIME-based messages such as EML files, while MSG files may require a separate Outlook message parser. After parsing the message, we still need to inspect the body and each relevant attachment independently. Each of these libraries can, of course, make sense within its own lane. If our application receives one predictable document type with a stable layout, an open-source implementation may give us all the control we need. The complexity rears its head when we want one production workflow to accept PDFs, Office files, email containers, and images all at once. In that case, we have to detect formats, route documents to the correct parser, render pages when native extraction fails, decide when OCR is necessary, and reconcile several different output structures. That's a lot. We also own the quality heuristics for that workflow. That includes the borderless-table detection challenge in addition to merged cells, rotated pages, repeated headers, image preprocessing, wrapped text, and validation thresholds. In other words, multi-format table extraction is better understood as a document-processing system than a single library call. Using a Normalized Table-Extraction API If maintaining separate extraction paths for each format feels like too much, we can send the input document to a dedicated table-extraction service that performs the format handling, recognition, and table analysis through a consistent API. We'll walk through one example that uses AI to consistently identify table structures in DOCX, PDF, XLSX, PPTX, EML, MSG, JPG, PNG, and WEBP input. The response JSON organizes the extracted content into tables, rows, and cells. We’ll access the endpoint through its generated .NET Core SDK. To begin, we’ll install version 1.0.0: C# dotnet add package Cloudmersive.APIClient.NETCore.DocumentAI --version 1.0.0 Once the package is installed, we can import the API, client, and model namespaces required for the request: C# using System; using System.Diagnostics; using Cloudmersive.APIClient.NETCore.DocumentAI.Api; using Cloudmersive.APIClient.NETCore.DocumentAI.Client; using Cloudmersive.APIClient.NETCore.DocumentAI.Model; The snippets provided below mirror the supplied SDK code directly; as code examples, they assume we’ll adapt placeholders and surrounding application details as needed. Configuring the API Client First, we’ll add our API key under the Apikey authorization name in the default configuration: C# Configuration.Default.AddApiKey("Apikey", "YOUR_API_KEY"); Creating the Extraction Client Next, we’ll create a new ExtractApi instance: C# var apiInstance = new ExtractApi(); Loading the Input Document We’ll assign a value to the optional recognition-mode parameter: C# var recognitionMode = "Advanced"; Advanced is the default recognition mode and provides the highest accuracy with slower processing, while Normal provides faster processing with lower accuracy for low-quality images. Next, we’ll open our input document as a FileStream: C# var inputFile = new System.IO.FileStream("C:\\temp\\inputfile", System.IO.FileMode.Open); Executing the Table-Extraction Request With recognitionMode and inputFile ready, we’ll pass them into ExtractTables and write the returned ExtractTablesResponse object to the debug output: C# try { // Extract Tables of Data from a Document using AI ExtractTablesResponse result = apiInstance.ExtractTables(recognitionMode, inputFile); Debug.WriteLine(result); } catch (Exception e) { Debug.Print("Exception when calling ExtractApi.ExtractTables: " + e.Message ); } Understanding the Response Structure A successful response should follow this general structure: JSON { "Successful": true, "TableResults": [ { "Title": "Invoice Line Items", "Rows": [ { "Cells": [ { "CellHeader": "Description", "CellValue": "Replacement filter" }, { "CellHeader": "Quantity", "CellValue": "3" }, { "CellHeader": "Unit Price", "CellValue": "$12.00" }, { "CellHeader": "Total", "CellValue": "$36.00" } ] } ] } ] } Successful tells us whether the extraction operation completed. TableResults is a collection because one document may contain multiple distinct tables; the endpoint naturally distinguishes between each and returns their results separately. Ultimately, this response is a pretty straightforward JSON mapping. Every table can include a Title followed by Rows. Every row contains a collection of Cells, and each cell provides an inferred CellHeader and extracted CellValue. Note that a structurally valid response does not guarantee every property will contain a value. A document may contain a table with no visible title, for example, so our workflow shouldn’t rely on Title as a required identifier. We should also definitely expect real documents to contain blank cells, inconsistent headers, and values that require additional parsing before validation or storage. Reading the Returned Tables in C# C# if (result.Successful == true && result.TableResults != null) { foreach (var table in result.TableResults) { foreach (var row in table.Rows) { foreach (var cell in row.Cells) { Console.WriteLine( $"{cell.CellHeader}: {cell.CellValue}" ); } } } } The simple nested loops I've included here get each header and value while preserving the table structure, leaving us free to map rows into dictionaries, database entities, CSV records, or custom models such as InvoiceLineItem. Note that we should probably avoid aggressive type casting; identifiers may need leading zeroes preserved, while currency and date values may require locale-aware parsing. Extraction may structure the data, but schema validation remains our application’s responsibility. The Full Implementation Here's a fully assembled example implementation including everything we just outlined above: C# using System; using System.Diagnostics; using Cloudmersive.APIClient.NETCore.DocumentAI.Api; using Cloudmersive.APIClient.NETCore.DocumentAI.Client; using Cloudmersive.APIClient.NETCore.DocumentAI.Model; namespace Example { public class ExtractTablesExample { public static void Main() { Configuration.Default.AddApiKey( "Apikey", "YOUR_API_KEY" ); var apiInstance = new ExtractApi(); var recognitionMode = "Advanced"; using ( var inputFile = new System.IO.FileStream( "C:\\temp\\inputfile", System.IO.FileMode.Open ) ) { try { ExtractTablesResponse result = apiInstance.ExtractTables( recognitionMode, inputFile ); Debug.WriteLine(result); if (result != null && result.Successful == true && result.TableResults != null) { foreach (var table in result.TableResults) { if (table == null || table.Rows == null) { continue; } foreach (var row in table.Rows) { if (row == null || row.Cells == null) { continue; } foreach (var cell in row.Cells) { if (cell == null) { continue; } Console.WriteLine( $"{cell.CellHeader}: " + $"{cell.CellValue}" ); } } } } } catch (Exception e) { Debug.Print( "Exception when calling " + "ExtractApi.ExtractTables: " + e.Message ); } } } } } Adding Production Guardrails Whether we use the API-based approach demonstrated above or assemble an open-source extraction system, a production pipeline still needs some guardrails around it. The exact implementation will differ, but the underlying goals are mostly the same: we want to control things like resource use & preserve traceability, and very importantly, we want to prevent questionable extraction results from quietly entering downstream systems. First, we should validate each file and enforce practical document and page limits before processing begins. With the API approach, page counts directly affect consumption. In an open-source system, those same long documents can consume substantial memory, CPU time, OCR capacity, and worker availability. Both implementations benefit from clear limits and a plan for handling unusually large documents. We should also try to retain enough context to audit each result. For the API workflow, that record might include the source document identifier, recognition mode, response status, table index, etc. An open-source workflow might additionally record which parser, OCR engine, preprocessing steps, model version, and fallback path were used. These details make extraction problems much easier to reproduce and diagnose later. Most importantly, we need to define what successful means at the application level. An API response with Successful set to true indicates that the extraction operation completed. Likewise, an open-source parser returning rows without throwing an exception only tells us that its processing path completed. Neither outcome proves that the expected table was found or that every extracted value is correct. If the extracted data affects payments, compliance decisions, inventory, or customer records, human review remains sensible for incomplete or internally inconsistent results. Automation should reduce the amount of manual work required, not remove the opportunity to catch a result that doesn’t make sense. Conclusion In this article, we separated table extraction from plain text recognition and saw why supporting extraction from PDFs, Office documents, email containers, and images can require several different processing paths. Open-source .NET libraries give us plenty of capable building blocks when our formats and layouts are controlled. A broad intake workflow, however, also needs document routing, OCR, layout analysis, output normalization, and ongoing quality logic, all of which can be burdensome to implement in a production environment. We then installed a Document AI .NET SDK and took a look at structuring a request to handle table extraction automatically. With sufficient validation incorporated around those results, the same pattern we just demonstrated can support invoice processing, reporting, database imports, reconciliation workflows, and other systems that need structured document data rather than another block of extracted text.

By Brian O'Neill CORE

Where the Data Actually Lives Every enterprise I have worked with hits the same wall. Mountains of data. Warehouses, ticketing systems, PDFs, old email archives. Most of that data is not ready for AI to use. Leadership wants a chatbot that can answer questions about policy and product specs. But no one knows where the data lives or how to get it ready. A bigger model will not fix that. It gets solved by data engineering. The Pattern That Shows Up Again and Again A chatbot wired straight to a foundation model, no retrieval layer at all. It answers from memory and gets the specifics wrong.Documents sitting in five different systems, none of them governed the same way.Embeddings computed once, at launch, and never refreshed again.A RAG pipeline built without anyone checking who has write access to the source documents. The gap is never the model. It is what happens before the model ever sees the question. This is not a problem reserved for large enterprises with dedicated ML teams. A three-person startup wiring a chatbot to their support docs runs into the exact same wall. The model does not care how big your organization is. If your data is messy and ungoverned, your answers will be too. What Snowflake Cortex Actually Does Snowflake Cortex is a set of AI functions built directly into the Snowflake platform. Embeddings, summarization, completion. Call them right inside SQL or through the Python connector. The data never has to leave. That matters more than it sounds. Every time data crosses a boundary to reach an external AI vendor, that is one more place it can leak, one more thing to govern, one more contract to review. Cortex removes the boundary. Teams use it two ways in practice. Straight from SQL, for lightweight transformations. Through Snowpark Python, when the logic needs to sit inside a broader application. Either way, the embedding or the completion happens inside Snowflake’s governed environment. Python cortex_embed.py import snowflake.connector conn = snowflake.connector.connect( user='YOUR_USER', password='YOUR_PASSWORD', account='YOUR_ACCOUNT', warehouse='COMPUTE_WH', database='ENTERPRISE_DB', schema='AI_SCHEMA' ) cur = conn.cursor() cur.execute( "SELECT SNOWFLAKE.CORTEX.EMBED_TEXT_768('e5-base-v2', 'Quarterly compliance policy summary') AS embedding" ) result = cur.fetchone() print(len(result[0])) # vector length Connect. Call a Cortex function inside a SQL statement. Pull the result back. That pattern repeats everywhere in this pipeline. What RAG Actually Is Retrieval-augmented generation is not complicated once you strip the marketing off it. Retrieve relevant text from your own data. Hand it to the model. Ask it to answer using only that context, not memory. That single design decision is what cuts hallucination and makes citations possible. The workflow, broken into its actual steps: a question comes in, gets embedded into a vector, the system searches for the closest matching chunks, and those chunks get passed to the model with the original question. The model answers from what it was given. Nothing more. Python rag_query.py def answer_question(question, cursor, top_k=3): cursor.execute( "SELECT SNOWFLAKE.CORTEX.EMBED_TEXT_768('e5-base-v2', %s)", (question,) ) query_vector = cursor.fetchone()[0] chunks = search_similar_chunks(cursor, query_vector, top_k) context = '\n\n'.join(chunks) return generate_answer(cursor, question, context) Everything else in this article is just filling in the details behind three steps. Embed. Retrieve. Generate. The Full Architecture Figure 1. Enterprise AI Architecture Using Snowflake Cortex and Retrieval-Augmented Generation Data flows in from enterprise sources. It gets cleaned and staged in Snowflake. Cortex generates embeddings. A vector store supports fast retrieval. A retrieval layer sits in between. The enterprise user sees none of this. They just ask a question. Most teams underestimate the plumbing. Embeddings and generation are easy. Reliable ingestion is hard. So is a fresh index. So is retrieval quality as the document set grows. That is where the real work is. Building the Pipeline This is standard data engineering. Read raw files. Validate. Stage. Merge into production using a key that blocks duplicate loads. Nothing exotic. Get it wrong here. Every answer downstream gets worse. It happens quietly. No error tells you why. Python load_documents.py import pandas as pd def load_and_clean(csv_path): df = pd.read_csv(csv_path) df = df.dropna(subset=['document_text']).drop_duplicates() df['document_text'] = df['document_text'].str.strip().str.replace(r'\s+', ' ', regex=True) return df def load_documents_to_snowflake(df, conn, table='DOCS_STAGING'): cur = conn.cursor() for _, row in df.iterrows(): cur.execute( f"INSERT INTO {table} (doc_id, source, document_text) VALUES (%s, %s, %s)", (row['doc_id'], row['source'], row['document_text']) ) conn.commit() cur.close() This is deliberately small. In production, batch inserts. Wrap them in a transaction. Add a merge step. This stops duplicate rows on reruns. What matters most: by this point, data is clean. It is already validated. Chunking Is Where Most Pipelines Quietly Fail Chunking matters more than expected. Too large dilutes relevance. Too small loses context. Most teams pick 300 to 800 tokens. They add some overlap too. Even that gets argued over. Python chunk_and_embed.py def chunk_text(text, chunk_size=500, overlap=50): words = text.split() chunks = [] start = 0 while start < len(words): end = start + chunk_size chunks.append(' '.join(words[start:end])) start = end - overlap return chunks def embed_and_store(cursor, doc_id, text): for i, chunk in enumerate(chunk_text(text)): cursor.execute( "INSERT INTO DOC_EMBEDDINGS (doc_id, chunk_id, chunk_text, embedding) " "SELECT %s, %s, %s, SNOWFLAKE.CORTEX.EMBED_TEXT_768('e5-base-v2', %s)", (doc_id, i, chunk, chunk) ) A sliding window chunks the document. Cortex computes the vector inside Snowflake, in the same insert. No separate embedding service to babysit. Change the chunk size later, and you are not tweaking a config. You are re-embedding everything. Querying With Vector Search With embeddings stored, retrieval becomes a nearest-neighbor search. Snowflake runs this natively. Plain SQL. No separate vector database to stand up. No extra system to operate. Python vector_search.py def search_similar_chunks(cursor, query_vector, top_k=3): cursor.execute( "SELECT chunk_text FROM DOC_EMBEDDINGS " "ORDER BY VECTOR_COSINE_SIMILARITY(embedding, %s) DESC " "LIMIT %s", (query_vector, top_k) ) return [row[0] for row in cursor.fetchall()] def generate_answer(cursor, question, context): prompt = f"Answer using only this context:\n{context}\n\nQuestion: {question}" cursor.execute( "SELECT SNOWFLAKE.CORTEX.COMPLETE('llama3-70b', %s)", (prompt,) ) return cursor.fetchone()[0] Rank by cosine similarity. Wrap the result and the question into a prompt. Call Cortex’s completion function. That is the whole RAG loop, closed. Production Is a Different Problem Than the Notebook Getting this working in a notebook takes an afternoon. Running it reliably, with monitoring, access control, and predictable costs, takes real engineering discipline. Figure 2. End-to-End Enterprise Data Engineering Pipeline for Snowflake Cortex and RAG What Actually Pays Off Cache embeddings for repeated queries. Do not recompute what you already have.Tag every chunk with its access permissions. Retrieval that ignores row-level security is a data leak waiting to happen.Log every retrieval and every generation call. You cannot debug a bad answer you did not record.Version your chunking strategy. Changing it later means re-embedding everything, whether you planned for it or not. Python log_wrapper.py import time def answer_with_logging(question, cursor, logger): start = time.time() answer = answer_question(question, cursor) duration = time.time() - start logger.info(f"query='{question}' duration={duration:.2f}s") return answer This kind of logging gets skipped early. It becomes essential the moment a real user reports a confusing answer, and you need a trail to investigate. Where This Breaks Every team hits the same failure modes the first time they run RAG against messy enterprise data. The Usual Suspects Stale index. The vector store stopped reflecting reality weeks ago, and nobody noticed.Chunk drift. Retrieval quality degrades because the chunk size was wrong from the start.Prompt injection through retrieved documents. Anyone with write access to your corpus has indirect access to your model.Latency that traces back to an unindexed similarity search or a context window stuffed too full. None of these is exotic. All of them are common. Stale indexes get fixed with a scheduled incremental refresh, not a full re-embed. Chunk drift gets fixed by testing a few chunk sizes against a fixed set of real questions. Latency gets fixed with tuning, not a redesign. Prompt injection is the one people fix last, usually after it has already happened once. Python sanitize_chunk.py def sanitize_chunk(chunk_text): forbidden_phrases = ['ignore previous instructions', 'disregard the above'] lowered = chunk_text.lower() for phrase in forbidden_phrases: if phrase in lowered: chunk_text = chunk_text.replace(phrase, '[removed]') return chunk_text This is a basic example. It is not a complete defense. Retrieved content is untrusted input. It works the same as user input in a web app. Treat it that way from the first line of code. This Does Not End at Launch Enterprise AI built on Snowflake Cortex and RAG is not about clever prompting. It is disciplined data engineering wearing a chatbot’s face. Clean ingestion. Sensible chunking. Governed embeddings. A retrieval layer that respects access control. That does more for answer quality than swapping in a bigger model ever will. Cortex removes the operational overhead of moving data somewhere else to get an embedding. RAG keeps the model grounded in what your organization actually knows, instead of what a foundation model memorized during training. If you are starting this kind of project, resist the urge to jump straight to the LLM integration. Get the pipeline right first. Measure retrieval quality against real questions from real users. Treat the whole thing as production software from day one, monitoring, logging, and security built in rather than bolted on later. Start with the data. The model is the easy part.

Every B2B SaaS product team knows this moment. You're trying to close a deal, and the prospect says, "We just need you to sync with our CRM. And our HRIS. Oh, and these three other tools. You can do that, right?" Your roadmap takes a hit, and your engineering backlog doubles overnight. And eventually someone says, "What about a unified API?" It sounds like the answer — one normalized schema, one auth model, and one point of connection for a dozen or more apps in a vertical. You buy it, hook it up, and ship the integrations before the quarter ends, the integration checkbox gets checked, and you move on. For a while, it works. But there's a problem most teams don't see until they start moving upmarket. For many SaaS teams, a unified API is the right first move. It's rarely the right last one. Unified APIs Exist for a Reason, and They're Good at What They Do Most apps in a category share the same data objects. CRMs have contacts, accounts, opportunities, and activities. HRIS platforms store employee, department, and compensation data. Ticketing systems track tickets, users, and statuses. A unified API vendor abstracts the data models for an app category into a common schema so that, rather than learning a dozen APIs, your devs learn one. For startups under pressure to ship quickly, that abstraction is valuable. You can launch integrations faster, reduce engineering work, and simplify auth across the board. If your customers need common objects and standard workflows, a unified API can meaningfully accelerate your roadmap. That's all positive. The negative shows up down the road. The Lowest Common Denominator Problem A normalized data model (which is what a unified API is based on) is, by definition, a reduced or simplified data model. To present a single schema across N apps, a unified API must identify the fields they have in common. The result is a model built on the smallest shared dataset. Anything that's app-specific is abstracted away, and anything proprietary is dropped. Unified APIs work until your customers stop being generic. Enterprise customers have Salesforce custom objects built for their unique processes. They have Workday compensation structures that don't fit a normalized HRIS schema. They have vertical-specific fields that are critical to their business processes. And, it's increasingly common for them to be running systems that the unified API vendor has never heard of. The moment a prospect asks you to sync a custom object, access a proprietary field, or connect to an app outside your unified API vendor's supported list, the abstraction layer is no longer sufficient. You either tell your prospect "No" or you build a custom, one-off integration anyway, which largely defeats the point of a unified API. At first, these seem like edge cases. Then you realize enterprise customers are the edge cases. And that they are bringing the highest-value deals in your pipeline. The "Zero Maintenance" Promise Doesn't Hold Up The biggest marketing claim of a unified API is that upstream API changes are no longer your problem: "They update their API, we handle the change." In reality, you're trading one type of maintenance for another. With native APIs, you worry about endpoint deprecations, auth updates, and rate limits. With a unified API, you worry about data lost in translation or debugging through an abstraction layer. When that happens for an enterprise customer, you can't just look at the target system's logs. You have to work through the unified API provider's black box. If the root cause is a nuance in how they handle a specific app's rate-limiting rules, your engineering team is now waiting on someone else's support ticket queue. The maintenance didn't go away. It just moved down the street. Complexity Comes Later The full cost of a unified API strategy rarely appears during implementation. Instead, it waits until things have settled into a steady rhythm and then shows up as operational complexity. Dual integration architectures – Once you need custom integrations alongside your unified API (and you will), your team will maintain two separate integration layers with different auth flows, error handling, retry logic, and monitoring. Every integration request now needs to go through a decision tree to determine which of these patterns (or perhaps even a new one) you should use for development.Data model constraints – Your app connects with the unified API's schema rather than to the underlying apps. When customers ask for fields the schema doesn't expose, your team builds manual workarounds, relocating rather than reducing the complexity.Vendor roadmap dependency – If your unified API provider doesn't support a specific endpoint, a webhook behavior, an advanced API feature, or a vertical SaaS platform your customer uses, you wait (or you build around it). Either way, the original value proposition isn't holding up to the rigors of reality.Escalation cost – Enterprise prospects bring technical evaluators. When those evaluators discover that your integration can't provide the specific data they depend on, the deal may end right there. That's not good for your bottom line. What the Workaround Trap Looks Like Most teams respond the same way when they hit these limits. They start building custom integrations in addition to those handled through the unified API. What began as a simplification strategy is starting to look like this: a unified API for common integrations, direct API connections for exceptions, custom middleware for unsupported workflows, separate auth handling, multiple sync models, and one-off transformation logic wherever it's needed. In short, that neatly ordered integration layer is no longer. The abstraction created to reduce maintenance has, in fact, increased it. Teams find they're burning an appreciable portion of their integration budget maintaining low-value integrations and working around the things their unified API vendor can't support. That's engineering time that isn't being devoted to your core product. Vertical SaaS Is the Forcing Function The continued fragmentation of B2B software makes this worse every year. Beyond mainstream CRMs and HR platforms, companies increasingly rely on industry-specific applications: systems narrowly designed and built for healthcare, manufacturing, financial services, and a score of other verticals. These systems rarely conform to standardized schemas. Many of them don't appear in any unified API vendor's list of supported apps. A unified API might help you connect to ten generic CRMs. It won't help much when your largest prospect is running Epic, Procore, or a heavily customized NetSuite instance. Those are the integrations that determine whether enterprise deals close. What Happens at Scale Unified APIs are usually evaluated based on how fast they help teams launch. However, the more important question is: "What happens when integration requirements grow more complex?" Because they always do. Every single time. As SaaS products mature, integration requests shift from "Can you connect to this category?" to "Can you support this exact workflow?" That move exposes the architectural limits of a unified API. And the teams that hit the limit mid-deal (or mid-contract) feel the immediate pain. Why Embedded iPaaS Is the Durable Foundation This is where embedded iPaaS platforms fundamentally differ from unified APIs: they aren't constrained to a single simplified schema. An embedded iPaaS gives your team a flexible integration foundation that handles both ends of the spectrum: the common apps that benefit from productized integrations, and the complex, vertical-specific, niche apps that don't fit any standardized model. Some of your customers need a basic CRM sync. Others need multi-flow orchestration, conditional business logic, extensive data mapping, and more. A rigid abstraction model breaks under those requirements. An embedded iPaaS doesn't. This Isn't "Unified APIs vs. Embedded iPaaS" Unified APIs still have value. For early-stage validation or straightforward category integrations at scale, they can accelerate time-to-market. Many mature teams use them alongside a more flexible platform for the scenarios where standardization works. But for most B2B SaaS teams, they are a way-station, not the destination. The mistake teams make is assuming the abstraction can scale indefinitely as customer complexity increases. But that's not true. It can't, and it doesn't. The bigger and more complex your customers get, the more a lowest-common-denominator approach becomes an obstacle instead of a shortcut.

Agent framework debates are mostly vibes. One engineer swears LangGraph is faster, another prefers the OpenAI Agents SDK, someone wants Google ADK because it feels future-proof. The team picks one, wires the workflow into its SDK, and the choice is welded in. Changing frameworks later means tearing out the wiring for one SDK and rebuilding the workflow on another, an expensive rewrite few teams take on. This tutorial makes that decision reversible and then settles it with data. You put the agent graph in LaunchDarkly and run four frameworks (LangGraph, Strands, OpenAI Agents SDK, and Google ADK) over the same topology, with the model pinned so the framework is the only variable. A LaunchDarkly experiment ranks them on graph latency and token use, with an LLM judge guarding quality. The results table tells you which framework runs your graph fastest without degrading it. This tutorial is the sequel to Compare AI orchestrators, which ran the same workflow across frameworks but kept the topology in each framework’s code. Here, the topology, routing, models, prompts, tools, and judge all live in LaunchDarkly, and each framework supplies only two functions. The experiment results do more than set a benchmark. The flag that splits experiment traffic also routes production. When one framework wins, you don’t rewrite the app; you change the flag to serve the winner. In a single loop, LaunchDarkly does three jobs: the graph definition, the experiment split, and the runtime control that ships the winner. The workload is a research-gap analysis over a set of arXiv papers. Two readers, approach-analyzer and contradiction-detector, read the same papers in parallel and fan in to gap-synthesizer, which writes the report. Prerequisites A LaunchDarkly account with AgentControl access, and your environment’s SDK keyPython 3.11+ and uvAn ANTHROPIC_API_KEY for the pinned model. OPENAI_API_KEY and GOOGLE_API_KEY are only needed if you run the optional native-model bake-off in Step 9The companion repo: ai-orchestrators on branch tutorial/graph-experiments The Experiment Design The comparison is controlled: same graph, same model, same papers, same judge, with the framework as the only variable. Mechanically, it runs in four stages: Bootstrap. manifest.yaml creates the node configs, graph, orchestrator flag, and judge in LaunchDarkly.Route. On each request, the app evaluates the orchestrator flag to pick a framework: langgraph, strands, openai-agents, or google-adk.Run. The dispatcher runs the shared graph as a directed acyclic graph (DAG). The two readers run concurrently and fan in to the synthesizer.Measure. Each run records how long the graph took, how many tokens it used, and whether the report passed the quality judge. The shape looks like this: ┌──▶ approach-analyzer ───────┐ intake (papers) ─────┤ ├──▶ gap-synthesizer ──▶ report └──▶ contradiction-detector ──┘ Step 1: Create the Graph, Flag, and Judge Everything starts from one file, config/graph_experiment_manifest.yaml. It declares the fetch_paper tool, four node configs (intake plus the three agents, pinned to claude-sonnet-4-5), the graph, the orchestrator flag, and the judge. First, clone the companion repo and install its dependencies with uv: Shell git clone https://github.com/launchdarkly-labs/ai-orchestrators cd ai-orchestrators git checkout tutorial/graph-experiments uv sync Next, set up a LaunchDarkly project. The bootstrap doesn’t create one, so create it with the LaunchDarkly MCP server, the projects agent skill, or the UI. Name it graph-experiments to match the value in .env.example, so the defaults work without edits. When it exists, copy its key into LD_PROJECT_KEY and its production environment SDK key into LD_SDK_KEY in .env. The runners and experiment harness use that SDK key to evaluate the flag and graph. The bootstrap also reads LD_API_KEY from .env to create the resources. Copy the example file to create your .env: Shell cp .env.example .env # then set LD_PROJECT_KEY, LD_SDK_KEY, and LD_API_KEY in .env With the keys in place, run the bootstrap: Shell uv run python scripts/launchdarkly/bootstrap.py config/graph_experiment_manifest.yaml This creates all four node configs, the research-gap-graph, the orchestrator flag (created off), and the gap-quality-judge attached to the gap-synthesizer node (its synthesizer-claude variation, set to 100% sampling). The judge scores the final report against the source papers, so it can verify grounding and citations. A judge can only check based on the information it has, so we give it the papers, not only an upstream agent’s analysis. When the graph ships, it is incomplete by design. The bootstrap creates the contradiction-detector config but wires only intake to approach-analyzer to gap-synthesizer, leaving the detector out. You’ll add it in Step 5 to complete the parallel fan-in. When it finishes, the bootstrap prints a link to your new agent graph. Open it and review the topology before moving on. The graph shows a straight line from intake to approach-analyzer to gap-synthesizer, with contradiction-detector created but not yet wired in. Step 2: The Dispatcher Runs the Graph The dispatcher is the heart of the project, and it’s the same code for every framework. It reads the graph as a DAG, runs the entry nodes concurrently, hands every node the papers as ground truth, and connects the readers at the fan-in node. The only framework-specific pieces are build_agent and invoke, which are passed in as arguments. The whole process is about 100 lines, built on the agent graph traversal methods in the SDK. The complete dispatcher.py is in the companion repo. The dispatcher carries the design in four parts: it builds the execution plan from the graph’s edges, composes each node’s input, runs every ready node concurrently each round, and records the graph’s metrics once per run. First, the dispatcher builds the execution plan from the graph’s edges, so the topology you draw in LaunchDarkly runs: Python for key, node in nodes.items(): for edge in node.get_edges(): target = edge.target_config if target in nodes: succ[key].append(target) preds[target].append(key) Next, every node receives the source papers and any upstream analyses, so each agent and the judge work directly from the source material rather than a summary handed down a chain: Python def compose_input(user_input, predecessor_outputs): parts = [f"=== SOURCE PAPERS ===\n{user_input}"] for key, out in predecessor_outputs: if out and out.strip(): parts.append(f"=== {key} ===\n{out}") return "\n\n".join(parts) Then each round runs every node whose predecessors have finished, concurrently, so the two readers fan out and fan in with no special casing: Python ready = [k for k in pending if all(p in done for p in preds[k])] results = await asyncio.gather(*(run_node(k) for k in ready)) Finally, the dispatcher records the graph’s metrics on each run, including the end-to-end latency the experiment ranks on: Python graph_tracker.track_duration(int((time.monotonic() - start) * 1000)) graph_tracker.track_total_tokens(TokenUsage(input=totals["in"], output=totals["out"], total=totals["in"] + totals["out"])) graph_tracker.track_path(path) graph_tracker.track_invocation_success() The dispatcher reads the topology at runtime, so reshaping the workflow in the UI, adding a node, or redrawing an edge takes effect on the next request with no code change. You’ll do exactly that in Step 5. Step 3: Each Framework Is a Thin Adapter Each framework implements build_agent(node_key, config, instructions) and async invoke(agent, input_text, tracker). Everything dynamic still comes from the LaunchDarkly node config: the model, the attached tools, and the instructions. LangGraph has a LaunchDarkly companion package, so its runner is only a few lines. The companion handles model creation, tool binding, and token tracking, so the adapter holds no framework plumbing of its own: Python def build_agent(node_key, config, instructions): llm = create_langchain_model(config) tools = build_tools(config, TOOL_REGISTRY) # binds only this node's attached tools return create_react_agent(llm, tools, prompt=instructions) async def invoke(agent, input_text, tracker): result = await tracker.track_metrics_of_async( lambda res: LDAIMetrics(success=True, tokens=sum_token_usage_from_messages(res.get("messages", []))), lambda: agent.ainvoke({"messages": [{"role": "user", "content": input_text}]}), ) messages = result.get("messages", []) for message in messages: for name in get_tool_calls_from_response(message): tracker.track_tool_call(name) text = _content_to_text(messages[-1].content) if messages else "" return text, sum_token_usage_from_messages(messages) Strands has no companion package, so its runner builds the model with a small provider-aware factory and binds tools with Strands’ native @tool. The contract is identical: Python def build_agent(node_key, config, instructions): return Agent( name=node_key, model=_create_strands_model(config), system_prompt=instructions or "Process the input and respond.", tools=_bind_tools(config), callback_handler=None, ) OpenAI Agents and Google ADK round out the four. For the comparison to stay fair, all four have to run the same model, but these two SDKs default to their own vendors’ models. LiteLLM, a thin adapter, lets them call any provider, so we point both at the pinned claude-sonnet-4-5 and keep the model identical across all four orchestrators. No OpenAI or Google servers are involved. Instead, LiteLLM translates the request format in-process, and the call goes straight to Anthropic with your key. Google ADK is fully companion-free, and OpenAI Agents uses the ldai_openai companion for token and tool-call telemetry even though it builds the model through LiteLLM. This experiment pins one model across all four frameworks, so every framework here runs Claude. Pointing each framework at its own vendor’s default model instead is a separate, optional exercise, the native-model bake-off in Step 9. The tool callables live in TOOL_REGISTRY, a plain {name: callable} map that each framework binds its own way. Step 4: Smoke Test the Graph Before you run any experiment, confirm the bootstrapped graph runs end to end. First, run one framework: Python uv run python orchestrators/verify_run.py langgraph It prints the path it took and the first part of the report. On the graph as it shipped, the path is intake -> approach-analyzer -> gap-synthesizer: intake runs its short pass, approach-analyzer reads the papers, and gap-synthesizer writes the report. There’s no contradiction-detector yet, and no error. The metrics land in the AgentControl UI under the graph you created. Step 5: Add the Parallel Fan-In In the UI Here’s the payoff of keeping the topology in LaunchDarkly: you finish building the workflow in the UI, with no redeploy, and the running app picks up the new shape on its next request. The contradiction-detector config already exists, with its fetch_paper tool attached. You wire it into the graph to add the second reader and form the parallel fan-in. To complete the graph: Click Agents in the LaunchDarkly sidebar.Click Agent graphs.Select research-gap-graph.Add the contradiction-detector node.Draw an edge from intake to contradiction-detector, then another from contradiction-detector to gap-synthesizer.Click Save. You add no routing logic: the edge itself is the route, because routing is structural. Re-run the smoke test: Shell uv run python orchestrators/verify_run.py langgraph The path now includes contradiction-detector, and because approach-analyzer and contradiction-detector run concurrently, their order can vary. You completed a multi-agent workflow from the UI, and the config you wired in already had its tool attached. You finished a multi-agent workflow from the UI, mid-development, and the dispatcher ran the new shape on the next request. No redeploy, no code change: the graph you draw is the graph that runs. Step 6: Smoke Test All Four Frameworks Before you collect experiment data, make sure all four frameworks can run the completed graph. One command runs all four in sequence: Shell uv run python orchestrators/verify_run.py all It runs each framework against the completed graph and ends with a pass/fail summary, one line per framework, exiting non-zero if any framework failed, so it works as a gate. Each framework prints the path it took and a preview of its report, then a final summary collects the results. A successful run looks like this: Plain Text ▶ Running 'langgraph' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer ▶ Running 'strands' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer ▶ Running 'openai-agents' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer ▶ Running 'google-adk' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer === smoke summary === ✓ langgraph ✓ strands ✓ openai-agents ✓ google-adk If a framework fails, its line shows an ✗ instead of a ✓ and the command exits non-zero. All four smoke tests against the pinned Claude model. ANTHROPIC_API_KEY is the only model key you need, because OpenAI Agents and Google ADK reach Claude through LiteLLM. The OpenAI Agents SDK turns on tracing by default and looks for OPENAI_API_KEY to export traces, so the openai-agents run may print a harmless tracing warning when that key is absent. It doesn’t affect the run. Step 7: Run It Through the Experiment Now you can use a LaunchDarkly experiment to rank the four frameworks on real traffic, on the same graph, with the model held constant. Because the model is fixed, the comparison is operational: which orchestrator delivers the model’s quality fastest, with the least token overhead. The bootstrap already created the flag, the judge, and the graph. These metrics are measured on each request, so do a one-time setup first: Make the request context kind available for experiments.Set the analysis unit of graph latency, tokens, and the judge metric to request. Then create the experiment in the UI: Create an experiment with the orchestrator flag as the treatment.Set the primary metric to Graph latency ($ld:ai:graph:duration:total, the time for a complete graph execution).Add tokens and $ld:ai:judge:gap-quality as secondary metrics.Set the audience to 100% and the randomization unit to request. Each run is a single request, there are no users in this workflow, and request is the unit LaunchDarkly measures AI and graph metrics by.Turn on the orchestrator flag, which the bootstrap created set to off, so it serves the experiment’s variations.Start an experiment iteration. We rank on latency and tokens because, with the model and the graph held constant, those are the things that genuinely differ: a framework can move quality only by degrading the plumbing, like a truncated report or a broken tool call. So $ld:ai:judge:gap-quality stays a guardrail that catches a framework “winning” by cutting corners, not part of the ranking. Swap the model, prompt, or tools later instead of the framework, and that same judge becomes your primary metric. Then drive traffic. The flag assigns each run one framework at random: Shell uv run python scripts/run_experiment.py --runs-per-category 6 That’s six runs over each of the six shipped topics, 36 in total. Assignment is random, so it usually fills all four variations, though it isn’t guaranteed. Each run analyzes the topic’s entire paper set, because gap analysis needs every paper to find real gaps. Open the experiment in LaunchDarkly: latency per variation, with tokens and $ld:ai:judge:gap-quality alongside. The winner is the framework with the best latency and lowest token use that doesn’t let quality slip. Because the model is pinned, cost is a fixed multiple of tokens, so the token column is also the cost ranking; for actual dollar figures, read them from Insights. Because the experiment holds everything but the framework constant, most of these bars land close, often within a few percent, which is by design. In our run, Strands won on speed: it ran the graph fastest, with quality holding at the guardrail. If you optimize for speed and quality holds, that makes Strands the orchestrator to ship for this workload. Six topics and one randomized split isn’t a large sample, so confirm the lead with more topics before you standardize on it. You can do that in Step 9. Step 8: Ship the Winner With Runtime Control The experiment gave you data. The reason to run it in LaunchDarkly, rather than a one-off script, is that acting on that data takes no deploy: the orchestrator flag that was the experiment treatment is also your production router. When a variation wins, stop the iteration and set the flag’s default to that framework. Every request routes to it on the next evaluation, with no redeploy. Then automate what you don’t want to babysit. An adaptive trigger watches a guardrail and changes a flag on its own when production drifts past it. The orchestrator you shipped is operational and won’t degrade by itself, so point the trigger at the model flag from Step 9: it fails over to a backup model when your primary provider has a bad day, the same guardrail driving a different flag. That closes the loop: experiment to find the winner, runtime control to ship it, and automation to keep it healthy. Step 9: Extend the Experiment Tighten the bands by adding more topics. Confidence comes from more distinct topics, not more runs over the same few. Download one with a title-phrase (ti:) query, and the harness picks it up automatically on the next run: Shell uv run python scripts/download_papers.py --query 'ti:"LLM-as-a-judge"' Make quality the headline by flipping a config, not a flag. The framework lives in the orchestrator flag because it is app-level routing, not a property of any agent. The model, the prompt, and the tool set are different: they live in the node configs, so you experiment on the config itself. Add a second variation to a node, such as gap-synthesizer with a stronger model or a tightened prompt, and run an experiment with that config as the treatment and its variations as the arms. Pin the framework by setting the orchestrator flag to one value and leave the graph alone, so the config is the only thing moving. The judge attached to the synthesizer already emits $ld:ai:judge:gap-quality, so quality is the primary metric with no new instrumentation. Now it genuinely moves, because a different model or prompt reasons differently about the same papers. Experiment on the graph shape with a graph-key flag. The dispatcher takes the graph key as an argument, so the shape is another value you can put behind a flag: Python graph_key = ld.variation("graph_shape", context, "research-gap-graph") result = await execute_graph(ai_client, graph_key, context, user_input, build_agent, invoke) Build two graphs with different keys: for example, a linear research-gap-graph-linear (intake to approach-analyzer to gap-synthesizer) against the parallel research-gap-graph, or one with an added critic node against one without. Make a multivariate graph_shape flag whose variations are those graph keys, evaluate it exactly as the app evaluates orchestrator, and set it as the experiment treatment with the framework and model held constant. You are measuring whether the extra structure earns its latency and quality, and because the dispatcher runs whatever shape the key resolves to, no runner or dispatcher code changes. You build the judge once, and it is the guardrail for the framework bake-off, and the headline metric for every model, prompt, tool, and shape you test next. Run a native-model bake-off. This experiment holds the model constant so the framework is the only variable. To compare each framework on its own default model instead, build separate node configs per framework. This is the optional bake-off the prerequisites mention. It’s a follow-up beyond this walkthrough, and the only part that needs OPENAI_API_KEY and GOOGLE_API_KEY. Whatever you flip, follow three rules: Change one variable at a time (the framework, the model, or the shape), never two. If you change more than one, you can’t attribute the win.Keep the quality guardrail on every run, because the fastest variant is often the one that quietly truncated its report or dropped a tool call.Earn confidence with distinct inputs, not repeats: a tight band around three repeated topics is still a tight band around the wrong number. To learn more about judge design, read When to add online evals and Evaluating with LLM-as-judge evaluators. To add a pre-production regression layer, read Offline evaluation of RAG-grounded answers. Recap and Next Steps Framework choice doesn’t have to be a one-way door. Put the topology in a LaunchDarkly agent graph, have each framework supply only build_agent and invoke, and let one experiment settle a question that usually gets answered by whoever argues hardest: pin the model, let the judge guard quality, and pick the orchestrator that delivers it fastest, with evidence in hand. Then keep going, because the framework is only the first swappable component. The same flag, experiment, and judge machinery compares models, prompts, tools, and whole graph shapes the same way, so “which is better” stops being a debate and becomes a measurement. And because the experiment and the runtime control are one flag, you never stop at a finding: you ship it, ramp it with a progressive rollout, and let an adaptive trigger hold the line in production while the AI iteration loop for reliable agents keeps the next change shipping behind eval gates. The complete code is in the sample repo. Get started with AgentControl, point the four frameworks at a graph your team actually runs, and settle the next framework argument with a number instead of a hunch.

The Model Context Protocol connects AI agents to your databases, APIs, and file systems. Out of the box, it connects them with no identity, no scoping, and no audit trail. The MCP specification acknowledges this gap explicitly. Its OAuth 2.1 authorization spec marks authentication as optional. The result, according to research published on Security Boulevard in April 2026, is that 53 percent of open-source MCP implementations ship with static API keys. Eighty-eight percent require backend authentication, but only 8.5 percent implement proper credential management. Every one of those static keys is a credential waiting to be stolen, a scope waiting to be abused, and an audit entry that will read "unknown agent executed query" when the incident report is written. This article builds the alternative. We will build an MCP server in Python that accepts tool calls only from authenticated agents, validates OAuth 2.1 Bearer tokens using stateless JWKS-based validation, enforces tool-level scopes and roles, maintains an infrastructure-level tool allow-list, and logs every access decision with the full delegation chain back to the human who authorized it. The complete companion project, roughly 350 lines of Python with a 13-test suite, is available on GitHub. Prerequisites You will need Python 3.12 or later and an OIDC-compatible identity provider. The examples use Auth0 (free tier works), but Okta, Keycloak, Entra ID, or any provider that exposes a /.well-known/jwks.json endpoint will work. Basic familiarity with OAuth 2.1 concepts and MCP server architecture is assumed. All code shown is extracted from the companion project. File paths reference code/src/. Architecture Every tool call flows through five gates before reaching your business logic: Architecture: Five-gate MCP tool call authorization pipeline. Gates two and three are infrastructure-level controls. System prompts are not security controls. An MCP server the agent has not been explicitly authorized to call should be unreachable. Period. Regardless of what the LLM decides to invoke. Part 1: JWKS-Based Token Validation The foundation of an identity-aware MCP server is stateless JWT validation. Every request carries a Bearer token issued by your OAuth 2.1 authorization server. The MCP server validates it against the provider's JSON Web Key Set, a public key document that lets you verify signatures without a network call to the IdP on every request. The JWKS Cache Create src/auth/middleware.py. We start with a cache that fetches the JWKS once and holds it in memory, refreshing every five minutes or on-demand when an unknown key ID appears (key rotation): Python class JWKSCache: """Cached JWKS with automatic refresh on unknown key id.""" def __init__(self, jwks_url: str, cache_ttl: int = 300): self._url = jwks_url self._ttl = cache_ttl self._keys: dict[str, dict] = {} self._last_fetch: float = 0 async def get_key(self, kid: str) -> dict: if not self._keys or (time.monotonic() - self._last_fetch) > self._ttl: await self._refresh() key = self._keys.get(kid) if key is None: logger.info("Unknown kid '%s', forcing JWKS refresh", kid) await self._refresh() key = self._keys.get(kid) if key is None: raise AuthError(f"Key '{kid}' not found in JWKS", 401) return key async def _refresh(self) -> None: if self._url.startswith("http"): async with httpx.AsyncClient() as client: resp = await client.get(self._url, timeout=10) resp.raise_for_status() jwks = resp.json() else: with open(self._url) as fh: jwks = json.load(fh) self._keys = {k["kid"]: k for k in jwks.get("keys", [])} self._last_fetch = time.monotonic() The get_key method is where the key rotation logic lives. When a token arrives with a kid the cache has never seen, we force a refresh before rejecting it. An unknown kid could mean a legitimate rotation, not an attack. We try once more before failing. In practice, this means you never need to restart your MCP server when your identity provider rotates signing keys. The Token Validator The validator uses the cache to verify every Bearer token. It checks five things, and the order matters: header validity, signature, issuer, audience, and expiry: Python class TokenValidator: def __init__(self, jwks_url: str, issuer: str, audience: str, clock_tolerance: int = 30): self._jwks = JWKSCache(jwks_url) self._issuer = issuer self._audience = audience self._clock_tolerance = clock_tolerance async def validate(self, token: str) -> ValidatedToken: # 1. Decode header to get the key id. unverified = jwt.get_unverified_header(token) kid = unverified.get("kid") if not kid: raise AuthError("Token header missing 'kid' claim", 401) # 2. Fetch the matching public key. jwk = await self._jwks.get_key(kid) # 3. Verify signature + standard claims. claims = jwt.decode( token, jwk, algorithms=["RS256"], issuer=self._issuer, audience=self._audience, options={"verify_exp": True, "require": ["exp", "iss", "sub", "aud"]}, ) # 4. Clock-tolerance check (belt-and-suspenders with the library). now = int(time.time()) if claims["exp"] + self._clock_tolerance < now: raise AuthError("Token has expired", 401) # 5. Extract scopes, roles, and delegation chain. scope_str = claims.get("scope", "") token_scopes = set(scope_str.split()) roles = claims.get("roles", []) delegation_chain = self._extract_delegation(claims) return ValidatedToken( subject=claims["sub"], email=claims.get("email"), roles=roles, scopes=token_scopes, delegation_chain=delegation_chain, ) The iss (issuer) check prevents tokens from a different authorization server from being accepted. The aud (audience) check prevents tokens intended for a different service from being replayed against yours. The exp check with clock tolerance handles the reality that clocks drift. Thirty seconds of tolerance is the pragmatic default recommended by the Upstash MCP OAuth deep-dive. The delegation chain extraction is worth examining separately. When an agent acts on behalf of a human who authorized it, RFC 8693's act claim carries that nesting. We recursively unpack it: Python def _extract_delegation(self, claims: dict) -> list[str]: chain = [] act = claims.get("act", {}) while act: sub = act.get("sub", "") if sub: chain.append(sub) act = act.get("act", {}) return chain A token issued directly to a human will have an empty delegation chain. A token issued to an agent acting on behalf of "[email protected]" will carry ["[email protected]"]. A multi-hop chain, human to orchestrator agent to sub-agent, carries both identifiers in order. This is what lets your audit logs trace every action back to a person. Part 2: The Two Mandatory Discovery Endpoints An MCP client connecting to your server needs to discover two things: that authentication is required, and where to get tokens. The MCP specification mandates two well-known endpoints for this, defined in RFC 9728 and RFC 8414, respectively. Create src/auth/discovery.py: Python def build_discovery_routes( resource_url: str, authorization_server_url: str, scopes_supported: list[str] | None = None, ) -> dict: async def protected_resource(request: Request) -> JSONResponse: return JSONResponse({ "resource": resource_url, "authorization_servers": [authorization_server_url], "bearer_methods_supported": ["authorization_code"], }) async def authorization_server(request: Request) -> JSONResponse: return JSONResponse({ "issuer": authorization_server_url, "authorization_endpoint": f"{authorization_server_url}/authorize", "token_endpoint": f"{authorization_server_url}/oauth/token", "jwks_uri": f"{authorization_server_url}/.well-known/jwks.json", "scopes_supported": scopes_supported or [ "database.read", "database.write", "email.send", "admin.users.read", ], "response_types_supported": ["code"], "grant_types_supported": ["authorization_code", "client_credentials"], "code_challenge_methods_supported": ["S256"], "token_endpoint_auth_methods_supported": ["none"], }) return { "/.well-known/oauth-protected-resource": protected_resource, "/.well-known/oauth-authorization-server": authorization_server, } Without these endpoints, MCP clients cannot auto-discover your authentication configuration. The client first hits your server without a token, receives a 401 with a WWW-Authenticate header pointing to the protected resource metadata, fetches it to confirm auth is required, then reads the authorization server metadata to learn the token endpoint and supported grant types. code_challenge_methods_supported: ["S256"] is not optional. MCP clients are public clients. They cannot keep a client secret, so PKCE is the only defense against authorization code interception. The NAPTHA AI reference implementation explicitly documents this. Part 3: Tool Definitions With Scope and Role Requirements Now we define the tools themselves. Each tool declares what scopes and roles are required to invoke it. These declarations live alongside the tool code, not in a separate config file. Proximity reduces the chance of drift between a tool and its authorization requirements. Create src/tools/database.py: Python # Each tool is a handler with declared requirements. TOOL_REGISTRY: dict[str, tuple[list[str], list[str], callable]] = { "read_customer_record": ( ["database.read"], # required scopes [], # required roles read_customer_record, # handler ), "update_customer_plan": ( ["database.write"], [], update_customer_plan, ), "list_all_customers": ( ["admin.users.read"], ["admin"], # admin role required list_all_customers, ), } A developer with database.read scope can read customer records but cannot update plans. A contractor with no scopes gets blocked from everything. An admin with admin.users.read scope and the admin role can list all customers. The registry is the single source of truth for access control. The server enforces it at request time without consulting a database. Here is one tool handler showing resource-level constraint enforcement: Python async def read_customer_record(customer_id: int, *, _token=None) -> dict: # Optional: enforce per-resource constraints from the token. if _token and hasattr(_token, "raw_claims"): constraint = _token.raw_claims.get("resource_constraints", {}) allowed_id = constraint.get("customer_id") if allowed_id is not None and customer_id != allowed_id: raise PermissionError( f"Token scoped to customer {allowed_id}, " f"requested customer {customer_id}" ) record = _CUSTOMER_DB.get(customer_id) if record is None: raise ValueError(f"Customer {customer_id} not found") return record The resource_constraints claim in the token is what turns "this agent can read customer data" into "this agent can read customer 48291 for the next sixty seconds." It is the difference between scoping to a database table and scoping to a row. Part 4: The Tool Allow-List Gate System prompts are not security controls. A prompt injection can rewrite an agent's intent mid-session and convince it to call a tool it was never meant to access. The only reliable defense is an infrastructure-level allow-list that rejects unauthorized tool calls regardless of what the LLM decides. The allow-list is derived directly from the tool registry. Any tool not in the registry is unreachable: Python ALLOWED_TOOLS: set[str] = set(TOOL_REGISTRY.keys()) This set is checked before scope and role evaluation. A tool that is not in the registry cannot be called, period. A tool that is in the registry but requires scopes the token does not carry gets a 403. A tool that is in the registry and the token carries the right scopes goes through. The distinction between "tool not in allow-list" and "tool forbidden for this agent" matters for debugging and audit. The first indicates a misconfiguration or an attack. The second indicates a legitimate agent attempting an unauthorized operation, which itself is worth logging. Part 5: The Audit Logger Every tool call, successful or blocked, produces an audit log entry with the full delegation chain. The format is JSON Lines: one JSON object per line, ingestible by any SIEM, Splunk, or grep. Create src/audit/logger.py: Python class AuditLogger: def __init__(self, filepath: str | Path = "audit.log") -> None: self._path = Path(filepath) self._path.touch(exist_ok=True) def record(self, event: str, token: ValidatedToken, tool_name: str = "", tool_args: dict | None = None, result_summary: str = "", error: str = "") -> None: entry = { "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "event": event, "correlation_id": str(uuid.uuid4()), "subject": token.subject, "email": token.email, "roles": token.roles, "scopes": sorted(token.scopes), "delegation_chain": token.delegation_chain, "tool": tool_name, "tool_args": tool_args or {}, "result": result_summary, "error": error, } with open(self._path, "a") as fh: fh.write(json.dumps(entry, default=str) + "\n") When an auditor asks "who authorized this data access," the answer is in the log, not in a code review three weeks later. A correctly logged tool call looks like this: Python { "timestamp": "2026-06-14T14:04:00Z", "event": "tool_call", "subject": "alice-developer", "email": "[email protected]", "roles": ["developer"], "scopes": ["database.read", "email.send"], "delegation_chain": ["bob-admin"], "tool": "read_customer_record", "tool_args": {"customer_id": 1001}, "result": "ok" } Delegation chain flow: Human → Orchestrator Agent → Sub-Agent → MCP Server. The delegation chain reads: Bob (admin) delegated to Alice's developer agent, which called read_customer_record for customer 1001 at 14:04 UTC. If your logs cannot produce that sentence, your AI identity program is not operational. Part 6: Assembling the Server The main server wires together the token validator, the tool allow-list, the scope and role checks, the tool handlers, and the audit logger. Every request flows through them in order. Create src/server.py. Here is the core request path: Python token_validator = TokenValidator( jwks_url=OIDC_JWKS_URL, issuer=OIDC_ISSUER, audience=OIDC_AUDIENCE, clock_tolerance=30, ) audit = AuditLogger(AUDIT_LOG_FILE) async def mcp_tool_endpoint(request: Request) -> JSONResponse: # 1 — Extract and validate the Bearer token. auth = request.headers.get("Authorization", "") if not auth.startswith("Bearer "): raise AuthError("Missing Bearer token", 401) token_str = auth[7:] try: token = await token_validator.validate(token_str) except AuthError: audit.record("auth_failure", ...) raise # 2 — Parse the tool invocation. body = await request.json() tool_name = body.get("tool", body.get("name", "")) tool_args = body.get("arguments", body.get("args", {})) # 3 — Tool allow-list enforcement. if tool_name not in ALLOWED_TOOLS: audit.record("tool_allow_list_block", token, tool_name=tool_name) return JSONResponse( {"error": f"Tool '{tool_name}' is not authorized"}, status_code=403, ) # 4 — Scope + role authorization. required_scopes, required_roles = get_tool_requirements(tool_name) if required_scopes and not token.has_any_scope(required_scopes): return JSONResponse( {"error": "Insufficient scopes", "required": required_scopes, "granted": sorted(token.scopes)}, status_code=403, ) if required_roles: if not (set(token.roles) & set(required_roles)): return JSONResponse( {"error": "Insufficient role", "required_one_of": required_roles, "have": sorted(token.roles)}, status_code=403, ) # 5 — Execute and audit. handler = TOOL_REGISTRY[tool_name][2] result = await handler(**tool_args, _token=token) audit.record("tool_call", token, tool_name=tool_name, tool_args=tool_args, result_summary=str(result)[:200]) return JSONResponse({"result": result}) The 401 response format is specified by the MCP specification. The WWW-Authenticate header with resource_metadata is how clients discover that authentication is required: Python async def auth_error_handler(request, exc): return Response( content='{"error":"' + exc.args[0] + '"}', status_code=401, media_type="application/json", headers={ "WWW-Authenticate": ( f'Bearer resource_metadata=' f'"{AUDIENCE}/.well-known/oauth-protected-resource",' f'error="invalid_token"' ), }, ) Part 7: The Demo Agent To verify the server end-to-end without configuring a real OAuth provider, the companion project includes a demo agent that generates self-signed tokens for three simulated identities. Run it with python demo/agent.py --demo. The demo creates three agents with progressively restricted access: Plain Text Agent 1: Alice — developer, scopes: database.read + email.send ✓ Can read customer records ✗ Cannot update plans (missing database.write) ✗ Cannot list all customers (missing admin role) Agent 2: Bob — admin, scopes: database.read + database.write + admin.users.read ✓ Can read customer records ✓ Can update plans ✓ Can list all customers Agent 3: Carol — contractor, scopes: (none) ✗ Blocked from everything This is not a theoretical exercise. In the Stryker attack of March 2026, a compromised admin credential, one identity, over-privileged, with no scoping, allowed attackers to remotely wipe 200,000 devices across 79 countries. The attack did not use malware. It used the platform's own legitimate wipe functionality. The credential had no scope limiting it to a subset of devices, no short lifetime, and no audit trail that would have surfaced the anomaly before tens of thousands of endpoints were erased. Part 8: Testing The companion project includes a 13-test suite that verifies every security gate. Run it with: Python python -m pytest tests/ -v The test matrix covers the decision table exhaustively: TestConditionExpectedNo tokenMissing Authorization header401Invalid tokenMalformed JWT401Expired tokenexp in the past401Valid token + correct scopedatabase.read calling read_customer_record200Valid token + wrong scopeemail.send calling read_customer_record403Valid token + missing scopedatabase.read calling update_customer_plan403Valid token + correct scopesdatabase.read database.write calling update_customer_plan200Valid token + wrong roledeveloper role calling list_all_customers403Valid token + correct roleadmin role calling list_all_customers200Unknown tooldelete_everything not in allow-list403Discovery: protected resourceUnauthenticated GET200Discovery: authorization serverUnauthenticated GET200Audit log entriesTool call with delegation chainWritten with full chain Each test generates a real RSA key pair, signs a JWT with it, loads a matching JWKS, and sends a request through the full server stack using Starlette's TestClient. No mocking of the auth layer. The tests exercise the actual token validation code path. Part 9: Common Pitfalls localhost vs 127.0.0.1 redirect URI mismatch. MCP clients running locally often register 127.0.0.1 as their redirect URI, but the authorization server redirects to localhost (or vice versa). The Upstash OAuth deep-dive documents this as the most common integration failure. Normalize both addresses at registration and at token exchange. Cursor re-registers OAuth clients on every connection. The Dynamic Client Registration endpoint must handle the same client identity registering repeatedly. Store by client identity, not by registration request. Idempotency is critical. Clock skew causing spurious rejections. A 30-second clockTolerance is the pragmatic default. Distributed systems have clock drift. Rejecting a valid token because the IdP's clock is 12 seconds ahead of yours is a self-inflicted outage. Forgetting to serve discovery endpoints over HTTPS. MCP clients will refuse to fetch well-known URIs over plain HTTP in production. If your server is behind a load balancer, ensure the resource_url reflects the externally visible HTTPS URL, not the internal service name. Logging Bearer tokens. Sanitize the Authorization header from request logs. A leaked Bearer token in your logging pipeline is an identity compromise waiting to happen. The audit logger in this project intentionally records the validated identity, never the raw token. Production Hardening Before deploying to production, lock down the following: PKCE (S256) is mandatory. MCP clients are public clients without a client secret. PKCE is the only defense against authorization code interception.Short-lived tokens. Fifteen to sixty minutes, with refresh token rotation. Each use of a refresh token invalidates the previous one.HTTPS only. HTTP must be rejected at the network level. The MCP security best practices specification explicitly prohibits plaintext.Session-based authentication is prohibited. The MCP spec mandates token-based authentication. No cookies, no sessions.Audit log rotation and retention. JSON Lines accumulate quickly at production throughput. Configure log rotation and feed the audit stream to your SIEM. What We Built We built an MCP server that accepts tool calls only from authenticated agents. It validates OAuth 2.1 Bearer tokens using stateless JWKS-based validation with automatic key rotation. It enforces tool-level scopes and roles. A developer with database.read cannot write. A contractor with no scopes gets blocked from everything. An admin with the right role and scope can list all records. It maintains an infrastructure-level tool allow-list that rejects unauthorized tool calls regardless of what the LLM decides. It logs every access decision with the full delegation chain, so an auditor can trace any action back to the human who authorized it. The standards to do this at scale are maturing rapidly. SPIFFE handles workload identity. RFC 8693 covers token exchange with delegation chains. The IETF AIMS framework addresses agent identity. The engineering to do it in a single Python file is deployable today. The companion project is available on GitHub with setup instructions, a working demo, and a 13-test suite. Clone it, configure your OAuth provider, and you have an identity-aware MCP server in under 200 lines of application code. GitHub repository: github.com/pravin-khandke/identity-aware-mcp-server Clone it and run the demo in under two minutes: Shell git clone https://github.com/pravin-khandke/identity-aware-mcp-server.git cd identity-aware-mcp-server python3 -m venv .venv && source .venv/bin/activate pip install -r requirements.txt python demo/agent.py --demo All code shown in this article is extracted from the repository. See src/auth/middleware.py for the JWKS validator, src/server.py for the full request pipeline, and tests/test_server.py for the 13-test suite.

Running AWS resources locally is a game-changer for engineering velocity, cost optimization, and developer autonomy. Traditionally, testing cloud infrastructure required deploying directly to a staging or sandbox AWS account. This workflow introduced painful friction points: waiting for slow cloud provisioning cycles, tracking down orphaned resources that inflate the monthly bill, and requiring a constant, high-speed internet connection. LocalStack solves this by emulating core AWS services, such as S3, SQS, DynamoDB, and other services directly on your local machine inside a Docker container. When paired with Terraform, you can safely write, plan, and apply infrastructure-as-code (IaC) configuration blueprints against this local simulator. This guide walks you through the definitive "happy path" for configuring LocalStack and Terraform, followed by a robust troubleshooting handbook for common architecture-specific and container networking errors. This allows you to provision these mock resources cleanly. This allows testing Terraform code with local resources without incurring real AWS costs, requiring internet connectivity, or dealing with slow cloud provisioning cycles. The Happy Path Setup Step 1: Setting the Stage: Launching LocalStack With Docker To get started, we need our local AWS cloud stack running inside a container. We will pull the official LocalStack image, set up our credentials, and spin up the container. First, pull the latest official image to your local machine: Before firing up the container, head over to the LocalStack Web App Dashboard to grab your personal access token (PAT). While LocalStack offers an open-source community edition, advanced features or specific emulated APIs may check for a valid token. Export this token into your shell environment so the container can authenticate and activate premium features on startup: Shell export LOCALSTACK_AUTH_TOKEN="ls-..." Now, launch the container. We need to map the primary edge gateway port (4566), which routes all inbound AWS API requests, along with the standard range of ports used by individual internal services (4510-4559). We also pass our token as an environment variable: Shell docker run --rm -it \ -p 4566:4566 \ -p 4510-4559:4510-4559 \ -e LOCALSTACK_AUTH_TOKEN=$LOCALSTACK_AUTH_TOKEN \ localstack/localstack Keep an eye on your terminal logs. LocalStack will quickly validate your token, pull your license configuration, and initialize the mock runtimes. You will see a clear notification when the edge proxy is fully ready to handle incoming API requests. Step 2: The S3 Sanity Check: Talking to LocalStack Before configuring our automation toolchain, let's run a quick manual sanity check using the standard AWS CLI. Because LocalStack runs entirely on your machine, we must override the default cloud routing by passing a custom --endpoint-url pointing to our local edge proxy. To verify that LocalStack is running and reachable, create a local S3 bucket and upload a test file using the AWS CLI. 1. Create a Bucket Shell aws s3 \ mb s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 2. Upload an Object Create a dummy text file and copy it into your new mock bucket: Shell aws s3 \ cp /tmp/demo.txt s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 3. List Objects Verify the object is safely stored inside the mock container: Shell aws s3 \ ls s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 Step 3: Writing the Blueprint: Configuring the Terraform Provider Now let's automate things. To instruct Terraform to deploy resources to our local simulator instead of the real AWS cloud, we must customize the AWS provider block. We enforce dummy credentials, bypass cloud-only identity validations, and explicitly force all API endpoints to route directly to http://localhost:4566. Providers Configuration Create a file named providers.tf with the following content: Markdown terraform { backend "local" { path = "terraform.tfstate" } required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } } provider "aws" { region = "us-east-1" access_key = "mock_access_key" secret_key = "mock_secret_key" skip_credentials_validation = true skip_metadata_api_check = true skip_requesting_account_id = true s3_use_path_style = true # Redirect all endpoints to LocalStack's edge port endpoints { apigateway = "http://localhost:4566" cloudwatch = "http://localhost:4566" dynamodb = "http://localhost:4566" ec2 = "http://localhost:4566" iam = "http://localhost:4566" lambda = "http://localhost:4566" rds = "http://localhost:4566" s3 = "http://localhost:4566" secretsmanager = "http://localhost:4566" sns = "http://localhost:4566" sqs = "http://localhost:4566" ssm = "http://localhost:4566" sts = "http://localhost:4566" } } SQS Resource Definition Next, define the SQS queue we want to provision. Create a file named main.tf: Markdown resource "aws_sqs_queue" "local_queue" { name = "my-local-queue" delay_seconds = 90 max_message_size = 2048 message_retention_seconds = 86400 receive_wait_time_seconds = 10 } output "queue_url" { value = aws_sqs_queue.local_queue.id } Step 4: The Moment of Truth: Initializing and Applying Configuration With our configuration defined, we can run Terraform. Ensure you are executing a native binary that matches your host system architecture (such as a native darwin_arm64 binary if you are working on an Apple Silicon machine) to prevent execution overhead. Initialize Terraform First, initialize the working directory to download the AWS provider plugins: Generate and Review the Plan Next, generate and review an execution plan. The plan output will detail our local queue configuration without attempting to connect to actual AWS endpoints: Apply the Plan Apply the plan to deploy the queue directly to LocalStack. Upon completion, Terraform will write your state file locally and output your new mock SQS queue URL: Step 5: Taking It for a Spin: Sending and Receiving SQS Messages To confirm that our Terraform-provisioned SQS queue is fully operational, let's capture the output URL and push a real message through it using the AWS CLI. 1. Send a Message Shell export QUEUE_URL="http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/my-local-queue" aws sqs send-message \ --endpoint-url=http://localhost:4566 \ --region us-east-1 \ --queue-url $QUEUE_URL \ --message-body "Hello from LocalStack SQS" 2. Receive the Message Shell aws sqs receive-message \ --endpoint-url=http://localhost:4566 \ --region us-east-1 \ --queue-url $QUEUE_URL { "Messages": [ { "MessageId": "1235d997-f60a-4e86-b248-aff3f5f41dde", "ReceiptHandle": "NzYxOThkMDAtMWJiOC00OGVhLTllMDEtNTU3ZTY3ZGQ5M2I4IGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6bXktbG9jYWwtcXVldWUgMTIzNWQ5OTctZjYwYS00ZTg2LWIyNDgtYWZmM2Y1ZjQxZGRlIDE3ODI4OTI2MzcuMjg2ODc1NQ==", "MD5OfBody": "88dc2faa42b899c03e12fd3ac96d714b", "Body": "Hello from LocalStack SQS" } ] } Your terminal will return a successful JSON payload containing your message body, confirmation IDs, and MD5 hashes, proving that the local loop is entirely complete. Event Verification in LocalStack Logs Checking the LocalStack container console confirms the queue creation, message send, and message fetch operations were handled successfully: Troubleshooting Guide Even on a happy path, local container networks and mixed system architectures can throw a wrench into your workflow. Here is how to fix the most common bottlenecks. The Apple Silicon (M1/M2/M3) Rosetta Loop Symptom: The LocalStack container crashes unexpectedly on startup, or loops endlessly while attempting to launch internal components like local Lambda runtimes, throwing qemu: uncaught target signal 11 errors. The Cause: LocalStack occasionally spins up secondary processes or helper binaries inside the container. If Docker Desktop is forced to emulate an x86_64 architecture via Virtualization frameworks on an ARM64 Apple Silicon chip, the emulation layer can break during heavy nested execution. The Fix: Ensure your Docker Desktop configuration has Use Virtualization framework enabled under Settings -> General, and turn on Rosetta for x86/amd64 emulation under the Features in Development tab. Alternatively, force Docker to fetch the native ARM64 container image by updating your execution command to include the specific platform flag: Shell docker run --platform linux/arm64 --rm -it -p 4566:4566 localstack/localstack "Port Already in Use" Symptom: Docker fails to bind ports, displaying an error message like: Bind for 0.0.0.0:4566 failed: port is already allocated. The Cause: A previous instance of LocalStack didn't shut down cleanly, or another local development tool is monopolizing port 4566. The Fix: Option 1: Check for lingering Docker containers Often, a container crashed or was backgrounded but didn't release the port. Find any container using 4566: Shell docker ps -a | grep 4566 If a container shows up, stop and remove it (replace <CONTAINER_ID> with your specific ID): Shell docker stop <CONTAINER_ID> docker rm <CONTAINER_ID> Option 2: Kill native background processes If Docker isn't holding the port, another process on your host machine is. You'll need to find its Process ID (PID) and force-quit it. Find the PID: Shell lsof -i :4566 Kill it (look for the number under the PID column): Shell kill -9 <PID> Wrapping Up Combining LocalStack and Terraform gives you a lightning-fast, zero-cost, offline sandbox for cloud infrastructure development. Once your environment is configured correctly with a valid personal access token, precise Docker port mappings, and native toolchains matched to your host CPU, you can prototype, test, and tear down AWS configurations in seconds. No more waiting for slow cloud deployments or tracking down orphaned cloud resources. Happy local provisioning!

A few weeks ago, I participated in a 24-hour AI hackathon where we built a product using AI. Necessary tools were provided, a large number of engineers participated eagerly, and a few business folks also joined to bring their ideas into a real-world product using AI. During brainstorming, people drafted end-to-end process flow diagrams and started working on development, using all the recent available models and platforms to build their product. When the development time window ended, it was time for presentations. As I watched each team present their results, I observed that they couldn’t automate the end-to-end process flow. What they had planned during brainstorming didn’t turn out to be a complete, end-to-end product. Most of the solutions followed the same pattern: they did something in one product and took the output to another product, and the output of that product went somewhere else to finish the loop. Since the transition was handled by a human, it was considered “human in the loop.” No doubt this speeds up the process, but it left a gap — between an AI system that helps finish certain tasks in a process flow and an AI system that finishes the process end to end. Which one is the better model to build? Let’s Try to Define Assist and Complete An assisting system sits inside a workflow. AI takes some kind of input and produces an output. A human then takes the output, reviews it, and decides what to do with it — moving to the next system with or without AI, and continuing the process flow. If we removed the AI from this process flow, it could still run, and the work would still get done. It would just take more time. For example, in a real-world tech process, a business team uses AI to document a requirement, which is converted into a JIRA story using the Confluence AI tool, and a developer reads it and uses Copilot or another AI assistant to develop the code. A complete system owns the workflow’s outcome. Here, the system takes the input, processes it, makes the decision, triggers an action, and writes a log documenting the actions it has performed. If we remove any component from this process flow, it won’t just slow down — it will completely stop. As a real-world example, there is a solution that takes all the reviews for a company, analyzes them, categorizes them, summarizes them, and then raises a ticket with that summary and assigns it to the respective team. As an evolution of this process, teams often look only at their current work and how they can leverage AI in it, rather than looking at the end-to-end process flow integration. This is one of the reasons why AI investment doesn’t show up in the metrics that matter. It was architected to help, for sure, but not to finish. The Four-Part Test After watching this pattern across a number of different workflows, I have formed a set of four questions to address whether a process is structurally ready for a complete system, or whether it’s just going to support an assisting one. Is the trigger unambiguous? Something has to start the cycle — a batch starts on a schedule, a microservice is triggered by a button in the UI, a sensor crosses a threshold. If the start of the workflow is itself a judgment call that varies by who is looking at it, then we don’t have a clear trigger, and the downstream process isn’t ready to be fully automated. Can you define the decision logic? “Define” isn’t simple, but the system’s decision logic can be made explicit, even if the underlying model is complex. In a reliability process flow system, you can define what should be considered an error, what the threshold is, and when to act. Contrast that with deciding whether a customer complaint is about to escalate into a PR problem — the reasoning experienced staff uses to answer that is often genuinely tacit, inconsistent between two employees, and resistant to being written down.Is it executable in a real system? A complete system needs more than just an AI model. It might need an API, a write permission, a button it can press — not a dashboard a human reads and acts on separately based on what they see in it. This sounds obvious, doesn’t it? But teams often build excellent reasoning and never build the hooks to act on it. Can the outcome be verified? This is the most important question of all. After the action completes, can the system check and confirm whether the right action was taken and that it worked? Did the email send, did the case close correctly, did the transaction process without a downstream error? Verifiability is the closing step of a fully automated system, and it’s what makes the system safe. It’s what catches a bad decision before it compounds tenfold. When a workflow answers all four of these questions well, it is a strong candidate for a complete system. When even one question fails, that’s a signal that the workflow should stay an assisting one, regardless of how sophisticated the underlying model is. Why Most Systems Assist But Don’t Fully Complete A simple, straightforward question is why so many production systems stop at assist. In my experience, there are three structural reasons: Narrow decision logic: If a system has narrow decision logic in the process flow — binary answers like yes or no, or a defined set of cases — it can often be handled within the process flow regardless of AI involvement. But people try to introduce AI here anyway, and when it picks an answer that wasn’t pre-defined, that’s treated as a failure.Evidence validation failure: Teams give AI the middle of the workflow because it’s recoverable — a human validates the output at the end and catches problems before real damage occurs. Handing over the full cycle means trusting the outcome, but most organizations haven’t built the measurement discipline to know when that trust is justified. They deploy the AI and measure in time savings, but they never run the harder experiment: tracking what the system’s success rate would actually be if it were allowed to complete the cycle unsupervised. Unresolved accountability: This one is more cultural than technical, but it’s a real blocker. When a human closes the loop, it’s clear who is responsible. But when a system closes it, that question gets harder — especially in any environment where decisions face external scrutiny, like an auditor, a regulator, or a customer dispute. Most organizations keep a human in the closing position by default, regardless of whether that person is actually adding judgment in the majority of cases. The teams that get real value out of AI in the next few years won’t be the ones who bolt AI onto every step they can find. They will be the ones who took the time to evaluate, workflow by workflow, whether they were building something that helps or something that finishes.

Reliable orchestration for small language models depends less on model sophistication than on the durability of event flow and state. Under the assumptions used here — small model instances, little or no local state, Kafka as the event backbone, Temporal as the orchestration layer and durable state store, and Java as the runtime — the safest design is to treat model invocations as replayable side effects, Kafka as the transport and ordering substrate, and Temporal Workflow state as the canonical record of conversational progress. In that design, Kafka provides high-throughput append-only event delivery and partition-local ordering, while Temporal persists Workflow Event History and can replay execution after failures. Exactly-once semantics remain meaningful inside Kafka’s consume-transform-produce boundary when transactions and read_committed are used, but once processing crosses into external systems such as model APIs, durable activities, or databases, correctness comes from idempotency, deduplication, sequence checks, and reconciliation rather than from a global exactly-once guarantee. Assumptions The most productive baseline is a narrow one. Each conversation, task, or model session is keyed so related events land on the same Kafka partition, preserving order only where order actually exists: within one partition, not across the topic. Each workflow instance owns one conversational state machine, stores the minimal context needed to decide the next action, and invokes model calls through Temporal Activities so failures, retries, and timeouts are visible and durable. Large prompts, attachments, or long transcripts are not kept as incidental JVM memory because Temporal persists inputs and outputs in Event History and large histories degrade replay latency; those artifacts belong in external storage with durable references held in workflow state. Analysis The central engineering mistake in LLM orchestration is to confuse transport delivery with business completion. Kafka can guarantee at-least-once delivery by processing records before committing consumer offsets, and it can provide exactly-once behavior for Kafka-to-Kafka pipelines by atomically updating produced records and consumed offsets with transactions. Kafka’s own design documentation is explicit that the producer is the transactional component and that read_committed is advisable when aiming for exactly-once processing. The same documentation also makes clear why the guarantee weakens at system boundaries: once consumed data must be coordinated with an external state store or side effect, the problem becomes cross-system consistency rather than log delivery. In a Temporal-based model pipeline, that means Kafka should usually be treated as the durable ingress path, while Temporal owns the authoritative notion of whether an event was applied to a conversation state machine. That separation suggests a simple rule. Offsets are transport progress; workflow state is semantic progress. A consumer should therefore commit offsets only after handoff to a durable semantic owner. In this architecture, that owner is the Temporal workflow receiving a signal. Temporal workflows behave like stateful services that receive Signals, Queries, and Updates, and the platform persists Event History so a crashed worker can replay the workflow and resume from the last recorded event. Signal handlers are allowed to mutate workflow state, and blocking coordination can be expressed safely with Workflow.await. Activity retries are configured through ActivityOptions and RetryOptions, with heartbeat support for long-running calls. Java @WorkflowInterface interface ModelFlow { @WorkflowMethod void run(String sessionId); @SignalMethod void onEvent(ModelEvent event); @QueryMethod long lastAppliedSequence(); } private final ModelActivities activities = Workflow.newActivityStub( ModelActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(20)) .setRetryOptions( RetryOptions.newBuilder() .setInitialInterval(Duration.ofMillis(250)) .setMaximumAttempts(5) .build()) .build()); private final NavigableMap<Long, ModelEvent> pending = new TreeMap<>(); private long nextSequence = 1; private ConversationState state = ConversationState.empty(); @Override public void onEvent(ModelEvent event) { pending.putIfAbsent(event.sequence(), event); } @Override public void run(String sessionId) { for (;;) { Workflow.await(() -> pending.containsKey(nextSequence) || state.closed()); if (state.closed()) break; var event = pending.remove(nextSequence); state = activities.applyEvent(state, event); nextSequence = event.sequence() + 1; } Workflow.await(Workflow::isEveryHandlerFinished); } @Override public long lastAppliedSequence() { return nextSequence - 1; } This workflow fragment does three important things at once. The @SignalMethod declares asynchronous event ingress, the @QueryMethod exposes durable progress for reconciliation, and the activity stub attaches retry policy directly to the state transition that may call a model endpoint or another dependency. The pending map is not a queue for throughput; it is a reordering guard. If Kafka redelivers a message or an upstream retry arrives out of sequence, putIfAbsent and the nextSequence gate prevent semantic duplication and preserve per-session causality. Finishing the run only after Workflow.isEveryHandlerFinished() avoids the Temporal-documented failure mode where a workflow completes or continues-as-new while a handler is still waiting on asynchronous work. The matching Kafka consumer must be deliberately conservative. Automatic commits are inappropriate because they advance transport progress in the background regardless of semantic application. Manual synchronous commits make the boundary explicit, and Kafka documents that committed offsets are the secure restart position, and that commitSync should write the next offset, meaning lastProcessedOffset + 1. The consumer is also not thread-safe, so per-partition in-order handling is easiest when one poll loop owns one consumer instance and performs durable handoff before commit. Java void pollLoop() { consumer.subscribe(List.of("model-events")); while (running.get()) { var records = consumer.poll(Duration.ofSeconds(1)); for (var partition : records.partitions()) { var batch = records.records(partition); for (var record : batch) { var eventId = header(record, "event-id"); if (!inbox.tryInsert(eventId, record.topic(), record.partition(), record.offset())) { continue; } var workflow = client.newWorkflowStub(ModelFlow.class, record.key()); workflow.onEvent(ModelEvent.from(record)); inbox.markApplied(eventId); } var nextOffset = batch.get(batch.size() - 1).offset() + 1; consumer.commitSync(Map.of(partition, new OffsetAndMetadata(nextOffset))); } if (inbox.backlog() > 50_000) consumer.pause(consumer.assignment()); else consumer.resume(consumer.assignment()); } } The durable inbox is the effective-once bridge. If the process crashes after signaling Temporal but before committing offsets, Kafka may redeliver, yet tryInsert suppresses reapplication. If upstream producers use Kafka transactions, the consumer should read with isolation.level=read_committed so aborted records stay invisible; Kafka’s configuration reference notes that read_committed returns only committed transactional messages and withholds records past the last stable offset while open transactions exist. Backpressure also belongs here. Kafka exposes pause and resume without forcing a group rebalance, and monitoring guidance explicitly recommends watching lag, fetch rate, poll timing, and commit latency to ensure consumers are keeping up. Context propagation is easiest when context is split into stable metadata and mutable conversational state. Stable identifiers such as trace ID, tenant, policy version, and conversation key belong in Kafka headers and Temporal headers so they survive hops across services and activities; Kafka’s ProducerRecord supports headers, and Temporal context propagators move custom key-value data across workflow, activity, and child-workflow boundaries. Mutable context, by contrast, should not live in worker memory or ad hoc caches. It belongs in the workflow state, often as a compact summary plus references to offloaded artifacts. Temporal’s documentation explicitly warns that all activity inputs and outputs are persisted, that long AI-style conversations grow history, and that large histories degrade workflow-task latency. For long-running sessions, Continue-As-New provides a checkpoint boundary with a fresh Event History while preserving the workflow identity chain. Reconciliation closes the last reliability gap. Even with careful commits, outages, manual replays, or producer bugs can create suspicion that a workflow missed an event. Temporal queries are read-only and must not mutate state or block, which makes them ideal for asking a workflow for its durable high-water mark and replaying any gap from the event store. Java void reconcile(String workflowId, long durableHighWatermark) { var workflow = client.newWorkflowStub(ModelFlow.class, workflowId); long applied = workflow.lastAppliedSequence(); eventStore.readRange(workflowId, applied + 1, durableHighWatermark) .forEach(workflow::onEvent); } This pattern works because the workflow does not trust delivery history alone; it trusts its own durable state. Observability then becomes the enforcement layer for those guarantees. Kafka should surface lag, request latency, retry rates, poll gaps, and buffer exhaustion, while Temporal should emit metrics through Micrometer, trace activity and workflow paths, and expose searchable workflow metadata through Search Attributes. Temporal also recommends monitoring replay latency because large histories, payload sizes, and cache churn drive recovery cost. Together, these signals reveal the difference between a system that is slow, a system that is duplicating work, and a system that is actually losing context. Conclusion Orchestrating small language models without losing events or context is fundamentally a durability problem, not a prompt-engineering problem. Kafka should be used for ordered transport and scalable ingestion, but semantic completion should be anchored in Temporal’s durable workflow state, where signals, sequence gates, retryable activities, queries, and replay make failures recoverable rather than ambiguous. Exactly-once remains valuable inside Kafka’s transactional envelope, yet end-to-end correctness across model calls and other side effects comes from explicit idempotency, deduplication, reconciliation, and bounded context management with external storage and continue-as-new. In a Java stack, that combination yields an architecture where duplicates become harmless, ordering becomes explicit, back pressure becomes controlled, and context survives crashes because it is recorded in durable state instead of being left in process memory.

By Akhil Madineni CORE

AWS can send audit logs to an attacker’s account unless denials are enforced at the network layer, while Azure doesn’t log network-block requests at all. The concept of a data perimeter was popularized by AWS [1] to establish organizational boundaries around identities, resources, and networks. In simple terms, AWS provides access controls to ensure that trusted identities access trusted resources from expected networks while blocking all outside access. This article explores how different cloud providers handle resource access logs and how it relates to data protection. It sets up an experiment where an outside identity with valid credentials accesses a trusted resource and is blocked by a policy in one of the scenarios. The experiment explains two scenarios that differ in where the deny decision is enforced. We find that the same request for resource access produces different log artifacts in AWS and Azure. AWS sends access logs containing caller-controlled metadata in both the identity and resource-owner accounts unless a network layer explicitly denies access. However, in Azure, resource access logs are only logged at the resource-owner’s subscription, and when access is blocked at the network layer, nothing is logged there either. Both behaviors have consequences for security teams collecting and analyzing audit logs. This article walks through both scenarios with lab experiments and reproducible code. Background AWS and Azure treat identities differently. In AWS, identities are not centralized into one single place — instead, they live at the account level. For example, if an organization contains 10 accounts, identities can be created in each of the 10 accounts. In comparison, in Azure, identities are centralized into one Entra ID tenant. Since a tenant is linked to multiple subscriptions containing the company’s resources, identities from the same tenant are configured to access resources inside subscriptions. In summary, the resource-owning entity in AWS (the account) also holds identities, whereas in Azure the resource-owning entity (the subscription) does not hold identities – those live in the Entra ID tenant. Secondly, AWS and Azure treat access logging differently. In AWS, CloudTrail logs API calls at the account level. For cross-account access, AWS lets customers configure CloudTrail such that when data events are enabled, the caller account and the resource-owning account get access events. For example, if an identity in Account-A accesses a resource in Account-B and gets denied, then the deny audit entry is logged in both Account-A and Account-B. This mirroring is what makes caller-controlled metadata visible to a malicious actor’s account [2]. In contrast, in Azure, resource access logs (for example, StorageBlobLogs) live in the storage account in the subscription, whereas identity logs (Entra ID) live with the tenant. These are separate systems with no automatic mirroring. This difference sets up why a correlation problem exists and why a network-layer block does not produce logs at the resource layer. Threat Model The threat model is as follows: an attacker brings their credentials inside a corporate network and accesses the company’s resource (like an S3 bucket). By doing this, the attacker tries to exfiltrate company data by encoding sensitive information in the HTTP user agent header, a caller-controlled field that appears in access logs. This allows data to leave the corporate environment in small chunks across multiple requests. The second threat is more nuanced. A security team that relies on resource-layer logs to detect unauthorized access attempts will miss requests that are blocked before reaching the resource. If the network drops the request silently, the resource (service) never logs it. An attacker who knows this can probe a corporate environment repeatedly without appearing in the audit trail that the security team is monitoring. Experiments AWS Experiment To set up this experiment, we have three accounts: a credential-owning account (identity), a VPC-owning account, and a resource-owning account. The identity is a Lambda function that tries to access an S3 bucket (resource). The Lambda function runs from a private subnet in a VPC and accesses the S3 bucket through an S3 VPC endpoint (AWS PrivateLink). All audit logs are sent to a third account – this is a typical Control Tower setup [3]. We test two scenarios: The bucket policy denies all untrusted identities — assume that the bucket policy denies access to our identity. However, the VPC endpoint policy allows all cross-account access. The bucket policy allows this untrusted identity. However, the VPC endpoint policy disallows cross-organization access. Scenario 1 When the request gets denied at S3, AWS CloudTrail generates a standard API event: JSON { "eventType": "AwsApiCall", "errorCode": "AccessDenied", "userAgent": "...", "requestParameters": {...}, "tlsDetails": {...} } The full log is in https://github.com/sureshgururajan/aws-data-exfiltration-demo/blob/main/testing-results/scenario1-log.md. In this case, the full request context is preserved. This includes: userAgent requestParameters TLS metadata Additional request context The main observation is that this event includes caller-controlled metadata in the userAgent field. Since customers can configure CloudTrail to log data events on both the caller account and the resource account, a malicious actor gets the same denial event in their account. Therefore, an attacker in an untrusted account can exfiltrate company data into their accounts by triggering these denied access requests on the company resource. Scenario 2 In the second scenario, if the VPC endpoint policy denies cross-account access (example), CloudTrail generates a different event: JSON { "eventType": "AwsVpceEvent", "eventCategory": "NetworkActivity", "errorCode": "VpceAccessDenied", ... } See the full log here. Instead of logging an AwsApiCall event, CloudTrail logs NetworkActivity with the errorCode: VpceAccessDenied and does not log the HTTP user agent header. More importantly, this event is not sent to the malicious actor or the resource owner’s account. Rather, the event is sent to the VPC endpoint owner’s account. In other words, the cause of the denial was a VPC endpoint policy, and therefore CloudTrail generates a NetworkActivity event rather than the API event and routes it to the VPC-owning account. This prevents the bad actor from stealing company data via CloudTrail. Azure Experiment To set up this experiment, we created two Azure subscriptions – one for identity and the other for the resource. An Azure function in subscription-A writes to a blob storage in subscription-B. The Azure function is registered as a system-assigned managed identity in the Entra ID tenant while turning off the shared access key for the blob storage to ensure only managed identities can access it [5]. The function uses DefaultAzureCredential to request a token from Entra ID and attempts to write to a file in the storage account. Since both subscriptions trust the same Entra ID tenant, the identity moves across subscriptions natively without needing an AssumeRole step. Like before, we run through two scenarios: Azure function has the Storage Blob Data Contributor role and the network path is open The Azure function attempts to write to the storage account but is blocked by the firewall. Scenario 1 When the request is allowed at the blob storage, the following logs are written: The Entra ID tenant gets a token request log when the Azure function uses default Azure credentials. This event does NOT contain any information about the actual API action being taken. The resource account StorageBlobLogs records a PutBlob event with the file name and IP address but doesn’t show the name of the managed identity. Sample log entry from StorageBlobLogs Plain Text TimeGenerated [UTC] - 2026-05-02T19:30:32.7306109Z OperationName - PutBlob CallerIpAddress - 172.24.1.71:9156 Uri - https://sgrstorageaccountinsubb.blob.core.windows.net:443/storage-container/test.json AuthenticationType - OAuth RequesterObjectId - 00daa177-96c6-4b29-9a5c-53ca603565e9 StatusCode – 201 UserAgentHeader - azsdk-js-azure-storage-blob/12.31.0 core-rest-pipeline/1.22.3 Node/22.22.2 (Linux 6.6.130.1-3.azl3; x64) The requester object ID field indicates which identity made the request but doesn’t reveal more details as to the identity itself. That part is left to the Entra ID logs as shown below. However, we can see that the userAgentHeader is logged. The difference with AWS is that in Azure, the StorageBlob log entry is not mirrored to Entra ID, i.e., the caller’s subscription. In Azure, it stays only in the resource owner’s subscription. Entra ID contains just the token issuance log: Sample log entry from Entra ID Plain Text Date (UTC),2026-05-02T19:30:32Z Request ID,25c5f7f7-4206-448d-817b-730744991701 Correlation ID,73cf7b90-c49b-40f0-800d-74e77e40717c Service principal ID,00daa177-96c6-4b29-9a5c-53ca603565e9 Service principal name,SureshTestingMultiCloud-Function Credential key ID, Credential thumbprint, Application,SureshTestingMultiCloud-Function Application ID ,57650788-dae5-416f-9da8-792b4ebbbb29 App owner tenant ID, Resource,Azure Storage Resource ID ,e406a681-f3d4-42a8-90b6-c2b029497af1 Resource tenant ID, Resource owner tenant ID,f8cdef31-a31e-4b4a-93e4-5f571e91255a Home tenant ID, Home tenant name, IP address, Location,", , " Status,Success Sign-in error code, Failure reason,Other. Conditional Access,Not Applied Scenario 2 In this scenario, we introduced a network-level block using the Storage Account Firewall while keeping the permissions intact. Entra ID logs still show a successful token issuance because the identity is valid and the scope is broad. However, the storage resource logs don’t log the request. Since the connection was dropped at the network layer before reaching the storage service plane, there is no “Access denied” event in the resource’s audit log. Sample log entry from Entra ID Plain Text Date (UTC): 2026-05-02T19:35:10Z Service principal name: SureshTestingMultiCloud-Function Application: SureshTestingMultiCloud-Function Resource: Azure Storage Status: Success Sample log entry from StorageBlobLogs 0 results for the KQL query: SQL // Query to check for any recorded activity after the network block StorageBlobLogs | where TimeGenerated > ago(1h) | where RequesterObjectId == "00daa177-96c6-4b29-9a5c-53ca603565e9" | project TimeGenerated, OperationName, StatusCode, StatusText, CallerIpAddress, Uri | sort by TimeGenerated desc This result shows that a network-level block is not visible in the resource layer. The Azure administrator sees a successful token issuance in Entra ID but nothing in StorageBlobLogs. To detect this, security teams need to go beyond resource-layer logs and enable additional logging layers such as NSG Flow logs or Defender for Storage - these are outside the scope of this experiment. Comparison scenarioawsazure Identity model Account-scoped Tenant scoped Who gets audit logs? (when available and enabled) Caller-side and resource-owner side (Scenario 1 only) Resource-owner side only Where are the audit trails located? CloudTrail is the logging service. CloudTrail logs are distributed across Caller account, the resource account, and the VPC-owning account Token issuance logs are in the Tenant (Entra ID) while resource access logs are in the Subscription Caller-controlled metadata visible? Yes, visible in caller account and resource account Yes, but included in resource account only What a network-layer block produces When using VPC endpoint policy, AwsVpceEvent is produced and is routed to the VPC-owner account. No logs in resource-owner account. No resource-layer log entry. Identity context in resource logs Full caller identity context included Only the caller ID in the form of RequesterObjectId. An operator must correlate this ID with service principal ID in Entra ID logs. Mitigation We saw that in AWS, CloudTrail can be configured to send log events on both the caller account and the resource account. An attacker can use this information to silently exfiltrate small amounts of data at a time. To mitigate this attack vector, an organization must: Run their compute services in an Amazon VPC — preferably in a private subnet, and Use VPC endpoints with endpoint policies [4] to access their AWS resources for the compute services. The endpoint policies must allow trusted identities to access the resource while blocking everything else. AWS already documents these controls in [1], but these experiments show how important it is to enforce these controls. This is in addition to all the controls that an organization already uses, such as Service Control Policies and Resource Control Policies — those policies control the maximum permissible action that can be taken by an identity/resource but do not control the CloudTrail logging behavior. While Azure doesn’t have the above attack vector specifically, it has a different problem — an operator must manually correlate Entra ID events with the resource event. An example would be an “identity journey” like — managed identity (like the Azure function) requests a token, then writes to a storage account. Therefore, some tooling must be built to correlate such events — for example, routing both ManagedIdentitySignInLogs and StorageBlobLogs into a single Log Analytics workspace is a minimum. Additionally, logs must be captured at different layers such as NSG flow logs/Defender for Storage that can provide anomaly detection beyond standard diagnostic logs. Conclusion In this article, we demonstrated how the same access request produces different results in AWS and Azure. In AWS, access logs were sent to the resource account or the VPC account depending on where the deny decision was enforced, while in Azure, access logs were only sent to the resource account. We saw that this difference comes from how each cloud provider fundamentally treats identities and resources. The implications of the experiment are that security teams in multi-cloud environments cannot assume that audit coverage works the same way across providers. Each provider models their identities and provides different data perimeter controls. Before designing data perimeter controls, security teams must understand each provider’s logging architecture and its differences. References [1] https://aws.amazon.com/identity/data-perimeters-blog-post-series/ [2] https://systemweakness.com/a-subtle-audit-log-consideration-in-aws-063752150b20 [3] https://docs.aws.amazon.com/controltower/latest/userguide/what-shared.html [4] https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-access.html [5] https://learn.microsoft.com/en-us/azure/storage/common/shared-key-authorization-prevent?tabs=portal

This guide explains zone-aware routing from a Kubernetes-first point of view. It covers: why zones matter in cloud platformswhich topology labels Kubernetes places on nodeshow Kubernetes first tried to solve locality through Servicewhat gaps remained after those Service-based featureshow Gateway API implementations such as Envoy Gateway and kgateway built on top of that foundation Why Zones Matter In cloud platforms, a zone is a logical failure domain inside a region. Zones usually have low-latency networking within the zone, but crossing zones can increase both latency and cost. That cost is not theoretical. AWS documents that traffic within the same Availability Zone is free, while traffic that crosses Availability Zones typically incurs data transfer charges, and cross-zone transfer is generally billed in both directions, so a single round trip can be charged twice. See: AWS Architecture Blog: Overview of Data Transfer Costs for Common ArchitecturesAmazon EC2 pricing: Data Transfer This is one reason distributed systems try to keep traffic local when they can, while still preserving failover to other zones. The Topology Information Kubernetes Already Has Kubernetes did not start by inventing zone-aware traffic policies. It started by carrying topology information on nodes. The two most important well-known labels are: topology.kubernetes.io/regiontopology.kubernetes.io/zone According to the Kubernetes reference, these labels are populated on Node objects by the kubelet or the external cloud-controller-manager when the cluster is integrated with a cloud provider. In non-cloud environments, operators can set them manually if the topology model still makes sense. Reference: Kubernetes well-known labels: topology.kubernetes.io/zone In managed clusters, these labels are commonly present by default. Here is the kind of node data Kubernetes typically exposes: YAML apiVersion: v1 kind: Node metadata: name: ip-10-0-12-34.ec2.internal labels: kubernetes.io/hostname: ip-10-0-12-34.ec2.internal topology.kubernetes.io/region: us-east-1 topology.kubernetes.io/zone: us-east-1a That topology data is useful for scheduling, spreading replicas, volume placement, and eventually traffic routing. The Original Service Model The original Kubernetes Service abstraction solved a different problem first: stable discovery and virtual IPs for ephemeral Pods. At the beginning, the model was simple: a Service selected a set of Podskube-proxy programmed forwarding rulestraffic could be sent to any healthy endpoint behind the Service That was excellent for reachability and abstraction, but it had no built-in notion of zone locality. The gap was straightforward: the Service abstraction knew which endpoints existed, but not that a client in zone-a should usually prefer endpoints in zone-a. Kubernetes' First Attempts to Improve Locality Through Services Kubernetes gradually added locality-aware behavior on top of Service, mostly by improving how endpoint selection works. Internal Traffic Policy One early mechanism was internalTrafficPolicy: Local. This tells kube-proxy to use only node-local endpoints for cluster-internal traffic. Example: YAML apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app ports: - port: 80 targetPort: 8080 internalTrafficPolicy: Local Reference: Kubernetes Service Internal Traffic Policy This helps with node locality, but it is not zone-aware routing. Its limitations are important: it is node-local, not zone-localif a node has no local endpoint, the Service behaves as if it has zero endpoints from that node's perspectiveit is too strict for many multi-zone workloads that want zonal preference, not node affinity So this was useful, but it did not really solve multi-zone locality. Topology Aware Routing With Services Kubernetes next introduced Topology Aware Hints, now called Topology Aware Routing. This works through two components: The EndpointSlice controller looks at endpoint and node topology.kube-proxy consumes hints from EndpointSlices and prefers endpoints closer to the client zone. Historically, the Service-side configuration was commonly exposed through the service.kubernetes.io/topology-mode: Auto annotation: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: backend ports: - port: 80 targetPort: 8080 Conceptually, the flow looks like this: This was Kubernetes' first real zone-aware answer at the Service layer. It is useful historical context, but it is no longer the clearest Service-level API to emphasize for new users. Traffic Distribution Preferences Kubernetes later added trafficDistribution as a clearer way to express routing preferences. In current Kubernetes documentation, the relevant zone-level preference is: PreferSameZone The older PreferClose name is documented as deprecated in favor of PreferSameZone, though you may still see PreferClose in some provider and implementation docs that have not yet caught up. Example: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend spec: selector: app: backend ports: - port: 80 targetPort: 8080 trafficDistribution: PreferSameZone Reference: Kubernetes Service trafficDistribution This is a better API shape than older annotations because it is explicit in the Service spec and described as a preference rather than a strict guarantee. In practice, that means current Kubernetes guidance emphasizes trafficDistribution: PreferSameZone, while the older topology-mode: Auto path is best understood as part of the feature's evolution. What Gap Remained After Service-Based Locality Kubernetes Services improved a lot, but they still left several gaps. The Behavior Is Best Effort Topology-aware routing is not a hard guarantee. Kubernetes documents multiple safeguard cases where the system falls back to cluster-wide routing. Examples include: too few endpointsimpossible balanced allocationmissing topology labels on one or more nodesmissing hints for one or more endpointsno hinted endpoint for the local zone That is correct for safety, but it means the behavior is heuristic and conditional. It Assumes a Certain Traffic Shape Kubernetes explicitly documents that Topology Aware Routing works best when traffic is roughly evenly distributed and when there are enough endpoints per zone. If most traffic originates from one zone, local subsets can overload while the global service still looks healthy. It Is Scoped to the Service Datapath This is the most important architectural gap. Service-level topology features influence how kube-proxy chooses endpoints for Service traffic. They do not automatically solve every higher-level data plane. In particular, they do not by themselves define: how an L7 gateway proxy should understand its own zonehow an Envoy-based gateway should configure locality-aware upstream load balancinghow a gateway controller should express stricter local preference versus simple best-effort localityhow policy should attach to particular routes, gateways, or backends That left room for Gateway API implementations to expose richer locality controls. Why Gateway API Implementations Stepped In Gateway API is intentionally expressive and extensible. It standardizes core routing objects, but implementations often add policy CRDs to expose features that are specific to their data plane. That distinction matters here: Gateway API itself does not define one universal, cross-implementation zone-aware policy. Instead, it gives implementations room to expose locality behavior in a way that matches their proxy and control-plane design. Reference: Gateway API overview This is where zone-aware routing became more explicit at the gateway layer. Instead of relying only on kube-proxy's Service behavior, gateway implementations can: understand the proxy's own localityread backend endpoint localityconfigure the underlying proxy's load balancer directlyexpose locality policies as route or backend-attached configuration Example of How Envoy Gateway Addresses the Gap Envoy Gateway supports two paths: Reusing Kubernetes Service-level locality such as Topology Aware Routing or trafficDistributionConfiguring zone awareness directly through BackendTrafficPolicy Reference: Envoy Gateway zone-aware routingEnvoy zone-aware routing Example BackendTrafficPolicy: YAML apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: zone-aware-routing spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: zone-aware-routing loadBalancer: type: RoundRobin zoneAware: preferLocal: minEndpointsThreshold: 1 force: minEndpointsInZoneThreshold: 1 That is a meaningful step beyond plain Service because the gateway layer is now explicitly participating in locality-aware upstream balancing. Example of How kgateway Addresses the Gap kgateway takes a similar approach in spirit: proxy locality is made explicit, and backend load-balancing behavior is configured through policy rather than relying only on Service heuristics. At a high level, kgateway combines: Gateway proxy locality configurationBackend-attached load-balancing policyNative Envoy locality-aware upstream load balancingEndpoint locality metadata that Envoy can use directly Architectural Summary The progression looks like this: Kubernetes Service solved stable discovery and reachability.internalTrafficPolicy improved node-local routing, but not zonal routing.Topology Aware Routing and trafficDistribution added zone-aware preferences to the Service datapath.Gateway API implementations extended the model so L7 gateways and proxies could make explicit locality-aware decisions themselves. Practical Takeaways Kubernetes already provides the topology metadata needed for zone-aware decisions.Service-native locality is useful, but it is heuristic and scoped to the Service datapath.Zone-aware traffic for gateways usually needs the gateway implementation to understand locality too.Modern Gateway API implementations fill that gap by attaching locality-aware load-balancing policy closer to the L7 data plane. Where Zone-Aware Routing Matters in Practice Zone-aware routing usually becomes worth the added operational attention when one or both of these are true: The workload has a tight latency budget, especially at p95 or p99The system moves enough east-west traffic that even a small per-GB cross-zone charge becomes material Common examples include: Gaming platforms, where matchmaking, player session state, inventory, and real-time coordination are sensitive to a few extra milliseconds of network delayFinancial services, where payment, quote, fraud, or checkout paths care more about predictable tail latency than average latencyLarge SaaS and enterprise control planes, where a gateway fans out to many internal APIs and the aggregate cross-zone traffic becomes a real monthly costAI inference, media delivery, logging, and telemetry pipelines, where payload sizes are large enough that bandwidth cost matters even when latency is less critical Worked Example: Multiplayer Gaming Backend Suppose a regional game API runs gateway proxies and backend pods in three zones. Players connect to a gateway in zone-a, and that gateway calls a player-state service that is also deployed in zone-a, zone-b, and zone-c. Assume the following: 25,000 requests per second reach the player-state service from zone-athe combined request and response payload is about 40 KiB per callcross-zone traffic is billed at a representative $0.01 per GBwithout zone awareness, only about one third of those calls stay in zone-a, while the other two thirds go to zone-b or zone-c Actual billing varies by provider, region, and direction of transfer, but the point of the example is that a seemingly small per-GB rate compounds quickly on hot service paths. That means the traffic volume from zone-a to the player-state service is about: 25,000 x 40 KiB per second, or roughly 1 GB/s totalif two thirds of that traffic crosses zones, that is about 0.67 GB/s of cross-zone trafficover a 30-day month, that is about 1.7 million GBat $0.01 per GB, that is about $17,000 per month in cross-zone transfer for just that one service path That is the cost side. The latency side can matter even more for the player experience. If each cross-zone hop adds only 1-3 ms, a request path that fans out to several internal services can add multiple milliseconds of extra tail latency. For a gaming workload, that can affect: matchmaking responsivenesssession join timethe smoothness of player state or presence updateshow stable the system feels during traffic spikes and retries This is why zone-aware routing is not only a cost optimization. In some industries, it is a user-experience and SLO control. Worked Example: Large SaaS Control Plane The same logic applies outside gaming. Consider a large enterprise SaaS platform where each incoming API request hits a gateway and then fans out to an auth service, tenant metadata service, feature-flag service, and audit pipeline. Even if each individual backend call is small, the gateway can generate a large amount of aggregate east-west traffic. In that kind of system, zone-aware routing helps in two ways: it removes avoidable cross-zone traffic from the steady-state hot pathit reduces the chance that a multi-hop request burns several extra milliseconds just on internal network distance For that kind of platform, the business case is usually a combination of lower regional data-transfer cost, tighter latency distributions, and better failure-domain alignment. Conclusion Zone-aware routing is the story of a single idea moving down the stack. Kubernetes started with topology labels on nodes, then taught the Service datapath to prefer local endpoints through internalTrafficPolicy, Topology Aware Routing, and trafficDistribution. Those features are valuable, but they are best-effort and they stop at the Service boundary, which leaves L7 gateways unable to reason about their own locality. Gateway API implementations such as Envoy Gateway and kgateway pick the idea up from there, making proxy locality explicit and pushing locality-aware load balancing into Envoy where it can act on real endpoint metadata. The practical guidance is short. Start with the Service-native controls, because they are simple and often enough. Reach for gateway-level locality policy when you have a tight tail-latency budget, or enough east-west traffic that cross-zone transfer becomes a line item you can see. In both cases, the goal is the same: keep traffic local when you safely can, and fail across zones when you must. Further Reading Kubernetes ServiceKubernetes Topology Aware RoutingKubernetes Service Internal Traffic PolicyKubernetes well-known topology labelsGateway API overviewAWS Architecture Blog: Data transfer costs