Deployment

Ground Truth for AI-Written Code Session capture, per-line attribution, and selection-bias-free agent benchmarks, on top of the Git host you already use. A technical overview for engineers and engineering leaders evaluating how much of their codebase is now written by AI agents - and who is accountable for it. 1. The Problem: Git Blame No Longer Tells the Truth On most teams, AI agents now write a large share of new code. But the tools that record who wrote what were built for humans. When an agent edits files in your working tree and you commit them, git blame attributes every one of those lines to you. The prompt that produced them, the model that ran, the cost, the number of turns, and whether the code survived the next sprint — none of it is recorded anywhere. That gap has real consequences: Provenance – no answer to “which agent, from which prompt, wrote this line?” during review or an incident.Cost and efficiency – no ground truth on what a feature cost in tokens and dollars, or which agent got therein fewer turns.Quality – no measure of whether agent-written code survives, or gets reworked and reverted days later.Comparison – “which agent is better for us?” answered by vibes, because every naive comparison is poisoned by selection bias (the hard tasks go to the agent you already trust). Origin closes that gap. It captures the full agent session — prompt, diff, tokens, cost, tools, duration — attributes every surviving line back to an agent and a prompt using Git as the source of truth, and turns that data into honest, selection-bias-free comparisons between agents. It runs on top of GitHub or GitLab; there is nothing to migrate. 2. How Origin Captures an Agent Session Capture is deliberately boring and durable. A one-time origin enable registers the machine, auto-detects installed agents (Claude Code, Codex, Cursor, GitHub Copilot, Gemini, Aider, Devin, Antigravity, and more), and installs two kinds of listeners: Agent hooks – Origin hooks fire on the agent’s lifecycle events (session start, each user prompt, eachtool/file edit, and stop/end). They record the prompt text, the per-turn file diff, token and cost counters, tool calls, and the model.Transcript watchers – for agents that keep a durable on-disk transcript (e.g., Codex’s rollout logs, Devin’s local session DB), Origin reads that record directly instead of depending on hooks. The principle: if there is an authoritative transcript, read it; hooks are for context and policy. Capture is resilient by design. It writes locally first, retries on a durable queue when the network is down, resolves session end from heartbeat liveness rather than a fragile inactivity timer, and is aware of Git work trees so parallel sessions don’t collide. Sessions that never produced real work are swept so counts reflect reality. Figure 1. Every AI coding session Origin captured — agent, model, cost, tokens, branch, and review status. This is the raw material everything else is built on. 3. Attribution: First-Author Wins, With Git as the Source of Truth Recording a session is easy; attributing lines correctly is the hard part, and it is where Origin is opinionated. The model is first-author-wins: a line is credited to whoever introduced it, and later edits never reclaim it. For pushed commits, Git is the ground truth — Origin reconciles its capture against the committed diff rather than trusting a possibly lossy hook stream. A suite of invariants guards the accounting so numbers never drift: InvariantWhat it guaranteesFirst-author-winsA line counts once, for its original author - no double-credit when it’s later touched.Git-truth reconciliationPushed-commit line counts come from the real diff, not the (lossy) live hookstream.Hunk-aware countingAdd/remove tallies parse diff hunks correctly; content lines aren’t miscounted.Writes never claim linesA write/format/no-op operation cannot claim authorship it didn’t earn.Missing-commit self-healA commit-and-exit race is reconstructed at read time from the transcript-attested SHA. Figure 2. One session, decomposed: each prompt and its diff (committed vs uncommitted), the linked commit, and a 100%-AI verdict - the ground truth per-line blame is built from. The AI Blame tab drills to the line level. 4. Prompt-Level Time Travel Because Origin records the state before every prompt, each prompt becomes a restore point. You can undo an agent’s changes — the files revert — without rewriting or losing your commits. 5. Honest Benchmarking: The Agent Scorecard Once sessions are captured and attributed, Origin computes a per-agent scorecard — efficiency, outcome, and survival — for your real work. The point of difference is honesty: the scorecard refuses to draw conclusions the data can’t support. MetricDefinitionCost/taskMean cost per completed session for the agent.Tokens/produced lineToken spend normalized to lines that actually shipped.Median turnsHow many prompts it took to finish - lower is tighter.First-pass approvalShare of reviewed sessions approved without changes.Cost/merged PRDollars per PR that actually merged (outcome, not activity).Code survival @ 7/30dFraction of authored lines still present a week/month later.Rework rateThe inverse - how much of the agent’s output got reverted or rewritten. The guardrails matter as much as the metrics: Minimum sample size – agents aren’t ranked on a handful of sessions; below a threshold, a metric is shown as “not enough data,” not a misleading average.Confidence intervals – ratio metrics (e.g., tokens-per-line) carry a CI, so a noisy small sample can’t masquerade as a clear winner.Estimated tokens excluded – sessions whose token counts were estimated rather than reported are flagged and kept out of the money math.Line-weighted authorship – the AI-vs-human percentage is weighted by lines, not session count, so one giant human commit doesn’t get outvoted by many tiny agent ones. Figure 3. The agent scorecard — cost, tokens-per-line, median turns, approval, and survival per agent, with sample-size and confidence guardrails. 6. Bake-Offs: The Selection-Bias-Free Comparison The scorecard measures agents on the work you happened to give them — and you give the hard tasks to the agent you trust, which skews every comparison. A bake-off removes that bias by construction: it runs the same prompt through N agents, each in its own isolated Git work tree, and lets you compare the results side by side. Every arm gets identical work. Architecture: The Server Schedules, Your Machine Executes Coding agents run on your machine, with your keys — Origin’s cloud can never run them. So a bake-off is split cleanly in two: the server owns the queue and the schedule; a local runner daemon owns execution. Each arm branches from HEAD into bakeoff/<id>/<agent>, the agent works autonomously and commits, and Origin correlates the result back to the branch via normal session capture — nothing extra to wire up. The list nests each arm’s session inline (cost, tokens, lines, status), filters by status/repo/agent, pages ten at a time, and rolls up a head-to-head agent comparison across every bake-off you’ve run. Deliberately, Origin does not auto-declare a winner. It tints the cheapest and fewest-turns arms to help you scan, but “cheapest” and “best” are not the same thing - only a human reading the diff can decide. You pick the winner.Figure 4. Composing a bake-off — the same prompt, two or more agents, each running autonomously in its own git work tree. Results stream back as sessions and roll up into a head-to-head comparison. 7. For Teams: Governance Without a Second Source of Truth Everything above is per-developer value that also aggregates for a team. On top of it, Origin adds an org layer: typed policies enforced across review, PR checks, and CI; AI auto-review of agent sessions; secret and PII scanning on captured diffs; budgets and cost controls with per-agent visibility; role-based access; and an organization dashboard that shows what share of the codebase is AI-authored, by whom, at what cost - line-weighted, not guessed. Because attribution is per line and travels with the repo (prompts are carried in Git notes, and a dedicated sessions branch makes context portable across clones), the governance view is derived from the same ground truth developers see — not a parallel system that drifts. 8. Architecture, Privacy, and Getting Started Local-first capture – session data is recorded on your machine first. A fully standalone mode keeps everything in the repo with no account at all.Sits on your host – GitHub and GitLab, multiple connected accounts, native Windows/macOS/Linux CLI.No repo migration.Portable provenance – prompts live in Git notes; the origin-sessions branch is a zero-tooling vehicle, so a fresh clone still has the history.CLI-native – the CLI is a single Node binary distributed via signed GitHub releases; the platform API runs on a small, boring stack (Express + Prisma). Origin turns the invisible half of your codebase — the half an agent wrote — into something you can read, attribute, price, and compare. Solo, it’s your provenance and undo button. For a team, it’s the ground truth under every AI-code decision.

Most engineering teams working on healthtech applications reach a point where someone asks a question that sounds simple but isn't: How do we make sure a developer testing a new feature can't accidentally access production patient data? The answer determines whether the architecture that follows will be auditable or not. Teams that answer it with process — "we have policies about that" — spend the next 18 months patching access-control gaps that reopen every time a new engineer joins or a new service gets wired in. Teams that answer it architecturally spend a week setting up AWS Organizations correctly and then largely stop thinking about it. This article covers the multi-account architecture pattern for HIPAA-compliant infrastructure — specifically, the account structure decisions that either enforce PHI workload isolation or make it a permanent source of audit findings. Why Single-Account PHI Isolation Fails at the Seams A single AWS account running production, staging, and development workloads creates a specific problem that IAM policies alone cannot fully solve. The issue is not that IAM is insufficient as a technology. IAM policies enforced within an account are only as reliable as the discipline of the people who manage them. A policy that restricts a developer's access to production RDS today can be modified tomorrow by anyone with sufficient IAM permissions. Nothing in the account structure itself prevents the boundary from being crossed. In practice, the gaps show up in predictable ways. A pipeline service role gets broad permissions during a sprint because scoping them properly would have taken an extra hour. An engineer copies an IAM role from staging to production because it was faster than creating a new one. A debugging session in production happens under an account that was supposed to be read-only. None of these are malicious decisions. They are the natural result of putting access control boundaries inside an environment where the people who need to cross them also have the permissions to do so. The access control problem that surfaces during security reviews is almost always this one — not a missing encryption setting or an unpatched vulnerability, but access boundaries that exist on paper and drift in practice. The Multi-Account Model: Enforcement at the Boundary AWS Organizations with a properly structured multi-account hierarchy solves this problem by moving the enforcement point outside the accounts being protected. The boundary is no longer an IAM policy that someone with IAM permissions can modify. It is an account boundary that the engineers inside those accounts cannot cross, enforced by Service Control Policies applied at the organizational unit level. The recommended structure has four organizational units under the root: a Security OU containing a Log Archive account and a Security Tooling account, a Production OU containing only the Production account where PHI workloads run, a Non-Production OU containing Staging and Development accounts, and a Shared Services OU containing the account used for CI/CD pipelines, DNS, and shared tooling. The Production OU sits under its own organizational unit with SCPs that restrict what can happen inside it, regardless of what IAM policies exist within the production account itself. An engineer whose IAM role in the development account grants broad permissions has those permissions scoped to the development account. Crossing into production requires a separate role, in a separate account, with a separate set of credentials. The architectural boundary is the enforcement mechanism, not the IAM policy. The Log Archive account under the Security OU serves a specific purpose: it is the only account to which CloudTrail logs from all other accounts are delivered, and it is an account to which production engineers have no write access. This means the evidence trail for PHI access events cannot be modified by the accounts generating those events - which is exactly what auditors verify when they ask about log integrity. Service Control Policies: What to Enforce at the OU Level SCPs applied to the Production OU are where the architectural enforcement becomes concrete. The first policy prevents anyone inside the production account from disabling CloudTrail, including account administrators: JSON { "Effect": "Deny", "Action": [ "cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "cloudtrail:UpdateTrail" ], "Resource": "*" } CloudTrail continuity across the full audit period is not something that should depend on engineering discipline. It should be architecturally enforced. An account that can leave the organization can escape every SCP applied to it. This policy closes that path: JSON { "Effect": "Deny", "Action": "organizations:LeaveOrganization", "Resource": "*" } PHI that moves outside defined regions may fall outside data residency commitments. This policy locks the production account to specific regions: JSON { "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "StringNotEquals": { "aws:RequestedRegion": ["us-east-1", "eu-west-1"] } }, "NotAction": [ "iam:*", "organizations:*", "route53:*", "budgets:*", "waf:*", "cloudfront:*", "globalaccelerator:*", "importexport:*", "support:*", "trustedadvisor:*" ] } EBS encryption is not enforced by default in all account configurations. This policy makes an unencrypted volume impossible to create in the production account: JSON { "Effect": "Deny", "Action": "ec2:RunInstances", "Resource": "arn:aws:ec2:*:*:volume/*", "Condition": { "Bool": { "ec2:Encrypted": "false" } } } Cross-Account Access: The Pattern That Doesn't Create New Gaps Multi-account architecture introduces a problem engineers feel immediately: how does anything talk to anything else? A CI/CD pipeline in the Shared Services account needs to deploy to production. A developer needs read access to production logs during an incident. A monitoring service needs metrics from all accounts. The answer is cross-account IAM roles with tightly scoped trust policies. A role created in the production account with minimum required permissions defines a trust policy that allows only specific principals from specific accounts to assume it, and only under specific conditions like MFA or an external ID: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::SHARED-SERVICES-ACCOUNT-ID:role/DeploymentRole" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "deployment-pipeline-prod" } } } ] } The deployment role in the Shared Services account can assume the deployment role in production - but only that role, only from that account, and only with the correct external ID. A developer's personal IAM credentials cannot assume it. An engineer who compromises the development account cannot use that foothold to pivot into production. This pattern creates cross-account access without creating a backdoor through the account boundary. The boundary holds because the trust relationship is explicit, narrow, and auditable through CloudTrail - every role assumption generates a log entry in both accounts. What This Architecture Makes Provable The operational argument for multi-account PHI isolation often focuses on security. The architectural argument that matters more for engineering teams dealing with audits and enterprise security reviews is about provability. In a single-account setup, proving that a developer did not touch production PHI during a given period requires auditing IAM policies, CloudTrail logs, and access history, and then arguing that the policies were correctly configured and consistently enforced throughout the period. There is always a gap between what the policy said and what actually happened, and that gap is what auditors probe. In a multi-account setup, the same question has a simpler answer. The developer's credentials are scoped to the development account. The development account has no access to the production account's resources. Access to production PHI requires a separate role assumption that is logged, requires separate credentials, and would appear immediately in CloudTrail. You are not arguing that the configuration was correct. You are pointing to an architectural boundary that makes the question moot. This shift from arguable to verifiable is what separates teams that sail through security reviews from teams that spend three weeks responding to follow-up questions. The Operational Overhead Is Smaller Than It Looks The most common objection to multi-account architecture from engineering teams is overhead. More accounts means more IAM configuration, more billing to reconcile, more consoles to log into. In practice, this friction is front-loaded and largely disappears once the structure is in place. AWS Control Tower reduces the account provisioning overhead significantly - new accounts inherit the correct SCP structure, logging configuration, and security baseline automatically. Account Vending Machine patterns built on top of Service Catalog or Terraform can provision a correctly configured new account in minutes. After the initial setup, adding a new account is not significantly more work than adding a new VPC. The billing concern is resolved through AWS Organizations consolidated billing, where all accounts roll up to a single payment method with unified cost visibility. The console switching concern is resolved through IAM Identity Center, which provides a single sign-on entry point across all accounts in the organization. The overhead that remains is real but small. The alternative - treating IAM policies inside a single account as the primary PHI protection mechanism - creates ongoing operational overhead that grows with the team and never fully goes away. Final Thoughts PHI workload isolation is an architectural problem, not a policy problem. IAM policies enforced inside an account are only as reliable as the operational discipline of the team maintaining them. Account boundaries enforced by SCPs at the organizational level are reliable by construction — they hold regardless of what happens inside the accounts they protect. The multi-account structure described here is not a compliance checkbox. It is the architecture that makes the access control claims in a security review actually true rather than approximately true with caveats. When an auditor asks how you prevent developer access to production PHI, the strongest answer available on AWS is an account boundary that the developer's credentials cannot cross. Building that boundary is a week of work. Not building it is a permanent source of audit findings.

Every few weeks, someone on my team, or in a client meeting, asks me the same question: "Which cloud should we use for our AI workloads?" I have been building enterprise integrations for over fourteen years now, and lately most of my time goes into RAG pipelines, vector databases, and agentic orchestration on top of these platforms. So I get this question a lot, and honestly, there is no single right answer. The right cloud depends on where your data already lives, what your compliance team will accept, and which models your architecture actually needs. In this article, I want to walk through the three big players, AWS Bedrock, Google Vertex AI, and Microsoft Azure AI Foundry, and share what I have learned working with these platforms in real enterprise settings, not just from reading marketing pages. AWS Bedrock Bedrock started as a model marketplace back in 2023, and it has grown into a full platform with Guardrails for content filtering, Knowledge Bases for RAG, and AgentCore for building agentic workflows. What I like most about Bedrock is the sheer breadth of models available behind a single API. You get Claude from Anthropic, Llama from Meta, Mistral, Cohere's Command models, and Amazon's own Nova family, all through one consistent interface. If your architecture needs to swap models without rewriting your integration layer, Bedrock makes that easier than the other two. Pros: Broadest model catalog of the three, so you are not locked into one vendor's models.Strong identity and governance story if you are already running on AWS, since it plugs directly into IAM, CloudTrail, and Macie.Bedrock is one of the few places where you get Claude with enterprise indemnification, which matters a lot when legal teams get involved.Provisioned throughput options give you predictable latency for production workloads that cannot tolerate spikes. Cons: If your organization is not already AWS-native, the onboarding curve is steeper than it looks.Cross-cloud portability is basically nonexistent. A model you fine-tune on Bedrock does not export cleanly to Vertex AI or Foundry. That is a real switching cost you should plan for on day one, not something to figure out later.Some of the newer agentic tooling is still maturing, so documentation gaps show up more than I would like. Google Vertex AI Vertex AI feels different from the other two because Google's DNA here is research first. If your team cares about multimodal capability, or you want access to Gemini models the moment they ship, Vertex AI tends to be ahead. It is also the strongest option if your data already lives in BigQuery, because the integration between Vertex and BigQuery for feature engineering and MLOps pipelines is genuinely smooth. Pros: Best fit for teams doing custom model training, not just calling a hosted API. AutoML and the broader MLOps tooling cut training time noticeably compared to the other two.Tight coupling with BigQuery is a huge advantage if your organization already runs its analytics there. You avoid a lot of data movement overhead.Gemini-first multimodal workflows, plus Google Search grounding for agents, which is something neither Bedrock nor Foundry offers natively.TPU support gives real throughput advantages for heavy batch processing. Cons: If your organization is not GCP-centric already, the value proposition weakens fast. You end up paying a data-gravity tax to move information into Google's ecosystem.Governance and compliance tooling, while solid, is not as battle-tested across regulated industries as AWS's certifications.The agent ecosystem, while improving, still trails Bedrock's AgentCore and Foundry's Azure AI Agents in terms of enterprise adoption stories I have personally seen. Azure AI Foundry Foundry, formerly Azure AI Services, is Microsoft's rebranded and expanded platform, and it is the one I have written about before because it is what my own recent client work has centered on. If your enterprise already lives inside Microsoft 365, Entra ID, and Azure infrastructure, Foundry removes almost all of the identity and governance friction you would otherwise deal with. That matters more than people expect once you are past the proof of concept stage and into actual production rollout with security review. Pros: Deep Microsoft 365 and Entra ID integration means your existing enterprise approvals and identity workflows extend naturally into your AI layer.Strong OpenAI-led model access, since Microsoft's partnership with OpenAI gives Foundry early and deep access to GPT-family models.Hybrid deployment options are genuinely better here than on the other two platforms, which matters if you have on-prem systems you are not ready to fully cloud-migrate.Roughly three-quarters of Fortune 500 companies already run on Microsoft's stack, so for a lot of enterprises Foundry is simply the path of least resistance. Cons: Model breadth is narrower than Bedrock's catalog, so if you need a specific non-OpenAI model family, you may find yourself stitching together a secondary platform anyway.Because it is tied so closely to Azure compute pricing, cost predictability requires more upfront modeling than teams expect.Some newer agentic and orchestration features are still catching up to what AWS has shipped with AgentCore. So Which One Should You Actually Pick? Here is the honest answer I give in client meetings: do not choose based on a benchmark screenshot or a features table. Choose based on where your data already lives and where your governance and compliance story already works. If you are AWS-first and want maximum model flexibility, go with Bedrock. If you are Microsoft-heavy and need your AI layer to inherit existing Entra ID and 365 approvals without a fight, Foundry is the path of least resistance. If your analytics already lives in BigQuery and multimodal Gemini capability is core to your roadmap, Vertex AI earns its place. What I am increasingly seeing among the teams I work with is a hybrid pattern. A primary cloud handles the bulk of regulated workloads, and a secondary cloud gets called in only when a specific model family is not well supported on the primary platform. It is not the cleanest architecture on paper, but it reflects how fast this space is still moving. None of these three platforms is standing still, and the leader on any given feature this quarter is not guaranteed to hold that spot by next year. My suggestion, whichever cloud you land on: build your RAG and orchestration layer with enough abstraction that swapping the underlying model provider is a configuration change, not a rewrite. That single decision will save you more pain than picking the "right" cloud ever will.

It stopped being just a packaging tool the day our onboarding doc got shorter instead of longer. Three weeks into a new ML platform job, I asked a coworker why the 'getting started' doc had a section called 'If conda breaks, try the alternative.' He laughed in a way that told me it wasn't a joke. Every new hire spent their first two days fighting Python versions, CUDA driver mismatches, and a vector database that someone had installed locally in 2022 and nobody dared touch. We had four individuals on the team, each with distinct working setups, and "it works on my machine" was no longer a mere punchline; it had become a regular agenda item during our daily standup meetings. That's the environment I inherited, and it's the reason I ended up rebuilding our entire local AI dev loop around Docker Compose instead of the notebook-and-prayer setup we'd been running. Why This Isn't Just a Packaging Problem The instinct on most teams is to treat Docker as something you reach for at deploy time. You write the model, get it working in a notebook, and only think about containers once it's time to ship. That instinct falls apart with AI workloads specifically because the dev-time dependencies are just as fragile as the prod ones. A GPU-backed embedding model, a local vector store, a retrieval service, and an orchestration layer all need to talk to each other during development, not just in production. If your local loop doesn't mirror that, you spend your debugging time chasing environment drift instead of chasing actual bugs. That was our exact situation, and it cost us roughly a day of onboarding per person plus a steady trickle of 'works for me' bug reports that turned out to be dependency version mismatches. The Setup We Rejected First Our initial response was to improve the Conda environment file and create a more detailed README. In hindsight, that was doomed from the start. Conda solved the Python dependency problem reasonably well but said nothing about the GPU driver version, the vector database binary, or the fact that two people were running Ollama locally with completely different default models pulled. We also floated the idea of just giving everyone a cloud dev environment with GPU access baked in. It solved the consistency problem, but the latency for interactive debugging was miserable, and the monthly bill for keeping GPU instances warm for a six-person team was not something I wanted to defend in a budget review. Neither approach addressed the real issue: we needed one definition of the environment that was runnable identically on a Mac laptop and a Linux workstation. What We Actually Built We moved the whole local AI stack into a single Compose file: an inference service running a small local model, a vector store, and the application layer, all networked together the same way they'd be networked in staging. Here's a trimmed version of what that looked like: YAML services: llm: image: ollama/ollama:latest volumes: ["ollama-data:/root/.ollama"] deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] vectordb: image: pgvector/pgvector:pg16 environment: POSTGRES_PASSWORD: devpass volumes: ["pgdata:/var/lib/postgresql/data"] app: build: ./app depends_on: [llm, vectordb] environment: OLLAMA_HOST: http://llm:11434 That file, plus a one-line 'docker compose up,' replaced two days of onboarding pain with about fifteen minutes. New hires no longer needed tribal knowledge about which conda channel had the right cuDNN build. It also resolved unforeseen bugs by ensuring everyone used the same version of the embedding model, eliminating reports of differing search results caused by dependency drift. The GPU Passthrough Headache Here's where things got tricky. GPU passthrough on Linux with the NVIDIA Container Toolkit is straightforward once it's configured, but it's not portable to Apple Silicon, and half our team was on M-series MacBooks. We ended up maintaining two Compose override files: one that requests GPU reservations for Linux workstations and one for Mac that falls back to CPU inference with a smaller quantized model, accepting slower generation for the sake of a working local loop. It's not elegant, and I still dislike maintaining two code paths for something as basic as "run the model," but the alternative was blocking half the team from working locally at all, which is worse. Where I'd Push Back on the Hype There's a growing narrative that Docker is quietly turning into a full AI platform with model registries, one-command local model pulls, and built-in GPU scheduling for dev. Some of that is genuinely useful, and I would rather not undersell it. But I'd push back on treating Docker as a replacement for a real experiment-tracking or model-serving platform in production. What it's good at is collapsing the dev-time chaos into something reproducible; it is not a substitute for proper GPU orchestration at scale, and teams that try to run Compose-style setups in production tend to relearn the lessons Kubernetes already solved, just slower and with worse observability. The platform shift is real at the development layer. I'm far more skeptical that it fully extends to production serving without a lot of additional tooling wrapped around it. Key Takeaways Treat local AI dev environments with the same seriousness as production ones. Dependency drift in embedding models and vector stores causes real, challenging-to-trace bugs.Conda and README discipline don't solve GPU driver and binary-level mismatches; a single Compose definition does.Plan for hardware heterogeneity early: GPU passthrough doesn't travel to Apple Silicon, so budget for a CPU fallback path.Don't overextend this pattern into production serving; Compose is a dev-loop win, not a Kubernetes replacement. Conclusion What changed for our team wasn't really about Docker getting new AI-specific features, though some of that helped. Realizing that the development environment for an AI application is as complex and failure-prone as production and treating it as an afterthought cost us real engineering hours each week. Whether Docker keeps expanding into model management and becomes a genuine AI platform, or whether that space gets carved up by more specialized tools, I think the underlying lesson holds either way: if your local AI loop isn't reproducible, nothing built on top of it will be either. I'm curious how far other teams have pushed this before Compose starts creaking. Is there a scale at which this pattern breaks down, or a project where you gave up and rebuilt around something heavier?

The first time I containerized a fine-tuned Llama model for a client's internal search tool, the build finished at 38 gigabytes. I remember staring at the terminal thinking there was no way that was right. It was right. The image included a CUDA base, PyTorch with every backend compiled in, model weights baked directly into the layer, and a pip cache that had not been cleaned. Pushing that to our registry took eleven minutes on a good connection. Pulling it onto a fresh node during an autoscale event took even longer, and by the time the pod was ready, the traffic spike it was supposed to handle had already passed. That's the moment I stopped treating LLM containers like regular application containers, because they are not the same animal at all. Why This Problem Actually Matters Most Docker advice out there is written for stateless web services, small images, fast cold starts, and horizontal scaling on demand. LLM workloads break almost every assumption baked into that advice. The artifact is huge, the runtime is GPU-bound, startup involves loading gigabytes into VRAM, and half your "application code" is actually a C++/CUDA binary blob you didn't write and can't easily trim. If you treat an inference container like a Flask app with a bigger base image, you end up with slow deploys, wasted GPU spend, and autoscaling that technically works but arrives too late to matter. The First Wrong Turn: One Image to Rule Them All Our early approach was a single monolithic image model with weights, tokenizer, inference server, and dependencies all baked together, rebuilt on every model version bump. It felt simple. It wasn't. Every retrain meant rebuilding a 30+ GB image even when the code hadn't changed a single line. Registry storage costs gradually increased until someone in finance questioned why our container registry bill resembled that of a second AWS account. Worse, rollbacks were painful because reverting to a previous model meant pulling an entire previous image rather than swapping a much smaller artifact. The solution that actually worked was separating the model weights from the serving image entirely. The image contains the runtime, the inference server (we used vLLM for most of our transformer workloads), and pinned dependencies. Weights live in object storage and are pulled at container start via an init container or a lazy loading entry point. The approach felt counterintuitive at first. Are we effectively transitioning the slower process to startup instead of build time? — but it turned out to be the right trade. Startup pulls are parallelizable, cacheable on the node, and don't bloat the registry. Build time dropped from twenty-plus minutes to under four. A Smaller Base Image Than You'd Expect This is where the challenges began. Everyone defaults to using nvidia/cuda:*-devel images because the framework documentation recommends them, but these devel images include the entire CUDA toolkit, which contains compilers that you will never use at runtime. Switching to the runtime variant and only installing the exact CUDA and cuDNN versions your framework's wheel actually needs cuts roughly 4GB off the base alone. A minimal multi-stage build looks something like this: Dockerfile FROM nvidia/cuda:12.1.0-devel-ubuntu22.04 AS builder RUN pip install --no-cache-dir vllm==0.4.2 FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 COPY --from=builder /usr/local/lib/python3.10 /usr/local/lib/python3.10 COPY --from=builder /usr/local/bin/python3.10 /usr/local/bin/ ENV MODEL_PATH=/mnt/models ENTRYPOINT ["python3", "-m", "vllm.entrypoints.api_server"] The build stage compiles anything that needs the full toolkit; the runtime stage only carries what's needed to execute. It's a basic Docker pattern, but I've seen it skipped constantly on ML teams because the assumption is always, "the model is the heavy part; the image doesn't matter." The model is heavy, sure, but a bloated base image adds real minutes to every autoscale event, and in production that's the difference between absorbing a traffic spike and dropping requests. The OOM Kill: Nobody Explained Well This is the war story I bring up most often. We had a container that ran fine locally and in staging, then got silently killed in production under load — no crash log, no stack trace, just a pod restart and a confused on-call engineer at 2 AM. It turned out to be the kernel OOM killer, not an application-level exception, because our memory limit accounted for the model weights in VRAM but excluded the growing KV cache for long-context requests plus the CPU-side tokenizer buffers. GPU memory and container memory limits are two completely separate accounting systems, and Kubernetes will happily kill your pod over host RAM even if your GPU has headroom to spare. The fix was unglamorous: we set explicit memory requests and limits with a real margin above peak KV cache usage, moved batch size and max sequence length into environment-configurable values instead of hardcoding them, and added a lightweight health assessment that reported GPU memory utilization alongside the standard liveness probe. None of that is exotic. All of it was missing because we'd copy-pasted a manifest template built for a stateless API and never revisited the resource math for a model that holds state in memory for the duration of a request. Where I'd Push Back on Common Advice A lot of guidance recommends one model per container for isolation, and for many teams that's right. But if you're serving several small fine-tunes of the same base model, that pattern wastes GPU memory by duplicating base weights across containers. We transitioned to a multi-adapter setup, where one base model is loaded once, and LoRA adapters are swapped for each request; this approach is more complex operationally but reduces the GPU footprint by nearly half. I wouldn't consider it a default; it represents a level of complexity that is justified only after demonstrating that plain per-model containers are indeed the bottleneck. I'd also push back on containerizing every workload the same way. Batch inference and real-time serving have almost opposite goals: one wants throughput and tolerates slow cold starts; the other needs rapid readiness and predictable latency. We split these into separate images with separate resource profiles, even though it meant more Dockerfiles. Fewer surprises beat fewer files. Key Takeaways Separate model weights from the serving image; bake them in the runtime and pull weights at startup from object storage.Use CUDA runtime images, not devel images, unless you genuinely compile something at container start.Account for GPU memory and host memory as two separate budgets; KV cache growth is the usual silent killer.Split batch and real-time serving into different images; their optimization goals are conflicting.Don't reach for multi-adapter serving or other density tricks until you've measured that plain per-model containers are actually the bottleneck. Closing Thought None of this required exotic tooling, no custom orchestrator, and no proprietary platform. It required treating the container as part of the model's runtime behavior rather than a packaging afterthought bolted on after the research work was done. The teams that struggle most with this approach usually aren't lacking Docker knowledge; they're applying web-service intuition to a workload that behaves nothing like a web service. If you're mid-migration on something similar, I'd genuinely ask: are you optimizing your image for build convenience or for what actually happens the moment traffic hits a cold node? Those answers are rarely the same, and figuring out which one you've been solving for is usually the first real fix.

If different Docker Engine versions are running simultaneously in a Docker Swarm cluster, this may lead not to an obvious service outage but to a more subtle scenario: partial traffic degradation on individual nodes. In this case, the issue appeared on one of the manager nodes, Traefik started reporting an unavailable status (health=0) for the router-app service, and the cause, according to the working hypothesis, was related to differences in iptables rules and overlay networking between Docker 28.1.1 and 28.2.2. On June 22, 2025, this exact scenario occurred in the production cluster of the backend infrastructure for a socially significant public transportation mobile application. The system serves about 2 million users, several tens of thousands of daily active users, and a total load of around 1000–1600 RPS, so even partial degradation at a single entry point affected a high-load segment of traffic and could have had a noticeable impact on SLA metrics if it had not been localized in time. Context At the time of the incident, the Docker Swarm cluster consisted of 5 manager nodes, several dozen worker nodes, and approximately 40–50 services. External HTTP traffic passed through Traefik, deployed on each of the five manager nodes, and was then routed by Traefik to the backend application containers. One of the key services was router-app, responsible for building public transportation routes on the frontend. It was one of the critical entry-point services with the highest SLA, and any disruption to its availability could have led to severe penalties from the customer, so any deviation in its availability required an immediate response. Grafana dashboards showed application availability through Traefik health-check statuses. Those statuses were generated based on HTTP health-check endpoints implemented by the developers for most services, primarily the most critical ones. This was enough to quickly localize the problem at the ingress traffic level. The cluster had one important characteristic. Some nodes were running Ubuntu 20.04 (focal), while others were running Ubuntu 22.04 (jammy), and different APT repositories were pulling different Docker Engine versions. As a result, after another scheduled Docker update on the nodes, the production environment ended up with a mix of nodes running 28.1.1 and 28.2.2 at the same time. The docker node ls screenshot additionally confirmed that mixed versions were present not only on worker nodes but also on manager nodes, including swarm4 and swarm5. How the Incident Manifested The incident was detected not through user complaints and not through a general service outage, but through Traefik monitoring. The triggered alerts showed partial unavailability of one of the router-app containers, after which the health-check dashboard confirmed that the issue affected not the entire service but one of the manager nodes. This is important because the red blocks on the dashboard did not indicate a complete outage of router-app. It meant that Traefik on one of the manager nodes started receiving health=0 when checking that service’s backend endpoint, while the other manager nodes continued to see the backend as healthy. In practice, it looked like this: traffic through one of the manager nodes stopped reaching the router-app containers correctly, but Traefik, running on all five manager nodes, automatically excluded requests to the unhealthy entry point. As a result, from the outside the incident appeared as partial degradation rather than full unavailability. That is exactly what made the situation tricky. Fault tolerance limited the impact of the incident, but the underlying cause remained inside the cluster and continued to affect one of the entry points. What Docker Showed After localizing the issue to one of the manager nodes, it became clear that the cause should be sought not in router-app itself but in the network path between Traefik and the backend containers. At the same time, docker service ps did not show a widespread service failure, and the containers still appeared as running. The next useful signal came from the dockerd logs on swarm5. Repeated messages appeared there, including Peer delete operation failed, neighbor entry not found, and errors related to deleting FDB and neighbor entries for the VXLAN interface vx-001001-5lk08. For example: Plain Text Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.366807202Z" level=warning msg="Peer delete operation failed" error="could not delete fdb entry for nid:5lk08r7jjvtq5idqggzeygmlv eid:4e7d63d00fffaa6be7ce6362f47acd7912f7c11e5ac6e018393722decc16c210 into the sandbox:neighbor entry not found for IP 10.170.0.37, mac 02:42:0a:1b:14:2c, link vx-001001-5lk08" Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.765092537Z" level=warning msg="error deleting neighbor entry" error="no such file or directory" ifc=vx-001001-5lk08 ip=10.170.0.138 mac="02:42:0a:1b:14:65" Such messages were highly consistent with problems in Docker Swarm’s overlay network. In essence, Docker was trying to delete network records that were no longer present in the tables, which usually points to desynchronization of network state at the VXLAN, FDB, or neighbor-table level. By themselves, these messages still did not provide a complete explanation, but they pushed the investigation in the right direction. It became clear that the problem was not in the application’s business logic but in the network layer on one of the nodes. Additionally, docker node inspect self --pretty on swarm5 showed that from Swarm’s point of view the node looked normal: State: Ready, Availability: Active, Raft Status: Reachable, Leader: No, while Engine Version was already 28.2.2. This was an important point: the control plane still considered the node healthy, even though at the traffic-flow and network-state level it was already behaving differently. Diagnostics The investigation was carried out at the node level. The tools used included docker node ls, docker version, apt-cache policy docker-ce, as well as ip link show, bridge fdb show, ip neigh show, and comparisons of iptables chains across different nodes. The key fact became visible after docker node ls. The cluster was not homogeneous in terms of Docker Engine version: some manager nodes and some worker nodes were already running 28.2.2, while the others remained on 28.1.1. This led to the assumption that the issue might be at the iptables rules level. After that, iptables had to be compared separately on healthy and problematic nodes. On swarm5, a full rules dump was collected using the combination of iptables -S, iptables -t nat -S, and iptables -t mangle -S. Those rules showed the DOCKER, DOCKER-FORWARD, DOCKER-INGRESS, and DOCKER-USER chains, as well as ACCEPT, DROP, and DNAT rules for traffic through docker_gwbridge, published ports, and ingress routing.... To test the hypothesis, not only the problematic swarm5 but also the first manager node, swarm1, was compared, where a stable stack with Docker 28.1.1 had long been running. On swarm1, the output of iptables -S and iptables -t nat -S showed the expected picture: the DOCKER and DOCKER-INGRESS chains contained a full set of ACCEPT and DNAT rules for all published ports (80, 8080–8082, and dozens of internal service ports) with symmetric dport/sport pairs, while DOCKER-USER effectively boiled down to a clean RETURN. Taken together with the dump from swarm5, this reinforced the conclusion that on nodes running 28.1.1, the iptables configuration for ingress and routing was consistent, and the differences seen on 28.2.2 were related not to manual changes but to the behavior of Docker Engine itself. After that, iptables had to be compared separately on other nodes running different Docker versions. On nodes with 28.1.1, the DOCKER and DOCKER-USER chains and the associated rules were in the expected state, whereas on nodes with 28.2.2 some of the required rules were missing or the chains were reduced to a minimal RETURN. This explained the observed behavior well. The services remained running, Swarm did not appear broken, but external traffic and part of the overlay routing through a specific manager node were working incorrectly, causing Traefik on that node to report health=0 for router-app. It is worth noting separately that journalctl -u docker and docker service ps did not provide a simple direct cause for the incident. They did not show a picture of a general failure, so the conclusion had to be assembled from several sources: Traefik monitoring, dockerd logs, Docker versions, and the state of iptables on different nodes. Fix Once the main hypothesis had narrowed down to mismatched Docker Engine versions, the solution was fairly straightforward: return the cluster to a homogeneous configuration by rolling back to version 28.1.1 as the fastest solution. The rollback was performed for swarm4, swarm5, and all worker nodes where version 28.2.2 had already been installed. To do this, a specific package version was pinned via apt, then Docker was restarted, and the installed version was verified. One version of the commands looked like this: Shell apt-cache madison docker-ce | grep 28.1.1 apt-get install docker-ce=5:28.1.1-1~ubuntu.22.04~jammy \ docker-ce-cli=5:28.1.1-1~ubuntu.22.04~jammy \ containerd.io && systemctl restart docker && docker --version Additionally, it made sense to check the package sources and, if necessary, remove conflicting APT entries so that the nodes would no longer receive an unsuitable Docker version from another repository. In practice, it looked like this: Shell sudo rm /etc/apt/sources.list.d/download_docker_com_linux_ubuntu.list sudo apt update After the rollback, docker version was checked again, as well as the DOCKER and DOCKER-USER chains. After Docker Engine had been unified to 28.1.1 on both manager and worker nodes, the issue disappeared. From the perspective of external behavior, this was confirmed immediately. Health checks in Traefik returned to the green zone, and the partial unavailability of router-app on one of the manager nodes could no longer be reproduced. Root Cause Based on the available data, the most well-founded working version is this: in this environment, Docker Engine 28.2.2 formed or applied iptables rules related to DOCKER, DOCKER-USER, FORWARD, ingress, and overlay networking differently. In a mixed cluster, this led to one of the manager nodes no longer forwarding traffic correctly to the router-app backend containers, even though from the perspective of the control plane and service state this did not look like a direct failure. It is important here not to overstate what the data allows. This case does not prove a universal upstream bug in Docker 28.2.2 for all Swarm installations, but it does show that even closely related Docker Engine versions can affect the cluster’s network plane differently, especially when different Ubuntu distributions and different package sources are present in production at the same time. What Follows From This The first conclusion is simple: Docker Swarm is sensitive to Docker Engine version mismatches. If some manager or worker nodes have been updated while others have not, this can lead not only to version drift as an organizational problem, but also to practical issues with traffic, published ports, and overlay routing. The second conclusion is that after updating Docker, it is necessary to check not only docker version but also the node’s network behavior. The minimum set includes iptables -L DOCKER -v -n, iptables -L DOCKER-USER -v -n, checking published ports, ingress/overlay state, and health checks from the edge proxy. The third conclusion is that it is useful to maintain a single baseline stack across all nodes. One Ubuntu LTS distribution, unified repositories, and the same update order reduce the chance that cluster state will remain formally healthy while part of the network traffic is already being handled incorrectly. The fourth conclusion concerns update order. In our case, that was exactly how it happened, but it is worth noting separately. When Docker is updated in a Swarm cluster, it is better to update worker nodes first, then manager nodes, and the leader last, while after each stage separately checking the node’s behavior under real traffic conditions (service availability, correct routing, and published ports). In our case, enhanced monitoring was in place, so no additional manual checks of node behavior in traffic were required: if any part of the infrastructure became unavailable, we would promptly receive an alert. The final conclusion relates to monitoring. In this case, Traefik not only helped limit the impact of the incident by routing around the unhealthy node, but also provided the first precise signal that the problem was localized to a specific entry point rather than existing at the level of the entire service or the entire cluster.

Introduction The meeting that changed the platform team's direction was not a technical one. It was a conversation with a product engineer who had been at the company for eight months and had never successfully deployed to production without help from someone on the platform team. Not because she lacked skill. She was smart, experienced, and had successfully launched production systems at two previous jobs, but getting a working service into production meant dealing with fifteen different configuration files across four repositories, figuring out how Helm values files and Kustomize overlays worked together, and knowing which of the three CI pipeline templates to use based on whether the service needed a sidecar, a job scheduler, or neither. She had read the documentation. It was accurate. It just didn't tell her what to do when the documented path didn't match the state of her specific service in her environment. The platform team had built powerful infrastructure. They had not built a usable platform. That distinction between infrastructure and platform is where most Kubernetes-based internal developer platform efforts go wrong, and it's worth being precise about what it means. Infrastructure vs Platform: A Practical Distinction Infrastructure is the machinery: the Kubernetes clusters, the networking layer, the CI pipelines, the secrets management system, and the monitoring stack. A platform is the interface that makes that machinery accessible to developers who aren't Kubernetes experts without requiring them to become ones. The confusion between the two produces a situation that's extremely common in engineering organizations: a technically sophisticated infrastructure that's effectively only usable by the people who built it. The test for whether you have a platform or just infrastructure is simple: can a developer who joined three months ago deploy a new service to production without asking anyone for help? Not by following a tutorial someone wrote last year that may or may not still be accurate, but through tooling that guides them through a current, correct process. If the answer is no, you have infrastructure. The platform is the missing layer. This statement is not an argument against complexity in the underlying system. Kubernetes is complex, and that complexity exists for beneficial reasons: flexibility, programmability, and a rich ecosystem. The platform layer should absorb the complexity, rather than exposing it to every developer who needs to ship a service. What the First Attempt Got Wrong The infrastructure team built the first version of the internal platform in their spare time, juggling it with other priorities. It consisted of a set of Helm chart templates, a GitHub Actions workflow library, and a wiki with deployment instructions. This approach is how most internal platforms start, and it has a predictable failure mode: the templates encode the assumptions of the people who wrote them, the wiki goes stale within weeks, and the gap between the documented process and the actual state of the infrastructure grows invisibly until it becomes a significant tax on every developer who hits it. The fundamental mistake was treating platform work as documentation work rather than product work. A wiki is not a platform. A set of templates that require understanding to use correctly is not a platform. A platform is software that makes the correct path the easy path, that validates inputs before they cause problems downstream, and that fails loudly and helpfully rather than silently and mysteriously. The second attempt started from a different premise: the platform is a product, developers are its users, and the measure of success is whether they can do their jobs without needing the platform team. The Abstraction Layer: Custom Resources and Admission Webhooks The technical decision that made the most difference was introducing a custom resource definition (CRD) that represented a service in the platform's domain model, not a Kubernetes Deployment or Service, but a higher-level construct that encoded the platform's opinionated defaults and generated the underlying Kubernetes objects from a much simpler specification. YAML # Platform-level CRD: what developers actually write apiVersion: platform.company.com/v1 kind: AppService metadata: name: payment-api namespace: production spec: image: payment-api:v1.4.2 tier: backend # drives resource limits, network policy replicas: 3 port: 8080 healthCheck: /healthz env: DATABASE_URL: secretRef: payment-db-credentials This twelve-line manifest replaced the hundred-plus lines of Kubernetes YAML that developers had previously been required to write and maintain. The controller running in the cluster, a standard Kubernetes operator built with controller-runtime, read the AppService resource and generated the Deployment, Service, HorizontalPodAutoscaler, PodDisruptionBudget, and NetworkPolicy that the platform's standards required, with defaults applied consistently across every service. The key design decision was what to expose in the CRD and what to hide. The tier field is a prime example: rather than exposing resource requests and limits directly, which requires understanding what values are appropriate for the cluster, the CRD accepts a tier label (frontend, backend, worker, batch) that maps to a predefined resource profile. A backend tier service receives a specific CPU and memory allocation appropriate for the cluster's node types. A batch tier service receives a different profile with different eviction priorities. The developer specifies intent; the platform enforces the appropriate configuration. Go # Controller logic: tier maps to resource profile (Go pseudocode) func resourceProfileForTier(tier string) corev1.ResourceRequirements { profiles := map[string]corev1.ResourceRequirements{ "frontend": { Requests: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("100m"), corev1.ResourceMemory: resource.MustParse("128Mi"), }, Limits: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("500m"), corev1.ResourceMemory: resource.MustParse("256Mi"), }, }, "backend": { Requests: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("250m"), corev1.ResourceMemory: resource.MustParse("256Mi"), }, Limits: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("1000m"), corev1.ResourceMemory: resource.MustParse("512Mi"), }, }, // batch, worker profiles follow same pattern } return profiles[tier] } Admission webhooks complemented the CRD by catching misconfiguration before it reached the cluster. A validating webhook checked every AppService manifest against a set of rules: the image tag must not be 'latest,' the health verification path must respond within the cluster, and secret references must exist in the target namespace and return a clear error message describing exactly what was wrong and how to resolve it. This shifted error detection from 'runtime, after deployment' to 'submission time, before anything breaks,' which dramatically reduced the debugging load on both developers and the platform team. The Golden Path and Its Limits The CRD and controller approach works well when services fit the platform's model. Here's where things became challenging: not all services fit the platform's model. A service that needed custom init containers, a service that required a specific affinity rule because of a hardware dependency, and a batch job with a complex retry policy that didn't map cleanly to the tier abstraction. Each of these required either extending the CRD or breaking the abstraction and falling back to raw Kubernetes YAML. The temptation is to keep extending the CRD to cover every case. Resist it. A CRD that tries to expose every Kubernetes feature is just a more complicated way to write Kubernetes YAML, and it loses the simplicity that made the abstraction valuable. The better model is a golden path, the CRD for the 80% of services that fit the standard model, and a documented escape hatch for the 20% that don't. The escape hatch is raw Kubernetes resources, maintained by the teams that need them, with the platform team providing support rather than ownership. The key is being honest with developers about which path they're on. A service using the AppService CRD gets platform-managed defaults, automatic updates when the platform evolves, and first-class support. A service using raw Kubernetes resources owns its own configuration and gets best-effort support. That distinction in the support model is what makes the trade-off legible rather than arbitrary. What I'd Do Differently In hindsight, the most important investment was the admission webhook, and I'd build it earlier. The CRD and controller took significant time to design and implement. The webhook could have been built in a few days and would have immediately improved the developer experience by catching misconfiguration at submission time rather than deployment time. Validation before generation is higher-leverage than generation that might produce something invalid. I'd also measure platform adoption from day one. Which teams are using the AppService CRD? Which teams are on raw Kubernetes? What's the conversion rate of new services to the platform abstraction? Without that data, the platform team relies on intuition instead of evidence to guide their investment. The teams that adopt slowly are often the ones with the most valuable feedback about where the abstraction doesn't fit, and they're also the teams most likely to be quietly maintaining fragile custom configurations that will become incidents later. When should you not build a CRD-based platform abstraction? If you have fewer than fifteen to twenty engineers deploying services, the overhead of designing, building, and maintaining a CRD-based platform abstraction almost certainly exceeds the value. Helm charts and excellent templates get you most of the way there with a fraction of the complexity. The operator pattern earns its cost when you have enough services that inconsistency becomes a real operational problem when the differences between how services are configured start causing incidents and nobody can tell you why a particular service is configured the way it is. Key Takeaways Infrastructure and platform are different things. Infrastructure is the machinery; a platform is the interface that makes machinery accessible without requiring expertise in its internals. Most Kubernetes-based IDPs stop at infrastructure. Custom resource definitions let you define a domain model that encodes your platform's opinions. Developers specify intent (tier, replicas, port); the controller generates the correct Kubernetes objects with consistent defaults applied. Admission webhooks shift error detection from runtime to submission time. A clear error message at kubectl apply is worth more than a mysterious pod crash two minutes later. Maintain a golden path for the majority of services and a documented escape hatch for the rest. A CRD that tries to cover every Kubernetes feature loses the simplicity that justified building it. Conclusion The platform engineer's job is to make complexity disappear, not by eliminating it, but by absorbing it into tooling so that the people building products don't have to carry it. That's a harder problem than building the infrastructure itself, and it requires a fundamentally different mindset: less systems engineering, more product thinking. Who are the users? What tasks do they need to accomplish? Where does the current experience fail them? The teams building internal developer platforms who get the process right tend to look, from the outside, like they have unusually productive engineering organizations. Individual contributions ship faster, incidents caused by misconfiguration drop, and the platform team spends less time on support and more time on improvements. The causal chain runs directly from platform quality to engineering output, even though it's usually measured differently. The open question is whether the CRD-based abstraction model scales to genuinely heterogeneous service fleets, the kinds of organizations where services span multiple languages, multiple deployment patterns, and multiple infrastructure dependencies. The golden path works when most services look similar enough that a shared abstraction is useful. What occurs to the platform model when 40% of services utilize the escape hatch? At that point, is the abstraction still earning its cost, or is it adding complexity without delivering the simplicity it promised?

Running AWS resources locally is a game-changer for engineering velocity, cost optimization, and developer autonomy. Traditionally, testing cloud infrastructure required deploying directly to a staging or sandbox AWS account. This workflow introduced painful friction points: waiting for slow cloud provisioning cycles, tracking down orphaned resources that inflate the monthly bill, and requiring a constant, high-speed internet connection. LocalStack solves this by emulating core AWS services, such as S3, SQS, DynamoDB, and other services directly on your local machine inside a Docker container. When paired with Terraform, you can safely write, plan, and apply infrastructure-as-code (IaC) configuration blueprints against this local simulator. This guide walks you through the definitive "happy path" for configuring LocalStack and Terraform, followed by a robust troubleshooting handbook for common architecture-specific and container networking errors. This allows you to provision these mock resources cleanly. This allows testing Terraform code with local resources without incurring real AWS costs, requiring internet connectivity, or dealing with slow cloud provisioning cycles. The Happy Path Setup Step 1: Setting the Stage: Launching LocalStack With Docker To get started, we need our local AWS cloud stack running inside a container. We will pull the official LocalStack image, set up our credentials, and spin up the container. First, pull the latest official image to your local machine: Before firing up the container, head over to the LocalStack Web App Dashboard to grab your personal access token (PAT). While LocalStack offers an open-source community edition, advanced features or specific emulated APIs may check for a valid token. Export this token into your shell environment so the container can authenticate and activate premium features on startup: Shell export LOCALSTACK_AUTH_TOKEN="ls-..." Now, launch the container. We need to map the primary edge gateway port (4566), which routes all inbound AWS API requests, along with the standard range of ports used by individual internal services (4510-4559). We also pass our token as an environment variable: Shell docker run --rm -it \ -p 4566:4566 \ -p 4510-4559:4510-4559 \ -e LOCALSTACK_AUTH_TOKEN=$LOCALSTACK_AUTH_TOKEN \ localstack/localstack Keep an eye on your terminal logs. LocalStack will quickly validate your token, pull your license configuration, and initialize the mock runtimes. You will see a clear notification when the edge proxy is fully ready to handle incoming API requests. Step 2: The S3 Sanity Check: Talking to LocalStack Before configuring our automation toolchain, let's run a quick manual sanity check using the standard AWS CLI. Because LocalStack runs entirely on your machine, we must override the default cloud routing by passing a custom --endpoint-url pointing to our local edge proxy. To verify that LocalStack is running and reachable, create a local S3 bucket and upload a test file using the AWS CLI. 1. Create a Bucket Shell aws s3 \ mb s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 2. Upload an Object Create a dummy text file and copy it into your new mock bucket: Shell aws s3 \ cp /tmp/demo.txt s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 3. List Objects Verify the object is safely stored inside the mock container: Shell aws s3 \ ls s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 Step 3: Writing the Blueprint: Configuring the Terraform Provider Now let's automate things. To instruct Terraform to deploy resources to our local simulator instead of the real AWS cloud, we must customize the AWS provider block. We enforce dummy credentials, bypass cloud-only identity validations, and explicitly force all API endpoints to route directly to http://localhost:4566. Providers Configuration Create a file named providers.tf with the following content: Markdown terraform { backend "local" { path = "terraform.tfstate" } required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } } provider "aws" { region = "us-east-1" access_key = "mock_access_key" secret_key = "mock_secret_key" skip_credentials_validation = true skip_metadata_api_check = true skip_requesting_account_id = true s3_use_path_style = true # Redirect all endpoints to LocalStack's edge port endpoints { apigateway = "http://localhost:4566" cloudwatch = "http://localhost:4566" dynamodb = "http://localhost:4566" ec2 = "http://localhost:4566" iam = "http://localhost:4566" lambda = "http://localhost:4566" rds = "http://localhost:4566" s3 = "http://localhost:4566" secretsmanager = "http://localhost:4566" sns = "http://localhost:4566" sqs = "http://localhost:4566" ssm = "http://localhost:4566" sts = "http://localhost:4566" } } SQS Resource Definition Next, define the SQS queue we want to provision. Create a file named main.tf: Markdown resource "aws_sqs_queue" "local_queue" { name = "my-local-queue" delay_seconds = 90 max_message_size = 2048 message_retention_seconds = 86400 receive_wait_time_seconds = 10 } output "queue_url" { value = aws_sqs_queue.local_queue.id } Step 4: The Moment of Truth: Initializing and Applying Configuration With our configuration defined, we can run Terraform. Ensure you are executing a native binary that matches your host system architecture (such as a native darwin_arm64 binary if you are working on an Apple Silicon machine) to prevent execution overhead. Initialize Terraform First, initialize the working directory to download the AWS provider plugins: Generate and Review the Plan Next, generate and review an execution plan. The plan output will detail our local queue configuration without attempting to connect to actual AWS endpoints: Apply the Plan Apply the plan to deploy the queue directly to LocalStack. Upon completion, Terraform will write your state file locally and output your new mock SQS queue URL: Step 5: Taking It for a Spin: Sending and Receiving SQS Messages To confirm that our Terraform-provisioned SQS queue is fully operational, let's capture the output URL and push a real message through it using the AWS CLI. 1. Send a Message Shell export QUEUE_URL="http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/my-local-queue" aws sqs send-message \ --endpoint-url=http://localhost:4566 \ --region us-east-1 \ --queue-url $QUEUE_URL \ --message-body "Hello from LocalStack SQS" 2. Receive the Message Shell aws sqs receive-message \ --endpoint-url=http://localhost:4566 \ --region us-east-1 \ --queue-url $QUEUE_URL { "Messages": [ { "MessageId": "1235d997-f60a-4e86-b248-aff3f5f41dde", "ReceiptHandle": "NzYxOThkMDAtMWJiOC00OGVhLTllMDEtNTU3ZTY3ZGQ5M2I4IGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6bXktbG9jYWwtcXVldWUgMTIzNWQ5OTctZjYwYS00ZTg2LWIyNDgtYWZmM2Y1ZjQxZGRlIDE3ODI4OTI2MzcuMjg2ODc1NQ==", "MD5OfBody": "88dc2faa42b899c03e12fd3ac96d714b", "Body": "Hello from LocalStack SQS" } ] } Your terminal will return a successful JSON payload containing your message body, confirmation IDs, and MD5 hashes, proving that the local loop is entirely complete. Event Verification in LocalStack Logs Checking the LocalStack container console confirms the queue creation, message send, and message fetch operations were handled successfully: Troubleshooting Guide Even on a happy path, local container networks and mixed system architectures can throw a wrench into your workflow. Here is how to fix the most common bottlenecks. The Apple Silicon (M1/M2/M3) Rosetta Loop Symptom: The LocalStack container crashes unexpectedly on startup, or loops endlessly while attempting to launch internal components like local Lambda runtimes, throwing qemu: uncaught target signal 11 errors. The Cause: LocalStack occasionally spins up secondary processes or helper binaries inside the container. If Docker Desktop is forced to emulate an x86_64 architecture via Virtualization frameworks on an ARM64 Apple Silicon chip, the emulation layer can break during heavy nested execution. The Fix: Ensure your Docker Desktop configuration has Use Virtualization framework enabled under Settings -> General, and turn on Rosetta for x86/amd64 emulation under the Features in Development tab. Alternatively, force Docker to fetch the native ARM64 container image by updating your execution command to include the specific platform flag: Shell docker run --platform linux/arm64 --rm -it -p 4566:4566 localstack/localstack "Port Already in Use" Symptom: Docker fails to bind ports, displaying an error message like: Bind for 0.0.0.0:4566 failed: port is already allocated. The Cause: A previous instance of LocalStack didn't shut down cleanly, or another local development tool is monopolizing port 4566. The Fix: Option 1: Check for lingering Docker containers Often, a container crashed or was backgrounded but didn't release the port. Find any container using 4566: Shell docker ps -a | grep 4566 If a container shows up, stop and remove it (replace <CONTAINER_ID> with your specific ID): Shell docker stop <CONTAINER_ID> docker rm <CONTAINER_ID> Option 2: Kill native background processes If Docker isn't holding the port, another process on your host machine is. You'll need to find its Process ID (PID) and force-quit it. Find the PID: Shell lsof -i :4566 Kill it (look for the number under the PID column): Shell kill -9 <PID> Wrapping Up Combining LocalStack and Terraform gives you a lightning-fast, zero-cost, offline sandbox for cloud infrastructure development. Once your environment is configured correctly with a valid personal access token, precise Docker port mappings, and native toolchains matched to your host CPU, you can prototype, test, and tear down AWS configurations in seconds. No more waiting for slow cloud deployments or tracking down orphaned cloud resources. Happy local provisioning!

AWS can send audit logs to an attacker’s account unless denials are enforced at the network layer, while Azure doesn’t log network-block requests at all. The concept of a data perimeter was popularized by AWS [1] to establish organizational boundaries around identities, resources, and networks. In simple terms, AWS provides access controls to ensure that trusted identities access trusted resources from expected networks while blocking all outside access. This article explores how different cloud providers handle resource access logs and how it relates to data protection. It sets up an experiment where an outside identity with valid credentials accesses a trusted resource and is blocked by a policy in one of the scenarios. The experiment explains two scenarios that differ in where the deny decision is enforced. We find that the same request for resource access produces different log artifacts in AWS and Azure. AWS sends access logs containing caller-controlled metadata in both the identity and resource-owner accounts unless a network layer explicitly denies access. However, in Azure, resource access logs are only logged at the resource-owner’s subscription, and when access is blocked at the network layer, nothing is logged there either. Both behaviors have consequences for security teams collecting and analyzing audit logs. This article walks through both scenarios with lab experiments and reproducible code. Background AWS and Azure treat identities differently. In AWS, identities are not centralized into one single place — instead, they live at the account level. For example, if an organization contains 10 accounts, identities can be created in each of the 10 accounts. In comparison, in Azure, identities are centralized into one Entra ID tenant. Since a tenant is linked to multiple subscriptions containing the company’s resources, identities from the same tenant are configured to access resources inside subscriptions. In summary, the resource-owning entity in AWS (the account) also holds identities, whereas in Azure the resource-owning entity (the subscription) does not hold identities – those live in the Entra ID tenant. Secondly, AWS and Azure treat access logging differently. In AWS, CloudTrail logs API calls at the account level. For cross-account access, AWS lets customers configure CloudTrail such that when data events are enabled, the caller account and the resource-owning account get access events. For example, if an identity in Account-A accesses a resource in Account-B and gets denied, then the deny audit entry is logged in both Account-A and Account-B. This mirroring is what makes caller-controlled metadata visible to a malicious actor’s account [2]. In contrast, in Azure, resource access logs (for example, StorageBlobLogs) live in the storage account in the subscription, whereas identity logs (Entra ID) live with the tenant. These are separate systems with no automatic mirroring. This difference sets up why a correlation problem exists and why a network-layer block does not produce logs at the resource layer. Threat Model The threat model is as follows: an attacker brings their credentials inside a corporate network and accesses the company’s resource (like an S3 bucket). By doing this, the attacker tries to exfiltrate company data by encoding sensitive information in the HTTP user agent header, a caller-controlled field that appears in access logs. This allows data to leave the corporate environment in small chunks across multiple requests. The second threat is more nuanced. A security team that relies on resource-layer logs to detect unauthorized access attempts will miss requests that are blocked before reaching the resource. If the network drops the request silently, the resource (service) never logs it. An attacker who knows this can probe a corporate environment repeatedly without appearing in the audit trail that the security team is monitoring. Experiments AWS Experiment To set up this experiment, we have three accounts: a credential-owning account (identity), a VPC-owning account, and a resource-owning account. The identity is a Lambda function that tries to access an S3 bucket (resource). The Lambda function runs from a private subnet in a VPC and accesses the S3 bucket through an S3 VPC endpoint (AWS PrivateLink). All audit logs are sent to a third account – this is a typical Control Tower setup [3]. We test two scenarios: The bucket policy denies all untrusted identities — assume that the bucket policy denies access to our identity. However, the VPC endpoint policy allows all cross-account access. The bucket policy allows this untrusted identity. However, the VPC endpoint policy disallows cross-organization access. Scenario 1 When the request gets denied at S3, AWS CloudTrail generates a standard API event: JSON { "eventType": "AwsApiCall", "errorCode": "AccessDenied", "userAgent": "...", "requestParameters": {...}, "tlsDetails": {...} } The full log is in https://github.com/sureshgururajan/aws-data-exfiltration-demo/blob/main/testing-results/scenario1-log.md. In this case, the full request context is preserved. This includes: userAgent requestParameters TLS metadata Additional request context The main observation is that this event includes caller-controlled metadata in the userAgent field. Since customers can configure CloudTrail to log data events on both the caller account and the resource account, a malicious actor gets the same denial event in their account. Therefore, an attacker in an untrusted account can exfiltrate company data into their accounts by triggering these denied access requests on the company resource. Scenario 2 In the second scenario, if the VPC endpoint policy denies cross-account access (example), CloudTrail generates a different event: JSON { "eventType": "AwsVpceEvent", "eventCategory": "NetworkActivity", "errorCode": "VpceAccessDenied", ... } See the full log here. Instead of logging an AwsApiCall event, CloudTrail logs NetworkActivity with the errorCode: VpceAccessDenied and does not log the HTTP user agent header. More importantly, this event is not sent to the malicious actor or the resource owner’s account. Rather, the event is sent to the VPC endpoint owner’s account. In other words, the cause of the denial was a VPC endpoint policy, and therefore CloudTrail generates a NetworkActivity event rather than the API event and routes it to the VPC-owning account. This prevents the bad actor from stealing company data via CloudTrail. Azure Experiment To set up this experiment, we created two Azure subscriptions – one for identity and the other for the resource. An Azure function in subscription-A writes to a blob storage in subscription-B. The Azure function is registered as a system-assigned managed identity in the Entra ID tenant while turning off the shared access key for the blob storage to ensure only managed identities can access it [5]. The function uses DefaultAzureCredential to request a token from Entra ID and attempts to write to a file in the storage account. Since both subscriptions trust the same Entra ID tenant, the identity moves across subscriptions natively without needing an AssumeRole step. Like before, we run through two scenarios: Azure function has the Storage Blob Data Contributor role and the network path is open The Azure function attempts to write to the storage account but is blocked by the firewall. Scenario 1 When the request is allowed at the blob storage, the following logs are written: The Entra ID tenant gets a token request log when the Azure function uses default Azure credentials. This event does NOT contain any information about the actual API action being taken. The resource account StorageBlobLogs records a PutBlob event with the file name and IP address but doesn’t show the name of the managed identity. Sample log entry from StorageBlobLogs Plain Text TimeGenerated [UTC] - 2026-05-02T19:30:32.7306109Z OperationName - PutBlob CallerIpAddress - 172.24.1.71:9156 Uri - https://sgrstorageaccountinsubb.blob.core.windows.net:443/storage-container/test.json AuthenticationType - OAuth RequesterObjectId - 00daa177-96c6-4b29-9a5c-53ca603565e9 StatusCode – 201 UserAgentHeader - azsdk-js-azure-storage-blob/12.31.0 core-rest-pipeline/1.22.3 Node/22.22.2 (Linux 6.6.130.1-3.azl3; x64) The requester object ID field indicates which identity made the request but doesn’t reveal more details as to the identity itself. That part is left to the Entra ID logs as shown below. However, we can see that the userAgentHeader is logged. The difference with AWS is that in Azure, the StorageBlob log entry is not mirrored to Entra ID, i.e., the caller’s subscription. In Azure, it stays only in the resource owner’s subscription. Entra ID contains just the token issuance log: Sample log entry from Entra ID Plain Text Date (UTC),2026-05-02T19:30:32Z Request ID,25c5f7f7-4206-448d-817b-730744991701 Correlation ID,73cf7b90-c49b-40f0-800d-74e77e40717c Service principal ID,00daa177-96c6-4b29-9a5c-53ca603565e9 Service principal name,SureshTestingMultiCloud-Function Credential key ID, Credential thumbprint, Application,SureshTestingMultiCloud-Function Application ID ,57650788-dae5-416f-9da8-792b4ebbbb29 App owner tenant ID, Resource,Azure Storage Resource ID ,e406a681-f3d4-42a8-90b6-c2b029497af1 Resource tenant ID, Resource owner tenant ID,f8cdef31-a31e-4b4a-93e4-5f571e91255a Home tenant ID, Home tenant name, IP address, Location,", , " Status,Success Sign-in error code, Failure reason,Other. Conditional Access,Not Applied Scenario 2 In this scenario, we introduced a network-level block using the Storage Account Firewall while keeping the permissions intact. Entra ID logs still show a successful token issuance because the identity is valid and the scope is broad. However, the storage resource logs don’t log the request. Since the connection was dropped at the network layer before reaching the storage service plane, there is no “Access denied” event in the resource’s audit log. Sample log entry from Entra ID Plain Text Date (UTC): 2026-05-02T19:35:10Z Service principal name: SureshTestingMultiCloud-Function Application: SureshTestingMultiCloud-Function Resource: Azure Storage Status: Success Sample log entry from StorageBlobLogs 0 results for the KQL query: SQL // Query to check for any recorded activity after the network block StorageBlobLogs | where TimeGenerated > ago(1h) | where RequesterObjectId == "00daa177-96c6-4b29-9a5c-53ca603565e9" | project TimeGenerated, OperationName, StatusCode, StatusText, CallerIpAddress, Uri | sort by TimeGenerated desc This result shows that a network-level block is not visible in the resource layer. The Azure administrator sees a successful token issuance in Entra ID but nothing in StorageBlobLogs. To detect this, security teams need to go beyond resource-layer logs and enable additional logging layers such as NSG Flow logs or Defender for Storage - these are outside the scope of this experiment. Comparison scenarioawsazure Identity model Account-scoped Tenant scoped Who gets audit logs? (when available and enabled) Caller-side and resource-owner side (Scenario 1 only) Resource-owner side only Where are the audit trails located? CloudTrail is the logging service. CloudTrail logs are distributed across Caller account, the resource account, and the VPC-owning account Token issuance logs are in the Tenant (Entra ID) while resource access logs are in the Subscription Caller-controlled metadata visible? Yes, visible in caller account and resource account Yes, but included in resource account only What a network-layer block produces When using VPC endpoint policy, AwsVpceEvent is produced and is routed to the VPC-owner account. No logs in resource-owner account. No resource-layer log entry. Identity context in resource logs Full caller identity context included Only the caller ID in the form of RequesterObjectId. An operator must correlate this ID with service principal ID in Entra ID logs. Mitigation We saw that in AWS, CloudTrail can be configured to send log events on both the caller account and the resource account. An attacker can use this information to silently exfiltrate small amounts of data at a time. To mitigate this attack vector, an organization must: Run their compute services in an Amazon VPC — preferably in a private subnet, and Use VPC endpoints with endpoint policies [4] to access their AWS resources for the compute services. The endpoint policies must allow trusted identities to access the resource while blocking everything else. AWS already documents these controls in [1], but these experiments show how important it is to enforce these controls. This is in addition to all the controls that an organization already uses, such as Service Control Policies and Resource Control Policies — those policies control the maximum permissible action that can be taken by an identity/resource but do not control the CloudTrail logging behavior. While Azure doesn’t have the above attack vector specifically, it has a different problem — an operator must manually correlate Entra ID events with the resource event. An example would be an “identity journey” like — managed identity (like the Azure function) requests a token, then writes to a storage account. Therefore, some tooling must be built to correlate such events — for example, routing both ManagedIdentitySignInLogs and StorageBlobLogs into a single Log Analytics workspace is a minimum. Additionally, logs must be captured at different layers such as NSG flow logs/Defender for Storage that can provide anomaly detection beyond standard diagnostic logs. Conclusion In this article, we demonstrated how the same access request produces different results in AWS and Azure. In AWS, access logs were sent to the resource account or the VPC account depending on where the deny decision was enforced, while in Azure, access logs were only sent to the resource account. We saw that this difference comes from how each cloud provider fundamentally treats identities and resources. The implications of the experiment are that security teams in multi-cloud environments cannot assume that audit coverage works the same way across providers. Each provider models their identities and provides different data perimeter controls. Before designing data perimeter controls, security teams must understand each provider’s logging architecture and its differences. References [1] https://aws.amazon.com/identity/data-perimeters-blog-post-series/ [2] https://systemweakness.com/a-subtle-audit-log-consideration-in-aws-063752150b20 [3] https://docs.aws.amazon.com/controltower/latest/userguide/what-shared.html [4] https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-access.html [5] https://learn.microsoft.com/en-us/azure/storage/common/shared-key-authorization-prevent?tabs=portal

This guide explains zone-aware routing from a Kubernetes-first point of view. It covers: why zones matter in cloud platformswhich topology labels Kubernetes places on nodeshow Kubernetes first tried to solve locality through Servicewhat gaps remained after those Service-based featureshow Gateway API implementations such as Envoy Gateway and kgateway built on top of that foundation Why Zones Matter In cloud platforms, a zone is a logical failure domain inside a region. Zones usually have low-latency networking within the zone, but crossing zones can increase both latency and cost. That cost is not theoretical. AWS documents that traffic within the same Availability Zone is free, while traffic that crosses Availability Zones typically incurs data transfer charges, and cross-zone transfer is generally billed in both directions, so a single round trip can be charged twice. See: AWS Architecture Blog: Overview of Data Transfer Costs for Common ArchitecturesAmazon EC2 pricing: Data Transfer This is one reason distributed systems try to keep traffic local when they can, while still preserving failover to other zones. The Topology Information Kubernetes Already Has Kubernetes did not start by inventing zone-aware traffic policies. It started by carrying topology information on nodes. The two most important well-known labels are: topology.kubernetes.io/regiontopology.kubernetes.io/zone According to the Kubernetes reference, these labels are populated on Node objects by the kubelet or the external cloud-controller-manager when the cluster is integrated with a cloud provider. In non-cloud environments, operators can set them manually if the topology model still makes sense. Reference: Kubernetes well-known labels: topology.kubernetes.io/zone In managed clusters, these labels are commonly present by default. Here is the kind of node data Kubernetes typically exposes: YAML apiVersion: v1 kind: Node metadata: name: ip-10-0-12-34.ec2.internal labels: kubernetes.io/hostname: ip-10-0-12-34.ec2.internal topology.kubernetes.io/region: us-east-1 topology.kubernetes.io/zone: us-east-1a That topology data is useful for scheduling, spreading replicas, volume placement, and eventually traffic routing. The Original Service Model The original Kubernetes Service abstraction solved a different problem first: stable discovery and virtual IPs for ephemeral Pods. At the beginning, the model was simple: a Service selected a set of Podskube-proxy programmed forwarding rulestraffic could be sent to any healthy endpoint behind the Service That was excellent for reachability and abstraction, but it had no built-in notion of zone locality. The gap was straightforward: the Service abstraction knew which endpoints existed, but not that a client in zone-a should usually prefer endpoints in zone-a. Kubernetes' First Attempts to Improve Locality Through Services Kubernetes gradually added locality-aware behavior on top of Service, mostly by improving how endpoint selection works. Internal Traffic Policy One early mechanism was internalTrafficPolicy: Local. This tells kube-proxy to use only node-local endpoints for cluster-internal traffic. Example: YAML apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app ports: - port: 80 targetPort: 8080 internalTrafficPolicy: Local Reference: Kubernetes Service Internal Traffic Policy This helps with node locality, but it is not zone-aware routing. Its limitations are important: it is node-local, not zone-localif a node has no local endpoint, the Service behaves as if it has zero endpoints from that node's perspectiveit is too strict for many multi-zone workloads that want zonal preference, not node affinity So this was useful, but it did not really solve multi-zone locality. Topology Aware Routing With Services Kubernetes next introduced Topology Aware Hints, now called Topology Aware Routing. This works through two components: The EndpointSlice controller looks at endpoint and node topology.kube-proxy consumes hints from EndpointSlices and prefers endpoints closer to the client zone. Historically, the Service-side configuration was commonly exposed through the service.kubernetes.io/topology-mode: Auto annotation: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: backend ports: - port: 80 targetPort: 8080 Conceptually, the flow looks like this: This was Kubernetes' first real zone-aware answer at the Service layer. It is useful historical context, but it is no longer the clearest Service-level API to emphasize for new users. Traffic Distribution Preferences Kubernetes later added trafficDistribution as a clearer way to express routing preferences. In current Kubernetes documentation, the relevant zone-level preference is: PreferSameZone The older PreferClose name is documented as deprecated in favor of PreferSameZone, though you may still see PreferClose in some provider and implementation docs that have not yet caught up. Example: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend spec: selector: app: backend ports: - port: 80 targetPort: 8080 trafficDistribution: PreferSameZone Reference: Kubernetes Service trafficDistribution This is a better API shape than older annotations because it is explicit in the Service spec and described as a preference rather than a strict guarantee. In practice, that means current Kubernetes guidance emphasizes trafficDistribution: PreferSameZone, while the older topology-mode: Auto path is best understood as part of the feature's evolution. What Gap Remained After Service-Based Locality Kubernetes Services improved a lot, but they still left several gaps. The Behavior Is Best Effort Topology-aware routing is not a hard guarantee. Kubernetes documents multiple safeguard cases where the system falls back to cluster-wide routing. Examples include: too few endpointsimpossible balanced allocationmissing topology labels on one or more nodesmissing hints for one or more endpointsno hinted endpoint for the local zone That is correct for safety, but it means the behavior is heuristic and conditional. It Assumes a Certain Traffic Shape Kubernetes explicitly documents that Topology Aware Routing works best when traffic is roughly evenly distributed and when there are enough endpoints per zone. If most traffic originates from one zone, local subsets can overload while the global service still looks healthy. It Is Scoped to the Service Datapath This is the most important architectural gap. Service-level topology features influence how kube-proxy chooses endpoints for Service traffic. They do not automatically solve every higher-level data plane. In particular, they do not by themselves define: how an L7 gateway proxy should understand its own zonehow an Envoy-based gateway should configure locality-aware upstream load balancinghow a gateway controller should express stricter local preference versus simple best-effort localityhow policy should attach to particular routes, gateways, or backends That left room for Gateway API implementations to expose richer locality controls. Why Gateway API Implementations Stepped In Gateway API is intentionally expressive and extensible. It standardizes core routing objects, but implementations often add policy CRDs to expose features that are specific to their data plane. That distinction matters here: Gateway API itself does not define one universal, cross-implementation zone-aware policy. Instead, it gives implementations room to expose locality behavior in a way that matches their proxy and control-plane design. Reference: Gateway API overview This is where zone-aware routing became more explicit at the gateway layer. Instead of relying only on kube-proxy's Service behavior, gateway implementations can: understand the proxy's own localityread backend endpoint localityconfigure the underlying proxy's load balancer directlyexpose locality policies as route or backend-attached configuration Example of How Envoy Gateway Addresses the Gap Envoy Gateway supports two paths: Reusing Kubernetes Service-level locality such as Topology Aware Routing or trafficDistributionConfiguring zone awareness directly through BackendTrafficPolicy Reference: Envoy Gateway zone-aware routingEnvoy zone-aware routing Example BackendTrafficPolicy: YAML apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: zone-aware-routing spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: zone-aware-routing loadBalancer: type: RoundRobin zoneAware: preferLocal: minEndpointsThreshold: 1 force: minEndpointsInZoneThreshold: 1 That is a meaningful step beyond plain Service because the gateway layer is now explicitly participating in locality-aware upstream balancing. Example of How kgateway Addresses the Gap kgateway takes a similar approach in spirit: proxy locality is made explicit, and backend load-balancing behavior is configured through policy rather than relying only on Service heuristics. At a high level, kgateway combines: Gateway proxy locality configurationBackend-attached load-balancing policyNative Envoy locality-aware upstream load balancingEndpoint locality metadata that Envoy can use directly Architectural Summary The progression looks like this: Kubernetes Service solved stable discovery and reachability.internalTrafficPolicy improved node-local routing, but not zonal routing.Topology Aware Routing and trafficDistribution added zone-aware preferences to the Service datapath.Gateway API implementations extended the model so L7 gateways and proxies could make explicit locality-aware decisions themselves. Practical Takeaways Kubernetes already provides the topology metadata needed for zone-aware decisions.Service-native locality is useful, but it is heuristic and scoped to the Service datapath.Zone-aware traffic for gateways usually needs the gateway implementation to understand locality too.Modern Gateway API implementations fill that gap by attaching locality-aware load-balancing policy closer to the L7 data plane. Where Zone-Aware Routing Matters in Practice Zone-aware routing usually becomes worth the added operational attention when one or both of these are true: The workload has a tight latency budget, especially at p95 or p99The system moves enough east-west traffic that even a small per-GB cross-zone charge becomes material Common examples include: Gaming platforms, where matchmaking, player session state, inventory, and real-time coordination are sensitive to a few extra milliseconds of network delayFinancial services, where payment, quote, fraud, or checkout paths care more about predictable tail latency than average latencyLarge SaaS and enterprise control planes, where a gateway fans out to many internal APIs and the aggregate cross-zone traffic becomes a real monthly costAI inference, media delivery, logging, and telemetry pipelines, where payload sizes are large enough that bandwidth cost matters even when latency is less critical Worked Example: Multiplayer Gaming Backend Suppose a regional game API runs gateway proxies and backend pods in three zones. Players connect to a gateway in zone-a, and that gateway calls a player-state service that is also deployed in zone-a, zone-b, and zone-c. Assume the following: 25,000 requests per second reach the player-state service from zone-athe combined request and response payload is about 40 KiB per callcross-zone traffic is billed at a representative $0.01 per GBwithout zone awareness, only about one third of those calls stay in zone-a, while the other two thirds go to zone-b or zone-c Actual billing varies by provider, region, and direction of transfer, but the point of the example is that a seemingly small per-GB rate compounds quickly on hot service paths. That means the traffic volume from zone-a to the player-state service is about: 25,000 x 40 KiB per second, or roughly 1 GB/s totalif two thirds of that traffic crosses zones, that is about 0.67 GB/s of cross-zone trafficover a 30-day month, that is about 1.7 million GBat $0.01 per GB, that is about $17,000 per month in cross-zone transfer for just that one service path That is the cost side. The latency side can matter even more for the player experience. If each cross-zone hop adds only 1-3 ms, a request path that fans out to several internal services can add multiple milliseconds of extra tail latency. For a gaming workload, that can affect: matchmaking responsivenesssession join timethe smoothness of player state or presence updateshow stable the system feels during traffic spikes and retries This is why zone-aware routing is not only a cost optimization. In some industries, it is a user-experience and SLO control. Worked Example: Large SaaS Control Plane The same logic applies outside gaming. Consider a large enterprise SaaS platform where each incoming API request hits a gateway and then fans out to an auth service, tenant metadata service, feature-flag service, and audit pipeline. Even if each individual backend call is small, the gateway can generate a large amount of aggregate east-west traffic. In that kind of system, zone-aware routing helps in two ways: it removes avoidable cross-zone traffic from the steady-state hot pathit reduces the chance that a multi-hop request burns several extra milliseconds just on internal network distance For that kind of platform, the business case is usually a combination of lower regional data-transfer cost, tighter latency distributions, and better failure-domain alignment. Conclusion Zone-aware routing is the story of a single idea moving down the stack. Kubernetes started with topology labels on nodes, then taught the Service datapath to prefer local endpoints through internalTrafficPolicy, Topology Aware Routing, and trafficDistribution. Those features are valuable, but they are best-effort and they stop at the Service boundary, which leaves L7 gateways unable to reason about their own locality. Gateway API implementations such as Envoy Gateway and kgateway pick the idea up from there, making proxy locality explicit and pushing locality-aware load balancing into Envoy where it can act on real endpoint metadata. The practical guidance is short. Start with the Service-native controls, because they are simple and often enough. Reach for gateway-level locality policy when you have a tight tail-latency budget, or enough east-west traffic that cross-zone transfer becomes a line item you can see. In both cases, the goal is the same: keep traffic local when you safely can, and fail across zones when you must. Further Reading Kubernetes ServiceKubernetes Topology Aware RoutingKubernetes Service Internal Traffic PolicyKubernetes well-known topology labelsGateway API overviewAWS Architecture Blog: Data transfer costs