DZone Spotlight

We built an LLM pipeline to help a large network operations team stay on top of trouble tickets. It ran quietly in production until the moment it was supposed to earn its keep. In early 2026, a major winter storm swept across a wide region and knocked out power to more than a million people; network equipment failed in bulk, tickets poured in, and the summarizer meant to help engineers triage the chaos went dark. The root cause was not a bug in the usual sense. There was no null pointer and no bad deploy. We hit the Azure OpenAI tokens-per-minute (TPM) limit, our retries made it worse, and we had no fallback. This is the anatomy of that failure, and the architecture we built afterward to treat an LLM like the rate-limited, non-deterministic dependency it actually is. The uncomfortable theme up front: our system was busiest during precisely the event it existed to handle. Demand and failure were correlated. If you put an LLM in front of any incident-driven workload, this will eventually be your story too. What the System Did Tickets in this environment originate from many channels, including network alarms, customer calls, emails, and proactive checks by operations staff. But by the time our pipeline sees them, they are already incidents and cases in ServiceNow. Our scope starts there. ServiceNow streams ticket events out of the box through Stream Connect into Kafka. Our application, running in Azure and orchestrated with LangGraph, consumes those events, retrieves related context from Azure AI Search, and calls the Azure OpenAI API to produce three kinds of summary: Status notifications for the customers affected by an outage,Ticket summaries for the technicians actively working a ticket, andExecutive summaries that roll up what is happening across a region. The value is simple. Ticket logs are long, noisy, and full of machine-generated entries. A technician picking up a ticket, or a manager gauging the blast radius of an outage, does not want to read pages of log. They want five sentences. The LLM gave them five sentences, and on a normal day it sat comfortably within quota. The Failure Timeline Then the storm hit. Equipment failed in bulk. The storm drove power outages past a million customers across a wide region, and our network equipment failed along with the grid. The alarm systems did exactly what they were designed to do: they fired, in volume.Tickets surged. They grew to roughly six times our baseline.The token load surged far faster. This is what caught us. Our load is not measured in requests; it is measured in tokens. Storm tickets did not just arrive more often, each carried a longer log (more alarms, more correlated events). So, a ~6× jump in tickets became closer to a ~15× jump in tokens per minute.We hit the TPM ceiling. Azure OpenAI began returning 429 Too Many Requests with a Retry-After header.Retries deepened the throttle. Every layer that could retry, did. That included the SDK, our wrapper, and LangGraph nodes re-running on failure, all in near-unison, with no jitter. Each retry wave slammed the limit together and pushed our effective token rate higherwhile we were already over budget. And because every retry of a generation is another paid, token-billed call, the retries spent the very budget we had blown..There was no fallback. When retires were exhausted, there was nowhere to go. There was no cheaper model and no degraded path. Summarization simply stopped The cascade. Customer status notifications stalled, technicians lost the ticket summaries they rely on, and executive summaries went stale. So, the team fell back to reading raw logs by hand. Summarization stayed degraded, on and off, for a multi-hour stretch, until we manually provisioned extra capacity and hand-routed traffic to other models to limp through the worst of it. The shape of the overload, with illustrative numbers to make the dynamic concrete: Metric Normal day Storm Summaries per minute ~40 ~240 (≈6×) Tokens per summary (log + context + output) ~3,300 ~8,000 (longer logs) Token demand ~132K TPM ~1.9M TPM Token quota ~250K TPM ~250K TPM Result ~53% utilization ~7.7× over → sustained 429s (The figures are illustrative estimates that preserve the real proportions, not exact production measurements.) The punchline is in the third row: a ~6× rise in tickets became a ~15× rise in tokens. That is the trap of a token-metered dependency, and the rest of this article is what it taught us. How the original outage cascaded — and why naive retries made it worse. Root Cause: An LLM is a Token-Metered Dependency, Not a Request-Metered One Most writing on resilience, including circuit breakers, retries, and bulkheads, is framed around microservices, and most of it applies here. But an LLM API breaks a few assumptions those patterns quietly rely on, and each broken assumption showed up in our incident. 1. The limit is tokens, not requests. Classic rate-limit thinking counts calls; Azure OpenAI quota is measured in tokens per minute. Your load therefore depends on the size of your inputs — and for a summarizer that is the worst possible coupling: it burns the most quota exactly when documents are longest, which during an incident is exactly when logs are longest. A request-rate dashboard would have looked merely elevated while our token rate was off the chart. 2. Retries spend the budget you are already over. On a normal REST API, a retry is cheap. On a token-metered, pay-per-token backend, every retried generation is another full charge against the limit you just exceeded. Naive retries do not just fail to help. They actively deepen the throttle. 3. Synchronized retries are a self-inflicted DDoS. With no jitter, failed calls backed off by the same amount and returned together, re-tripping the limit on a clock. It is the classic retry storm, amplified by point #2 because each retry is token-expensive. 4. No fallback means peak demand is a single point of failure. One model, one deployment, one path is fine until that path is throttled, and it will be throttled at peak. 5. Demand correlates with failure. A summarizer for incident tickets is, by definition, busiest during incidents. The load spike and the operational emergency are the same event. Capacity planned for the average is capacity planned for the calm before the thing you actually built the system for. The Fix: Classify, Route by Severity, and Govern the Token Budget The redesign treats the LLM as a scarce, metered resource and spends it deliberately, turning the frantic, manual capacity-adding and model-rerouting we did by hand during the storm into a permanent, automatic capability. Schedule in Redis, not Kafka. Our Kafka topics are shared by many interfaces and kept generic, so we could not repurpose them for prioritization. Instead, our consumer reads the generic stream and pushes work into Redis priority queues, where all the scheduling logic lives. Kafka stays the durable ingestion layer — a natural backpressure buffer, so a storm surge piles up safely in the log instead of hammering the model, and consumer lag becomes our early-warning storm metric. Classify with a tiny model. A small, local ML classifier scores each ticket by priority, severity (P1–P5), and customer impact, using fields already on the ServiceNow ticket. It is deliberately not an LLM call: during a storm every Azure OpenAI token is contested, so spending premium tokens just to decide how to spend premium tokens is exactly backwards. When the classifier is unsure, it routes up, because under-serving a real P1 is far worse than over-spending on a P4. Route by severity to isolated capacity. Each tier gets the cheapest treatment that still meets its need: Severity Routes to Why P1 / P2 Premium model deployment (related incidents coalesced into one regional rollup) High stakes, exec-facing; worth the tokens P3 / P4 Separate, cheaper model deployment "Good enough" at a fraction of the tokens P5 Non-LLM extractive summary (error counts, key fields, first/last events) Zero tokens; also a universal degraded mode The key trick is that the cheaper tier is a different model, so it draws from a different Azure OpenAI quota pool — a flood of low-severity tickets cannot cannibalize the premium tier's TPM. This needs no provisioned throughput; two standard deployments on different models give you quota isolation for free. Govern the token rate. A shared, Redis-backed token budget gates every LLM call: we estimate a job's tokens before dispatch and only proceed if the rolling per-minute budget allows, per deployment. Retries use bounded exponential backoff with jitter and honor Retry-After; the first worker to see a 429 sets a global cooldown the whole fleet respects, so the retry storm cannot form. Low-priority queues age and get promoted so they are never starved, and at-least-once delivery is made safe with idempotency keyed on ticket plus log version. Put together, the request path becomes: ServiceNow → Stream Connect → Kafka → classifier → Redis priority queue → token governor → the right model (or extractive fallback). The queue absorbs the spike, the governor respects the ceiling, and severity routing decides who gets the scarce premium tokens when there are not enough to go around. The redesigned pipeline: tickets are classified by severity, scheduled through Redis with a token governor, and routed to isolated model tiers. What We Expect (By Design) With this in place, the same storm should behave very differently. The 429 cascade cannot recur by construction. The governor caps dispatch at quota, so overflow becomes bounded queue lag — low-priority summaries delayed by minutes — rather than total failure.Premium capacity is protected. Routing roughly the top 15% of tickets to the premium tier and coalescing related incidents keeps it within quota even under the surge.Cost falls. Moving the bulk of volume to a cheaper model and the long tail to zero-token extraction projects on the order of a 50–65% blended token-cost reduction. Takeaways Plan capacity in tokens, not requests: Your load is driven by input size, which spikes exactly when you can least afford it.Design for the spike, not the average: Assume demand correlates with failure.Make retries jittered, bounded, and 'Retry-After'-aware: Remember each retry costs tokens.Tier your models by importance: Put cheap or non-LLM paths under the long tail, and isolate premium capacity on its own quota pool.Always keep a degraded mode: A rough summary delivered beats a perfect one that never arrives.

Logging is one of the oldest practices in software engineering, yet in distributed systems it remains one of the most poorly implemented. Most teams log, but very few log well. The gap between having logs and having useful logs becomes painfully visible the moment a production incident occurs at 2 AM across a system running dozens of microservices. This article focuses on structured logging: what it is, where teams consistently go wrong with it, and the concrete practices that separate log data you can actually act on from log noise that burns engineering hours during incidents. If you are building or operating distributed systems today, structured logging is not optional. It is the foundation on which every other observability signal- traces, metrics, alerts- depends. What Structured Logging Actually Means Structured logging means emitting log entries as machine-readable key-value pairs rather than arbitrary free-text strings. Instead of this: Plain Text [ERROR] 2026-07-10 03:14:22 - Failed to process payment for user 84729, reason: timeout You emit this: JSON { "timestamp": "2026-07-10T03:14:22Z", "level": "error", "service": "payment-service", "event": "payment_processing_failed", "user_id": 84729, "reason": "timeout", "duration_ms": 3001, "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7" } The difference sounds cosmetic. It is not. The first format requires regex parsing and string matching to extract meaning. The second is immediately queryable, aggregatable, and, crucially, correlatable with traces and metrics from other services handling the same request. The Five Mistakes Distributed Systems Teams Make With Logs 1. Logging Without Context Propagation In a monolith, a single log line tells you where in the codebase an event occurred. In a distributed system, a log line without a correlation identifier tells you almost nothing. If Service A calls Service B which calls Service C, and Service C fails, you need a shared identifier, typically a trace ID, that threads through all three services' logs so you can reconstruct the full request journey. The fix is context propagation: passing a trace ID through every request, injecting it into every log entry, and configuring your logging library to include it automatically. In practice, this means integrating your logging setup with OpenTelemetry or a similar tracing framework from day one, not as an afterthought. When your log entries include trace_id and span_id fields, you can jump from a log entry to its full distributed trace in a single query; that capability compresses incident diagnosis from hours to minutes. 2. Inconsistent Field Naming Across Services In a microservices architecture developed by multiple teams, field-naming inconsistencies compound into a real problem at scale. One service logs user_id, another logs userId, a third logs uid. One service logs errors under error, another uses err, another uses exception. When you need to query across services during an incident, this inconsistency forces per-service query variations, slowing everything down. Establish and enforce a logging schema across your organization. Define a canonical set of field names for common concepts, user identifiers, request identifiers, error fields, latency fields, and make that schema part of your service standards. Libraries like structlog in Python or logrus/zap in Go make it straightforward to enforce common fields at the logger initialization level, so teams can't easily deviate from the schema accidentally. 3. Logging at Wrong Severity Levels Severity level misuse is endemic. INFO logs that should be DEBUG. Application errors logged as WARN because the developer did not want to trigger alerts. Business logic exceptions logged as ERROR when they are expected and handled. Over time, this degrades the signal value of severity levels to the point where teams stop filtering by level entirely. Adopt and document clear severity semantics for your organization: DEBUG: information useful only during active development; should not run in productionINFO: normal operational events (service started, request received, job completed)WARN: unexpected conditions that are recoverable and do not require immediate actionERROR: failures that require investigation; every ERROR should eventually be investigated or suppressed with documented justificationFATAL: unrecoverable failures; service cannot continue Treat severity levels as a contract with your future on-call self. 4. Over-Logging Hot Paths High-throughput services that log every incoming request at INFO level generate enormous log volumes that create three problems: storage costs escalate, log search performance degrades, and genuinely important events get buried in noise. A service processing 10,000 requests per second generates over 860 million log lines per day from request logging alone. Use sampling for high-frequency, low-severity log events. Most observability platforms and log monitoring tools support log sampling natively; you configure a sampling rate for specific log patterns, keeping representative data without keeping everything. For example, sample 1% of successful payment processing logs but keep 100% of error logs. This dramatically reduces volume while preserving signal fidelity where it matters. 5. Treating Logs as a Standalone Signal Logs become exponentially more powerful when they are correlated with traces and metrics. A spike in error logs is interesting. An error log spike correlated with a latency metric increase correlated with a trace showing a database connection timeout is actionable in seconds. Teams that treat logs as independent from their other observability signals are leaving significant diagnostic capability on the table. If you are not already running OpenTelemetry, start there. It provides a unified SDK for instrumenting logs, traces, and metrics in a way that ensures they carry shared context identifiers. Once your logs carry the same trace IDs as your distributed traces, your observability signals become correlated by default, not by manual investigation. A Practical Logging Schema to Start With Here is a minimal structured logging schema that covers the majority of production use cases across distributed services: JSON { "timestamp": "ISO-8601 UTC", "level": "debug|info|warn|error|fatal", "service": "service-name", "version": "1.4.2", "environment": "production", "event": "snake_case_event_name", "message": "Human-readable description", "trace_id": "OpenTelemetry trace ID", "span_id": "OpenTelemetry span ID", "user_id": "optional", "request_id": "optional", "duration_ms": "optional, numeric", "error": { "type": "TimeoutError", "message": "Connection timed out after 3000ms", "stack": "optional, omit in high-volume paths" } } This schema is opinionated but extensible. Services add domain-specific fields as needed while every entry maintains the common fields that make cross-service correlation possible. Conclusion Structured logging in distributed systems is not about logging more; it is about logging intentionally. The practices that separate teams who resolve incidents in minutes from teams who spend hours in log archaeology come down to four things: consistent field naming, trace context propagation, disciplined severity usage, and treating logs as a correlated signal rather than an isolated one. Get these right, and your logs become a first-class observability asset during incidents. Get them wrong, and you have the worst of both worlds: high storage costs and low diagnostic value. The patterns outlined here are not theoretical; they are the difference between incident response that feels like detective work and incident response that feels like reading a timeline.

Every on-call engineer understands this situation well. An alert fires at 2 a.m., engineers spend the first five minutes figuring out what it means, the next few minutes searching Confluence for the relevant runbook, and finally start doing something useful. By that point, an automated system that could have classified the alert and retrieved the right procedure, proposed a remediation plan, and opened a ticket in thirty seconds has saved you nothing because it didn’t exist. That’s the problem this article addresses. We are going to build a working incident triage agent using .NET 10 and .NET Aspire 9 that does exactly that chain of steps automatically. The agent receives an HTTP alert payload, which classifies it using a Groq-hosted LLM, retrieves the matching runbook section from a Qdrant vector store, asks the LLM to propose remediation steps, and escalates to PagerDuty (through a local stub). If the severity warrants it, it writes a full audit record. The system will automatically follow all these without human involvement. What makes this integration not have any single component? It’s the combination of MCP as the tool contract, Aspire as the wiring layer, and a small eval harness that prevents the agent from quietly drifting over time. Let's go through how it's built. What are we Actually Solving? Before we bring AI into this, let’s be honest about what the actual problem is because “AI for incident triage” sounds impressive but means nothing without a clear picture of what exactly the AI is doing. When an alert goes out, the on-call engineer has three jobs Is this serious? (figuring out the severity before doing anything else)What do I do about it? (find the right procedure and follow it)Who else needs to know? (escalate the right people and open a ticket) Here are the things: Job one is mostly pattern matching, Job two is where it gets interesting, and Job three is completely mechanical but needs to be well understood, including failure modes, database connection pool exhaustion, memory leaks, and disk pressure. So, the answer is already written down somewhere in your runbook. Because the engineer is not thinking; they are searching. An LLM is genuinely good at steps one and two when given the right context. It can classify alerts and turn a runbook excerpt into a clear list of actions. The tricky part is making sure it gets the right runbook excerpt in the first place. If you ask an LLM to fix a memory leak without giving it the memory runbook, you’ll get a generic answer. So, give the right context, and you get something genuinely useful. Solution Architecture The solution is split into six focused .NET Aspire projects. Each project has a single, well-defined responsibility. App Host is the entry point to run. It doesn’t serve HTTP traffic or business logic. Its only job is to tell Aspire what services exist, which one needs to start, and which configuration needs to be injected into each of them. Think of it as the framework that describes the whole system. Services Defaults is a shared library that all other projects reference. It sets up the things every service should have, including structured logging with Serilog, distributed tracing, health check endpoints, and service discovery. So, these are all wired up with a single builder.AddServiceDefaults() call and never think about it again. Agent Service is the front door. It exposes one endpoint POST/triage and drives the five-step pipeline from start to finish. It doesn’t classify alerts, talk to Qdrant, and doesn’t know what pager duty is. It just calls the right tools in the right order and assembles the final response. MCP Tool Server is where the actual work happens. It hosts four MCP tools (alert classification, runbook lookup, PagerDuty escalation, and audit writing) and exposes them over HTTP using the Model Context Protocol. The Agent Service calls these tools by name without knowing anything about their internal implementation. PagerDuty Stub is a throwaway stand-in for the real PagerDuty API. In development, you do not want to fire real pagers or need a PagerDuty account just to test the escalation step. The stub accepts the same payload, logs it, and returns a synthetic ticket. Swap it for the real endpoints in production by changing one config value. Evals Harness is a safety check. It fires six carefully chosen alerts at the live agent and checks that the responses match expectations. If fewer than five pass, the process exits with a non-zero code, and your continuous integration pipeline fails. It is the thing that tells you when a model update or a config change has quietly broken something. The data flow for a single alert looks like this. The AgentService and McpToolServer are deliberately separate processes. The agent knows nothing about embeddings, Qdrant, or PagerDuty. It only knows how to call MCP tools by name. This is the core benefit of MCP. In the future, if we update the MCP server, the agent doesn’t change at all. MCP Tool Server The McpToolServer is an ASP.NET Core minimal API that exposes four tools over the MCP streamable HTTP transport. Each tool is a static class annotated with `McpServerToolType` and `McpServerTool`. C# [McpServerToolType] public static class AlertClassifierTool { [McpServerTool, Description("Classify an alert and return severity, category, and confidence.")] public static async Task<AlertClassification> ClassifyAsync( [Description("The raw alert text to classify")] string alertText, IChatClient chatClient, ILogger<AlertClassifierTool> logger, CancellationToken ct) { var prompt = $""" You are an incident classifier. Classify the following alert: {alertText} Respond with JSON only: {{ "severity": "Critical|High|Medium|Low", "category": "short category label", "confidence": 0.0-1.0, "reasoning": "one sentence" } """; var response = await chatClient.GetResponseAsync(prompt, new ChatOptions { ResponseFormat = ChatResponseFormat.Json }, ct); return JsonSerializer.Deserialize<AlertClassification>(response.Text) ?? throw new InvalidOperationException("LLM returned empty classification"); } } The `IChatClient` and `ILogger` parameters are injected by the MCP framework via ASP.NET Core’s dependency injection container. The tool itself is stateless, a plain static method. This keeps unit testing straightforward and allows you to pass in a mock `IChatClient`, call the method, and assert on the result. The `RunbookLookupTool` follows the same pattern but takes an `IEmbeddingGenerator<string, Embedding<float>>` and a `QdrantClient` instead of a chat client. C# [McpServerTool, Description("Find the most relevant runbook excerpts for a given incident category.")] public static async Task<List<RunbookExcerpt>> LookupAsync( [Description("Incident category from classification")] string category, IEmbeddingGenerator<string, Embedding<float>> embedder, QdrantClient qdrant, IConfiguration config, CancellationToken ct) { var topK = int.Parse(config["Qdrant:TopK"] ?? "3"); var colName = config["Qdrant:CollectionName"] ?? "runbooks"; var embedResult = await embedder.GenerateAsync([category], cancellationToken: ct); var vector = embedResult[0].Vector.ToArray(); var hits = await qdrant.SearchAsync(colName, vector, limit: (ulong)topK, cancellationToken: ct); return hits.Select(h => new RunbookExcerpt( Title: h.Payload["title"].StringValue, Content: h.Payload["content"].StringValue, Score: (float)h.Score)).ToList(); } The vector query uses cosine similarity, so (high memory usage on API node) still finds the memory-pressure runbook even though the wording doesn’t match. The embeddings capture semantic meaning, not keyword overlap. Custom Embeddings with Nomic AI Nomic AI `nomic-embed-text-v1.5` model produces 768-dimensional vectors at very low cost. The only catch is that Nomic uses a non-standard API path (`POST/v1/embedding/text` rather than the OpenAI-compatible `/V1/embeddings`), so we can’t use the default OpenAI embedding adapter from `Microsoft.Extensions.AI`. Instead, we implement `IEmbeddingGenerator<string, Embedding<float>>` directly. C# internal sealed class NomicEmbeddingGenerator( IHttpClientFactory httpClientFactory, string model, ILogger<NomicEmbeddingGenerator> logger) : IEmbeddingGenerator<string, Embedding<float>> { public EmbeddingGeneratorMetadata Metadata { get; } = new("nomic", providerUri: null, defaultModelId: model); public async Task<GeneratedEmbeddings<Embedding<float>>> GenerateAsync( IEnumerable<string> values, EmbeddingGenerationOptions? options = null, CancellationToken cancellationToken = default) { var client = httpClientFactory.CreateClient("nomic"); var requestBody = new NomicEmbedRequest(model, values.ToList(), "search_document"); using var response = await client.PostAsJsonAsync( "embedding/text", requestBody, NomicJsonContext.Default.NomicEmbedRequest, cancellationToken); response.EnsureSuccessStatusCode(); var result = await response.Content.ReadFromJsonAsync( NomicJsonContext.Default.NomicEmbedResponse, cancellationToken) ?? throw new InvalidOperationException("Nomic returned an empty response body"); return new GeneratedEmbeddings<Embedding<float>>( result.Embeddings.Select(v => new Embedding<float>(v)).ToList()); } public object? GetService(Type serviceType, object? serviceKey = null) => null; public void Dispose() { } } This class implements the full `IEmbeddingGenerator<string, Embedding<float>>` contract from `Microsoft.Extensions.AI`, so the rest of the codebase, including the `RunbookLookupTool`, sees a standard interface and never needs to know it’s talking to Nomic rather than OpenAI. The `JsonSerializable` source generation at the bottom of the file `NomicJsonContext` is important for trimming-safe serialization and for performance in hot paths. Both the request and response records must be at namespace scope (not nested inside the generator class) for the source generator to work correctly. This is a common mistake that produces `SYSLIB1032` at compile time. The Agent Service The Agent Service is where the triage pipeline is assembled. It uses Semantic Kernel to handle the remediation step (where we need prompt rendering and the injection filter) and calls all other steps via `McpClient.CallToolAsync`. The pipeline in `DotNetAspireTriageAgentService.cs` looks like this. C# // Step 1 — Classify var classification = await _mcpClient.CallToolAsync<AlertClassification>( "ClassifyAsync", new { alertText = payload.AlertText }, ct); // Step 2 — Runbook lookup (skip for Medium/Low) List<RunbookExcerpt> runbooks = []; if (_lookupSeverities.Contains(classification.Severity)) { runbooks = await _mcpClient.CallToolAsync<List<RunbookExcerpt>>( "LookupAsync", new { category = classification.Category }, ct); } // Step 3 — Remediation (via Semantic Kernel for prompt filter support) var proposal = await _kernel.InvokePromptAsync<RemediationProposal>( RemediationPromptTemplate, new KernelArguments { ["alert"] = payload.AlertText, ["runbooks"] = JsonSerializer.Serialize(runbooks), ["severity"] = classification.Severity }, cancellationToken: ct); // Step 4 — Escalate var escalation = await _mcpClient.CallToolAsync<EscalationResult>( "EscalateAsync", new { classification, correlationId = payload.CorrelationId }, ct); // Step 5 — Audit await _mcpClient.CallToolAsync( "WriteAuditAsync", new { classification, proposal, escalation }, ct); Defending Against Prompt Injection Prompt injection is a real concern in agentic systems where user-supplied text ends up literally inside an LLM prompt. An attacker who controls the alert body could try to override the system prompt and redirect the agent’s behavior. Prevent here uses Semantic Kernel’s `IPromptRenderFilter`, which fires after the prompt template is rendered but before the rendered string is sent to the model. C# public sealed class PromptInjectionFilter( InjectionDetectionContext context, ILogger<PromptInjectionFilter> logger) : IPromptRenderFilter { // Matches common injection patterns: "ignore previous instructions", // "disregard your system prompt", role-switching attempts, etc. private static readonly Regex InjectionPattern = new( @"(?i)(ignore\s+(all\s+)?(previous|prior|above)\s+instructions?" + @"|disregard\s+(your\s+)?(system\s+prompt|instructions?)" + @"|you\s+are\s+now\s+(?:a\s+)?(?:an?\s+)?\w+" + @"|act\s+as\s+(if\s+you\s+are\s+)?(?:a\s+)?(?:an?\s+)?\w+)", RegexOptions.Compiled | RegexOptions.CultureInvariant); public async Task OnPromptRenderAsync( PromptRenderContext context, Func<PromptRenderContext, Task> next) { await next(context); // let the template render first if (context.RenderedPrompt is not null && InjectionPattern.IsMatch(context.RenderedPrompt)) { context.RenderedPrompt = InjectionPattern.Replace( context.RenderedPrompt, "[SANITISED]"); this.context.InjectionDetected = true; logger.LogWarning( "Prompt injection attempt detected and sanitised — correlationId={CorrelationId}", context.Arguments["correlationId"]); } } } The filter doesn’t abort the request. It sanitizes the offending text and sets a flag that the agent includes in the response. This is a deliberate choice where failing silently is worse than completing with a sanitized prompt, because a failed triage means a missed escalation. The response `injectionDetected` field lets downstream systems know that something suspicious happened without stopping the pipeline. Handle Everything Together with .NET Aspire The AppHost is where everything comes together. Every service, dependency, and API key is declared in one place. When we run this project, Aspire reads those declarations and automatically starts the entire system in the correct order. C# var builder = DistributedApplication.CreateBuilder(args); // API keys from user-secrets or appsettings.json var groqApiKey = builder.AddParameter("GroqApiKey", secret: true); var nomicApiKey = builder.AddParameter("NomicApiKey", secret: true); // Qdrant container — persisted between restarts var qdrant = builder.AddQdrant("vectorstore") .WithLifetime(ContainerLifetime.Persistent); // PagerDuty development stub var pagerDutyStub = builder.AddProject<Projects.DotNetAspireTriageAgent_PagerDutyStub>( "pagerduty-stub"); // MCP Tool Server — waits for Qdrant and the PagerDuty stub var pagerDutyStubEndpoint = pagerDutyStub.GetEndpoint("http"); var mcpServer = builder.AddProject<Projects.DotNetAspireTriageAgent_McpToolServer>("mcp-tools") .WithReference(qdrant) .WithReference(pagerDutyStub) .WaitFor(qdrant) .WaitFor(pagerDutyStub) .WithEnvironment("Groq__ApiKey", groqApiKey) .WithEnvironment("Nomic__ApiKey", nomicApiKey) .WithEnvironment("PagerDuty__StubEndpoint", ReferenceExpression.Create($"{pagerDutyStubEndpoint}/pagerduty-stub/incidents")); // Agent Service — waits for the MCP server builder.AddProject<Projects.DotNetAspireTriageAgent_AgentService>("agent-service") .WithReference(mcpServer) .WaitFor(mcpServer) .WithEnvironment("Groq__ApiKey", groqApiKey); builder.Build().Run(); Three things in this code are worth understanding properly before moving on. .WithReference() vs .WithEnvironment(): These two look similar but do various jobs. When you call .WithReference(Qdrant), you are telling Aspire to figure out Qdrant’s host, port, and credentials at runtime and automatically inject the full connection string into McpToolServer. We do not need to mention it hardcoded anywhere. ReferenceExpression.Create. This one trips people up the first time. When McpToolServer needs to call the PagerDuty stub, it needs the stub’s full URL including the path (like domain/pagerduty-stub/incidents). The problem is you do not know the port number at the time you write the code; in this case, Aspire assigns it dynamically at startup. So instead of hardcoding a URL that will break on someone else’s machine, for this we write ReferenceExpression.Create($"{pagerDutyStubEndpoint}/pagerduty-stub/incidents") and let Aspire fill in the real address when it starts up. WaitFor This tells Aspire not to start McpToolServer until Qdrant and the PagerDuty stub are fully up and ready. Without it, McpToolServer would try to connect before they are ready and crash on the very first run. Once everything is running, the Aspire dashboard gives you a live view of the whole system. The resources tab shows all four services with their current health status and the URLs Aspire assigned to each one. The graph tab is even more useful when you are onboarding someone new to the project. It draws the exact dependency map you declared in the codebase, which service depends on which, which API keys go where, and how everything connects. Note: if a service fails to start, this graph tells you immediately which dependency in the chain is the problem instead of you having to read through logs across four different console windows. PagerDuty Stub Rather than mocking PagerDuty calls in covers or requiring a real PagerDuty account, the solution includes a lightweight stub service. It is a genuine Aspire project registered in Apphost. C# app.MapPost("/pagerduty-stub/incidents", async (HttpRequest request) => { // ... read and log the body ... var response = new PagerDutyStubResponse( Incident: new StubIncident( Id: correlationId, Status: "triggered", Number: Random.Shared.Next(1000, 9999))); return Results.Created( $"/pagerduty-stub/incidents/{correlationId}", response); }); Because the stub is a real Aspire project, its URL is dynamically allocated by Aspire and injected into McpToolServer via `ReferenceExpression.Create`. This means there are no hardcoded ports that break when someone else is already using that port, and the stub starts and stops with the rest of the solution. Swapping it for the real PagerDuty events API in Production means changing a single config value, the URL injected via `WithEnvironment`. Runbook Seeding on Startup The McpToolServer seeds its Qdrant collection on startup using a hosted service. It checks whether the collection already exists before doing any work, which means subsequent restarts are near-instant. C# public sealed class RunbookSeeder( QdrantClient qdrant, IEmbeddingGenerator<string, Embedding<float>> embedder, IConfiguration config, ILogger<RunbookSeeder> logger) : IHostedService { public async Task StartAsync(CancellationToken ct) { var collectionName = config["Qdrant:CollectionName"] ?? "runbooks"; var exists = await qdrant.CollectionExistsAsync(collectionName, ct); if (exists) { logger.LogInformation("Runbook collection already exists — skipping seed"); return; } await qdrant.CreateCollectionAsync(collectionName, new VectorsConfig(new VectorParams(size: 768, distance: Distance.Cosine)), ct); } } The runbooks test the most common failure categories, including high CPU, memory pressure, database connection exhaustion, disk saturation, network timeout, and pod restart loops. Each is stored as a Qdrant point with title and content payload fields that `RunbookLookupTool` reads back on retrieval. Eval Harness AI systems have a subtle problem that unit tests don’t catch. So, the agent can quietly get worse over time. A model version bumps, someone tweaks a prompt, a config value changes, and suddenly your critical alerts are coming back as medium with no error thrown anywhere. You only find out when a real incident gets missed. The Evals project is the safety net for exactly this. It fires six alert payloads at the live agent and checks that each response matches the expected severity, category, and escalation behavior. If fewer than five pass, the build fails. It is the same idea as a unit test suite, except it is testing the intelligence of the agent, not just the correctness of the code. Key Takeaways .NET Aspire service coordination makes it practical to run a multi-service AI agent system, including a vector database, an MCPToolServer, and an LLM-backed agent, locally with a single `dotnet run` command.The Model Context Protocol (MCP) gives you clean, language-agnostic control for exposing agent tools over HTTP, so the agent and its capabilities can evolve independently without tight coupling.Combining Nomic AI embeddings with a Qdrant vector store lets you attach a runbook knowledge base to an AI agent without fine-tuning a model that will help semantic search retrieve the right production even when the alert wording doesn’t match the runbook text exactly.Groq’s OpenAI-compatible API with `llama-3.3-70v-versatile` provides sub-second structured JSON responses, which is fast enough to complete a full five-step triage pipeline including classify, retrieve, remediate, escalate, and audit in under three seconds on most workloads.Adding a Semantic Kernel `IPromptRenderfilter` to scan every prompt render before it reaches the LLM is a lightweight, zero-overhead way to defend against prompt injection in agentic pipelines. Prerequisites To follow along with the code in this article, you will need: Visual Studio 2026 (17 or later) with the .NET Aspire package installed, or the .NET 10 SDK (10.0.300 or later) if you prefer using a terminal.Docker Desktop (4.x or later) must be running before you start because .NET Aspire automatically starts a Qdrant container.Groq API key (free get from console.groq.com) used for the alert classification and remediation via `llama-3.3-70v-versatile`.Nomic AI API key (free get from atlas.nomic.ai) used for runbook text embeddings via `nomic-embed-text-v1.5`. Note: No cloud subscription is required. Both API keys have generous free quotas that comfortably cover development and testing. Conclusion What we have built is a working blueprint for an AI triage agent that respects software engineering discipline, clean boundaries between components, a tool contract that survives dependency changes, prevents misuse, and a regression harness that makes model-level drift a continuous integration failure rather than a surprise. The combination of .NET Aspires coordination, Mcp tool abstraction, Groq’s low-latency inference, and Nomic embeddings means you can stand up a full agentic pipeline locally, with realistic dependencies, in the time it takes to run `dotnet run`. The development experience matters because it determines how quickly you can experiment, iterate, and validate changes. The next natural extensions are a persistent audit store, a document ingestion pipeline for runbooks, and a feedback loop that uses closed incidents to refine the classification prompts. All three can be added as new MCP tools without changing the agent. Appendix The complete source code for this article, including all six projects, runbook seed data, eval harness cases, and configuration examples, is available in the GitHub repository. You can clone it, run it locally with a single command, and use it as a starting point for your own incident triage pipeline. Full source code is available at the GitHub Repository.

When we started to work on microfrontend migration on one of our projects, the architecture looked great on paper (like always): one host shell, several remote apps, and teams could deploy independently on their own timelines. But in practice it wasn't so clean. One part kept getting on my nerves: actually mounting remote React components inside the host. Each microfrontend came with the same glue code. Load the remote bundle, create a React root, render the component, keep track of the mounted instance, push updated props into it when the host re-renders, and clean up listeners on unmount. And do not forget to handle load failures. It wasn't especially hard code. But it was just the kind of code nobody wants to repeat. Another problem is type safety, which had a habit of disappearing exactly where I wanted it most. Inside the remote, TypeScript understood the component props perfectly. But at the host boundary, that often collapsed into unknown and as any. If a remote added a required prop or renamed an existing one, the host usually did not find out from the compiler. After doing this a few times across different projects, I decided the pattern deserved a real abstraction instead of one more copy-pasted wrapper. What I Wanted It should be part of my toolkit package and shouldn't be really hard. Something much more practical. The goal was simple: Remove repetitive host-side boilerplateKeep prop types across the host/remote boundaryWork with separate bundles and separate React rootsAvoid shared stores, global registries, and code generationFit into an existing Module Federation setup without changing how remotes are versioned or deployed That idea transformed to @mf-toolkit/mf-bridge. The Base The package has two parts: one wrapper on the remote side, and one host component that takes care of the integration. On the remote side, you define the entry once: TypeScript import { createMFEntry } from '@mf-toolkit/mf-bridge/entry' import { CheckoutWidget } from './CheckoutWidget' export const register = createMFEntry(CheckoutWidget) On the host side, you render the bridge where the remote should appear: import { MFBridgeLazy } from '@mf-toolkit/mf-bridge' <MFBridgeLazy register={() => import('checkout/entry').then(m => m.register)} props={{ orderId, userId } fallback={<CheckoutSkeleton/>} /> That’s all. With MFBridgeLazy, the host doesn’t have to deal with all the hassle of loading things on demand, setting up the root, updating stuff, cleaning up, or handling event listeners — the tool does it all. Plus, because the register function has clear types, the host can automatically figure out what props the remote component needs. If the remote component suddenly needs a new prop, you’ll see a TypeScript error right away during development, not after the app is already live and causing problems. How Prop Updates Travel This was the part I wanted to keep as boring and predictable as possible. Once a remote component is mounted, it lives in its own React root. That means the host cannot simply re-render it as if it were a normal local child. The host still needs a way to send updated props into that remote tree every time its own state changes. There are plenty of ways to solve this: shared stores, shared context, global event buses, custom registries. I wanted the smallest possible mechanism that stayed local to each mounted microfrontend. So `mf-bridge` uses the one thing both sides already share: the mount element. When the host re-renders with new props, the bridge dispatches a `CustomEvent` on that specific DOM element. The remote listens to events on that same element and re-renders with the new props. That is it. I like this approach for a few reasons. First, it is naturally isolated. If you have several microfrontend slots on the same page, each one has its own mount element, so updates do not bleed across instances. Second, it does not need a shared module graph or global state container just to move props around. Third, it keeps the contract very explicit: the host owns the mount point, and the props, and the remote owns how it renders them. Internally, the package wraps this in a small typed DOM event bus, but consumers do not really need to think about those details. Why This Helped More Than Just Saving Lines of Code The obvious benefit is less boilerplate. If a page has five remote slots, I no longer end up with five slightly different wrappers all doing the same lifecycle work. But the bigger benefit is moving problems earlier in the process. Before this, the host/remote boundary was often exactly where type information got blurry. That made one of the most important contracts in the system feel surprisingly fragile. A remote could evolve, and the host would not always know it had fallen out of sync. With mf-bridge, prop inference flows from the remote entry to the host usage. That changes the feedback loop. A contract mismatch becomes a compile-time problem instead of an incident report. There is also a reliability benefit in the lifecycle handling. The package takes care of the repetitive, easy-to-forget parts: Lazy loading with a fallback UIClean mount and unmount behaviorProp streaming on re-rendersListener cleanupError handling when the remote fails to loadOptional preloading and retry behaviorOptional hooks for setup and teardown on the remote side when you need DI or per-mount initialization None of these features are individually groundbreaking. The value is that they come together in one small, reusable bridge instead of being re-implemented in every host wrapper. The Cases I Wanted to Be Sure About When the basic version started to work, I spent a bit more time on some of the scenarios that usually make microfrontend wrappers fragile. One of those cases was multiple instances of the same remote on a single page — a widget in the main content area, a compact version in a sidebar, or the same remote mounted in a few different places. I wanted to make sure what updates stayed local to the exact mount point instead of leaking. Using the DOM element itself as the transport turned out to be a very practical way to preserve that isolation. Another important case was failed loading. I didn't want the host to end up with a blank hole in the UI just because a remote bundle failed on the first attempt. That is why the bridge supports fallbacks, preloading, and retry behavior. I think that kind of thing makes an integration feel solid. And sure, we should not forget about what happens when the problem is rendering. If a remote drops during render, I do not want that failure to destabilize the whole host page. So error handling became part of the design too: we keep the failure contained to the mount point, surface the error to the host, and make recovery possible when new props arrive. Then there is setup and unmount — that case is covered, too. Where It Fits Compared to React.lazy or Portals This package is not a replacement for React.lazy, and it is not trying to be cleverer than React. If your component lives in the same bundle and the same React tree, React.lazy is still the natural tool. If you just want to render into a different DOM node inside the same tree, portals are great. mf-bridge is for the awkward case those tools do not cover well: a component living across a Module Federation boundary, loaded from a separate bundle, mounted into its own React root, but still expected to behave like a first-class part of the host page. That is the gap I wanted to close. A Small Package, Not a New Platform I also cared quite a bit about keeping the package lightweight. It has zero production dependencies and uses the browser's native CustomEvent API for prop streaming. In practice, that means less surface area, fewer moving parts, and one less utility layer to debug when something goes wrong. The goal was never to build a microfrontend platform. It was simply to remove a recurring nuisance and make the host/remote boundary feel safer. Sometimes that is enough to justify a package. I published it as @mf-toolkit/mf-bridge. Repository, docs, and examples: github.com/zvitaly7/mf-toolkit. If you are working with Module Federation and you already have a small pile of hand-written wrappers around remote React components, this may save you some time. And if you have solved the same problem in a completely different way, I would genuinely be curious to compare notes.

I watched a pull request get opened, reviewed, revised, and merged last week without anyone on the team writing a line of code by hand. A failing test triggered it. An agent read the stack trace, found the root cause in a config file three directories away, patched it, and tagged a human for sign-off. Nobody blinked. That's the part that surprised me — not the capability, but how unremarkable it's become. This is the actual shift happening in software teams right now. Autonomous AI agents have moved past the demo stage and into daily engineering workflows, and the gap between “AI helps me code faster” and “AI runs the task end-to-end while I review the output” has mostly closed. Gartner now projects that roughly 40% of enterprise applications will embed task-specific agents by the end of 2026, up from under 5% just last year — one of the steepest adoption curves the analyst firm has tracked. If you're a developer, a founder, or a tech lead still treating autonomous AI agents as a novelty, this is the year that assumption stops holding up. What Actually Changed For a couple of years, “AI-assisted development” meant autocomplete with better taste. You typed, the model suggested, you accepted or rejected. Useful, but you were still the one driving every keystroke. Agentic AI flips that relationship. Instead of suggesting the next line, autonomous AI agents take a goal — fix this bug, migrate this schema, write tests for this module — and work through the steps themselves: reading the codebase, running commands, checking the output, and correcting course when something breaks. That loop of plan, act, observe, and retry is the actual definition of agentic AI, and it's why the category feels so different from the copilots that came before it. Claude Code was one of the tools that pushed this into the mainstream for individual developers, handling multi-file refactors and terminal commands with minimal hand-holding. Cursor built a similar experience directly into the editor. But the bigger story in 2026 is what happened at the platform level: GitHub's Agent HQ turned “which AI assistant should I use” into “which combination of agents should handle this ticket,” letting teams route work across Copilot's Agent Mode and third-party models from a single dashboard, with human approval gates before anything touches a protected branch. Where the Orchestration Layer Comes From None of this multi-agent coordination works without a shared way for agents to reach tools, files, and other services — which is exactly the gap Anthropic's Model Context Protocol (MCP) was built to close. MCP gives an agent a standard way to discover and call outside tools instead of every vendor inventing its own integration format, and it's become close to a default for agent orchestration since Anthropic donated its stewardship to the newly formed Agentic AI Foundation (AAIF), a Linux Foundation initiative that also now governs projects like Goose and AGENTS.md. That neutral home matters: it means an agent built by one team can plug into infrastructure built by another without a custom adapter for every pairing. If your team is scoping actual implementation work — repository intelligence, agent orchestration across a codebase, or a production-ready rollout rather than a weekend experiment — this is usually the point where it's worth bringing in people who've built agentic workflows before rather than reverse-engineering the architecture from blog posts. The 2026 Agent Landscape, Compared Six tools keep coming up in conversations with other developers this year, and each one solves a slightly different problem: Tool Best for Runs where Claude Code Deep, multi-file coding tasks and repo-wide reasoning Terminal / IDE GitHub Copilot (Agent Mode) In-editor task delegation tied to your existing workflow VS Code, GitHub Cursor Fast iterative coding with tight human-in-the-loop control Standalone editor GitHub Agent HQ Orchestrating multiple vendor agents under one governance layer GitHub platform OpenClaw Personal automation across messaging apps, not just code Self-hosted Hermes Agent Self-improving agents that build their own reusable skills over time Self-hosted Notice the split: some of these are coding-first (Claude Code, Copilot, Cursor), some are orchestration layers for running several agents together (Agent HQ), and a couple — OpenClaw and Hermes Agent — treat “developer tool” as just one use case inside a broader personal or team automation runtime. That's the direction agentic workflows are heading generally: fewer single-purpose bots, more general-purpose agents you point at whatever task needs doing, whether that's a codebase, a customer inbox, or AI-powered social media tools handling a content calendar. Where Autonomous AI Agents Are Actually Winning The realistic use cases in production right now are narrower than the hype suggests, and that's a good thing — narrow and reliable beats broad and flaky. Dependency and CI maintenance. Agents opening PRs for failing tests, outdated packages, or flaky pipeline steps — low-risk, high-volume, exactly the kind of work teams were happy to hand off first.Repository intelligence. Understanding a large, unfamiliar codebase fast enough to answer “where does this value actually get set” without a human spending an afternoon grepping.Multi-agent QA. One agent writes the feature, a second reviews it for security and style, a third checks it against the test suite — a pattern GitHub's Agent HQ and similar orchestration setups are explicitly designed around.Business process agents. Outside of pure dev work, enterprise AI agents are handling ticket triage, lead qualification, and reporting — the same goal-driven execution model, aimed at business operations instead of code. The Limitations Nobody Skips Past I'd be doing you a disservice if I didn't mention the failure mode Gartner keeps flagging alongside the adoption numbers: more than 40% of agentic AI projects are expected to be shelved by 2027, mostly because teams scoped them too broadly or skipped governance until after something went wrong. Fully autonomous agents making irreversible decisions without a review step are still a bad idea in most production environments. The teams getting real value are the ones treating agentic AI development as an engineering discipline — scoped tasks, audit trails, and a human who can pull the plug — not as a replacement for judgment. Getting Started Without Betting the Roadmap If you're evaluating this for your own team, start narrow. Pick one bounded, low-stakes workflow — dependency bumps, doc updates, a single well-tested service — and let an agent run it end-to-end for a few weeks before expanding scope. Set explicit guardrails (no direct merges to protected branches, mandatory human review on anything touching auth or billing), and measure actual time saved rather than assuming it. Most teams that get this right end up with a meaningful share of merged work coming from agents within a couple of months, with humans doing the scoping and reviewing rather than the typing. FAQs What are autonomous AI agents? Autonomous AI agents are AI systems that can plan, execute, and adjust multi-step tasks toward a goal with minimal step-by-step human input, as opposed to tools that only respond to single prompts. How is agentic AI different from a regular chatbot or copilot? A chatbot answers one prompt at a time. Agentic AI runs a loop — plan, act, check the result, retry if needed — and can call tools, run code, and make decisions across many steps before reporting back. Are autonomous AI agents actually production-ready in 2026? For scoped tasks like dependency updates, test fixes, and repository analysis, yes. For fully unsupervised, high-stakes decisions, most teams still keep a human review step, and Gartner data backs up that caution. What is the Model Context Protocol (MCP) and why does it matter? MCP is an open standard that lets AI agents connect to external tools, files, and services in a consistent way, rather than needing custom integration code for every combination of agent and tool. Do I need a multi-agent system, or is one agent enough? Depends on the task. Single agents handle most day-to-day coding work fine. Multi-agent systems earn their complexity when you need separation of concerns — one agent building, another reviewing, another testing. How do I get started with agentic AI development for my team? Pick one narrow, low-risk workflow, set clear guardrails around what the agent can touch without approval, and expand scope only after you've measured real results — not assumed them.

We all saw the rise and fall of GraphQL. The technology was hip at the time, and then we discovered it was slow, very complex, and it was easy to shoot yourself in the foot on security. REST won that fight. One major factor that went in favor of REST was that every language speaks it, every developer understands it, and you don’t need to run a special server just to serve a GraphQL API. But does this still stand true in the age of AI? Let us try to unpack this question and see if this time it could be different for GraphQL? There’s a New API Consumer, and It Doesn’t Think Like Humans For years, APIs had two audiences: first, the services (predictable, hard-coded integrations) like APIs talking to APIs, and humans using apps (who don’t mind a bit of extra data; nobody notices 40 fields traveling across the wire while the screen only renders 10 fields). AI agents are a third audience, and they behave nothing like the first two. Think of it like this: a human browsing a shopping site doesn’t care if the product page quietly loads size charts, reviews, and shipping data if the human is not interested in those. An AI agent, though, has to read every field it’s handed, and every one of those fields sits in its memory, costing money and crowding out the things it actually needs to think about. It’s less like browsing and more like being handed the whole filing cabinet when you asked for one folder. Over-Fetching Isn’t Just Wasteful for Agents; It’s Expensive in a Different and Costly Currency Let us assume an agent asks “who manages this account?” A typical REST endpoint hands back the entire user record, the email, address, and ten other fields. This is because building a trimmed-down endpoint for every possible question is a lot of upfront engineering work. A human skims past the noise. An agent has to carry it around for the rest of the conversation, like packing your whole closet for a weekend trip because folding a smaller bag felt like too much effort. GraphQL flips that: the agent asks for exactly “manager name and email,” and that’s all that comes back. The N+1 Problem, Agent Edition Anyone who’s worked with databases knows the pain: you fetch a list of 10 orders, then make 10 more calls to get customer details for each one. REST APIs often have the same shape. For an agent, every one of those round trips is another context-window hit and another few seconds of latency, like sending ten separate texts instead of one paragraph. GraphQL lets the agent ask for orders and their customers in a single request. A Schema the Agent Can Actually Read REST documentation is a promise: “this is what the API looks like, we hope, as of whenever someone last updated the docs.” When it drifts out of date, an agent’s fallback is basically the same as a stressed junior developer’s: search the web, then go read the source code. GraphQL bakes the documentation into the API itself. The agent can ask the server, at runtime, “What exists, what does it need, what’s deprecated?” It’s the difference between asking a new coworker to guess your team’s tools from an outdated wiki page, versus just asking the tool itself how it works. Security That Matches How Agents Actually Work Most REST permission systems are coarse, calendar.read, repos.write and so on. Fine for a human logging into one app with one role. But an agent might handle customer support in one breath and billing cleanup in the next, and you don't want it holding a master key for both. GraphQL checks access field-by-field, not just endpoint-by-endpoint. That means you can grant an agent “read the customer’s name” without also granting “read their payment history”, even if both live on the same object. It’s the difference between giving someone a key to the building versus a key to one specific drawer. Errors an Agent Can Actually Act On REST failure: “400 Bad Request.” Sometimes JSON, sometimes an HTML page; format varies by provider and sometimes even within the same provider. GraphQL failure, “the field user.team.name failed, no read access on team 7." That's something an agent can act on directly; it can even retry a different query, ask for permission, or explain the problem to a person instead of burning another model call just to figure out what went wrong. Where REST Still Wins, and Probably Always Will This isn’t “GraphQL beats REST.” Caching is nearly free with REST; every CDN on earth understands it natively. GraphQL caching is a genuine engineering project. Uploading a file or streaming video over REST is simple; doing it over GraphQL is awkward. And running a GraphQL server is real operational overhead REST doesn’t have. So what’s the actual comeback? Not GraphQL replacing REST for humans and services. More like this shape, Human or agent → MCP server / CLI tool → GraphQL → your actual backend Today, most people bolt an MCP server onto REST, then hand-build the exact “shape” of every response, field by field, tool by tool, basically reinventing what GraphQL already does natively. Put GraphQL underneath instead, and the MCP layer can just pass the agent’s query straight through, precise fields, typed schema, field-level permissions, structured errors, all included. I’m not saying rip out your REST APIs. I’m saying the layer sitting between AI agents and your systems might quietly end up looking a lot like GraphQL, and if you’re building tools for agents right now, this is worth an experiment.

I work as a data analyst at a legal services company. Part of my work involves protecting sensitive data during the Test Data Management (TDM) process. Many other departments in the company need test data to develop an application. Copying the production data for test sounds like a good plan. But because the test environment usually has lower cybersecurity requirements, this will cause customer privacy data leaks. So, my job is to mask the sensitive data to protect customer privacy. When it comes to my job, the first thing that comes to many people’s minds is that my work involves masking sensitive data. For example, changing the email address from [email protected] to [email protected]. Masking data is indeed important, but before we jump to the masking step, there's one basic question: Which column contains sensitive data, and how can I find it? In this article, I will introduce a pipeline designed to identify sensitive data columns before masking steps. Structure of the Pipeline Please find the identifying sensitive data pipeline structure workflow chart below: Identifying Sensitive Data Pipeline Structure Workflow Before I start introducing each stage, I’d like to mention two points. The first point: The original intent behind this pipeline structure design was to save time spent locating sensitive data. Server usage is billed based on duration. In a perfect world, the system would balance efficiency and accuracy. However, in practice, efficiency takes precedence in order to cut costs. The second point: The pipeline also had to preserve data usability for testing. In some cases, data privacy controls must be designed in a way that does not break core application workflows. For key columns such as primary keys and foreign keys, they need to preserve join functions and application workflows. So in practice, we usually leave them unchanged. Apply Column Name and Pattern Matching First, and quite intuitively, many columns' names are really straightforward and can be easily identified. For example, full name, phone number, and email. After the very first easy screening, some columns can be identified by hardcoded Python scripts, based on the specific column name pattern. However, there is an issue at this stage. I can identify columns containing sensitive data using customer email. But if there is another column named customer email address that hasn't been included in the hardcoded script, I won't be able to detect it. Besides that, relying solely on column names isn't always reliable. Take the notes column, a free-text field, for instance. It often appears as an optional field after the main information has been entered. Most people will leave it blank or write some insignificant things. But sometimes customers do write something, such as Our CEO Everett would like you to prioritize processing the ABC document. Please email them to [email protected] as soon as you finish, and then call 123-456-7890 to notify him. If I don't mark this column as need masking, the customers' private information will be exposed. Check Historical Decisions After the initial filtering step, I will check the historical decisions database for specific columns, such as the notes column mentioned earlier. If the database indicates that the historical decision for column notes is to mask it, then that column will be masked during the current round. Even if the notes in this specific round contain no sensitive data. For example, no privacy-related information is mentioned. There is no guarantee that the data in the next refreshed cycle will remain free of sensitive information. Send Ambiguous Columns to AI for Review and Analyze Sample Values Here comes the highlight of the entire pipeline. Sometimes, column names are somewhat ambiguous. Or it's unclear whether certain rows contain sensitive data. Let's take the notes column mentioned earlier again. It might be empty. Or it could contain a message like When food is delivered, please ring the doorbell and call my wife Bobi, thereby the sensitive information gets leaked. I started using the spaCy library from Python for Natural Language Processing (I will refer to this term as NLP later in this article). While spaCy isn’t a Large Language Model (I will call this term LLM), it certainly performs NLP analysis. However, the sampling process was time-consuming. I would sample the entire dataset if it had fewer than 50,000 rows, but randomly select 50,000 rows if it exceeded that limit. In a later version of the workflow, I switched to OpenAI: this time, I just need to select a sample of 100 rows and send them via API to the AI/LLM for analysis. The AI then generates a masking recommendations database, which will undergo manual review later. Accuracy improved significantly after we began using LLMs. It rose from 80% with spaCy to approximately 93% after switching to OpenAI. This 93% figure was determined by having human analysts conduct a column-by-column analysis in parallel with my development of the pipeline and automation scripts. So the result is benchmarked against manual reviews. Furthermore, this figure represents an average obtained after two rounds of actual TDM data masking operations and several additional rounds of testing. Regarding the remaining 7% of errors, false positives accounted for about 90%, and false negatives for only 10%. This is important because missing sensitive data is much more serious than over-flagging a column for review. Compared to manually analyzing a medium-sized schema containing 100 tables for 64 hours. An automated script can complete the analysis in just 2 hours. However, please note that this 2-hour timeframe does not include the time required for subsequent manual review. Human Review and Store Recommendations and New Decisions After the AI/LLM finishes analysis, human analysts will review the mask recommendations database generated by the AI. Each row in the database generates a report containing the user ID, database name, table name, column name, masking suggestion, masking rule, and analysis date. Then, humans will review the mask suggestions and corresponding masking methods. For example, the AI-generated mask suggestion database is: AI-generated Mask Suggestion Database Example As a human analyst, at this stage, I can review the masking suggestion generated by the AI. I would agree with the suggestion to mask the data. However, regarding the masking rule, I would review it and change it to set it to a blank value. Manual review needs to randomly sample 500 rows and analyze them individually to reach a final mask decision. In this new process, human analysts only need to review a single row of AI-generated mask decisions and mask rules. The switch saves time significantly. During a new round of the TDM data masking process, some new columns will be identified by AI and flagged as requiring masking. The new masking decision will be added to the existing historical decisions database after manual review. Send to Data Governance and Send to Business Customer and Get Feedback After our TDM team identifies and masks the sensitive data columns, we submit our results to the Data Governance department for a secondary manual review. Their review process differs slightly from ours. Our team focuses on using business knowledge to determine whether a column contains sensitive data. And we’re also responsible for developing more efficient identification & masking procedures. However, the Data Governance department needs to review and provide more accurate masking decisions. Because their team members have better knowledge of how to decide whether a column should be masked and of the appropriate masking method. After our two departments conducted two rounds of manual review, we sent the masked data results to our business customers' departments. They will use this data for testing and provide us with feedback based on their specific needs. For example, we recommended masking customer_id with a generated synthetic number. But doing so will change primary and foreign keys, thereby breaking database linkages. So, our business customer departments advised us against masking those columns. Conclusion and Future Improvement Plans Successfully masking sensitive data begins with accurately identifying the columns containing such data. Many people skip this and jump straight to the more interesting masking process. In my view, however, getting this step wrong will fail the rest of the workflow as well. The pipeline I designed isn't perfect. And I have a few ideas for improving the "Apply column name and pattern matching" component in the future. Since we’ve already used OpenAI, why not let the AI detect new patterns when analyzing ambiguous columns? We could have the AI generate a dynamic pattern database that updates automatically with every refresh cycle. It would also help us continuously update and refine our historical decisions database.

Supply chains are graphs. Suppliers feed into warehouses, warehouses feed into distribution centers, and distribution centers feed into retailers. When we model them that way — as nodes and relationships rather than rows and columns — we unlock a set of tools that gives us the ability to ask questions about connectivity, paths, and the structural importance of individual nodes. In this article, we'll build a supply chain, load it into Neo4j via Apache Spark, use NetworkX to identify the most critical nodes in the network, and then simulate a real-world disruption to find alternative routes. The full source code is available on GitHub. The Stack Each tool in the stack does what it does best: ToolRoleApache Spark (local mode)Data generation, transformation, and loading into Neo4jNeo4j (remote, AuraDB)Graph storage and native variable-length path queriesNetworkXBetweenness centrality - identifying the most critical nodesPlotlyInteractive visualization throughout One tool conspicuously absent from this list is Neo4j's Graph Data Science (GDS) library. We'll come back to why and what to reach for when you outgrow the approach described in this article. Setting Up Neo4j AuraDB AuraDB is Neo4j's fully managed cloud database. A free tier is available with no credit card required. Sign up at console.neo4j.io/graphacademy.Create a new AuraDB Free instance.When the instance is created, download or note the credentials - the connection URI, username, and password. Neo4j only shows the password once, so save it somewhere safe.Once the instance is running, open the built-in Query tab and verify connectivity: cypher MATCH (n) RETURN count(n) . This should return 0. We are ready to load data. Before starting Jupyter, export the connection details as environment variables in your shell: Shell export NEO4J_URI=neo4j+s://xxxx.databases.neo4j.io export NEO4J_USERNAME=your_username_here export NEO4J_PASSWORD=your_password_here export NEO4J_DATABASE=your_database_name_here The notebook reads these at startup and raises an error immediately if any are missing. The Data Model The supply chain has four layers connected by SHIPS_TO relationships: Plain Text Suppliers -> Warehouses -> Distribution Centers -> Retailers Each SHIPS_TO relationship carries three properties: cost (shipping cost in dollars)distance (km)capacity (maximum units per shipment) We'll generate a synthetic but reproducible dataset using Faker and NumPy with a fixed random seed, giving us 20 suppliers, 12 warehouses, 10 distribution centers, and 30 retailers with 125 routes across all three layers. Loading the Graph With Spark Spark earns its place in the pipeline by handling the loading step. The Neo4j Spark Connector translates Spark DataFrames into Cypher MERGE statements under the hood, handling the graph write for us: Python spark = ( SparkSession.builder .master("local[*]") .appName("SupplyChainResilience") .config("spark.jars.packages", SPARK_CONNECTOR) .config("neo4j.url", NEO4J_URI) .config("neo4j.authentication.basic.username", NEO4J_USERNAME) .config("neo4j.authentication.basic.password", NEO4J_PASSWORD) .getOrCreate() ) The connector JAR resolves automatically from Maven Central on first run. In a real pipeline, this step would read from S3, a data warehouse, or a Kafka topic and stream records into Neo4j continuously. One important detail is that we'll clear the database before each load using Cypher's IN TRANSACTIONS syntax so each run starts from a clean slate: Cypher MATCH (n) CALL (n) { DETACH DELETE n } IN TRANSACTIONS OF 10000 ROWS We'll then confirm the database is empty before writing new data to the database. Betweenness Centrality With NetworkX Betweenness centrality answers a specific question: if we looked at every possible shortest path between every pair of nodes in the network, how often does each node appear on one of those paths? A node with high betweenness acts as a bridge through which many shortest paths pass. If it disappears, many routes break. A node with low betweenness is peripheral - the network barely notices if it goes offline. We'll pull the graph out of Neo4j via Spark into a NetworkX DiGraph and compute centrality using shipping cost as the edge weight, so the algorithm finds shortest paths by lowest cost rather than fewest hops: Python edges_sdf = ( spark.read.format("org.neo4j.spark.DataSource") .option("query", "MATCH (a)-[r:SHIPS_TO]->(b) " "RETURN coalesce(a.id, a.name) AS source, " " coalesce(b.id, b.name) AS target, " " r.cost AS cost") .load() ) edges_pd = edges_sdf.toPandas() G = nx.DiGraph() for _, row in edges_pd.iterrows(): G.add_edge(row["source"], row["target"], weight = row["cost"]) centrality = nx.betweenness_centrality(G, weight = "cost", normalized = True) Figure 1 shows the full supply chain network before any disruption. Each node type is color-coded: suppliers in blue, warehouses in orange, distribution centers in teal, and retailers in red-orange. The density of connections between layers gives a first impression of where bottlenecks might exist. Figure 1. Full Supply Chain Network Once computed, we'll write the scores back into Neo4j via Spark so Cypher queries can use centrality as a filter or sort key without recomputing it every time. Figure 2 shows the top 15 nodes ranked by betweenness centrality. The length of each bar reflects how often that node appears on a shortest path between other nodes in the network. A longer bar indicates a node that carries a disproportionate share of shortest-path traffic. Figure 2. Top 15 Nodes by Betweenness Centrality Why Not GDS? Neo4j's Graph Data Science (GDS) library has a native gds.betweenness.stream() procedure that runs the same algorithm inside the database using advanced processing. For our small-node demo dataset, NetworkX is instant and requires no additional setup. But nx.betweenness_centrality() runs in O(n * m) time and loads the entire graph into memory. At tens of thousands of nodes, both of those properties become problems. That is exactly where GDS comes in. If you are using Neo4j AuraDB, the same algorithm is available through Aura Graph Analytics — a service that connects directly to your AuraDB instance. The rest of the notebook — Spark for data loading, Plotly for visualization, native Cypher for shortest path — works identically on AuraDB without any changes. Simulating a Disruption With centrality scores computed, we'll identify the highest-scoring node that is a Supplier or Warehouse and mark it as disrupted in Neo4j: Python with driver.session(database = NEO4J_DATABASE) as session: session.run( "MATCH (n {id: $id}) SET n.disrupted = true", id=disrupted_id ) We'll deliberately restrict disruption to Suppliers and Warehouses. Distribution centers are fewer in number, and each carries more routing burden, making them more likely to be sole bridges whose removal severs the network entirely. A warehouse disruption is a more realistic scenario and produces richer alternative-route results. Finding Alternative Routes With Native Cypher With the disrupted node flagged, we'll use Neo4j's built-in variable-length path matching to find alternative routes that avoid it: Cypher MATCH (s:Supplier), (r:Retailer) WHERE s.disrupted IS NULL AND r.disrupted IS NULL MATCH path = (s)-[:SHIPS_TO*..6]->(r) WHERE none(n IN nodes(path) WHERE n.disrupted IS NOT NULL AND n.disrupted = true) WITH s, r, path, reduce( cost = 0.0, rel IN relationships(path) | cost + rel.cost ) AS total_cost ORDER BY total_cost ASC RETURN s.id AS source, r.id AS target, [n IN nodes(path) | coalesce(n.id, n.name)] AS path_nodes, round(total_cost, 2) AS total_cost, length(path) AS hops LIMIT 10 This query is available on both local Neo4j and AuraDB with no additional plugins required. A typical result looks like this: Plain Text source target total_cost hops S006 R001 94.35 3 S007 R026 148.86 3 S007 R017 155.19 3 S019 R026 165.88 3 The cheapest alternative route bypasses the disrupted node entirely at a total shipping cost of $94.35. Note that MATCH (s:Supplier), (r:Retailer) creates a cartesian product for every Supplier/Retailer pair, which is fine for our small dataset. For larger graphs, you would normally constrain the source and destination. The network after disruption is shown in Figure 3. The disrupted node is highlighted in red, and the best alternative route is shown in green, tracing the lowest-cost path from supplier to retailer that avoids the failed node entirely. Figure 3. Best Alternative Route After Disruption Figure 4 compares the top alternative routes by total shipping cost and number of hops. A route with more hops may still be cheaper - the cost comparison makes that trade-off explicit and gives logistics planners a clear basis for decision-making. Figure 4. Alternative Route Cost and Hop Comparison Gotchas and Lessons Learned This project required some debugging. Here are the issues worth knowing about before you try this yourself. Java Version Compatibility PySpark 3.5.x officially supports several versions of Java. However, Java 23 removed javax.security.auth.Subject.getSubject(), which Spark's Hadoop dependency calls during startup. On Java 23 or later, this produces a cryptic UnsupportedOperationException: getSubject is not supported error and Spark never starts. The solution is to install Java 21 LTS alongside any existing Java installation and point PySpark at it before starting Jupyter. Here, for example, using Homebrew on Apple hardware: Shell brew install openjdk@21 export JAVA_HOME=/opt/homebrew/opt/openjdk@21 export PATH="/opt/homebrew/opt/openjdk@21/bin:$PATH" Any existing Java installation is unaffected outside that shell session. The Neo4j Spark Connector 6.x support for Spark 4.x is in active development, so upgrading PySpark to avoid the Java issue is a future option. Relationship Write Deadlocks When writing relationships via the Neo4j Spark Connector with multiple Spark partitions, concurrent writes can deadlock inside Neo4j as transactions compete for the same node locks. The error looks like this: Plain Text ForsetiClient can't acquire EXCLUSIVE NODE_RELATIONSHIP_GROUP_DELETE because it would form a deadlock wait cycle The solution is to call .coalesce(1) on the DataFrame before writing relationships, which forces Spark to write them sequentially from a single partition: Python sdf.coalesce(1).write.format("org.neo4j.spark.DataSource") ... Node writes do not need this because they do not acquire the same lock types. Stale Data Between Runs In the Jupyter notebook's write configuration, the Spark Connector's Overwrite mode merges on node keys but does not remove relationships that existed in a previous run but are absent from the current one. If the dataset size changes between runs, old relationships accumulate alongside new ones, interfering with the graph structure. The solution is to clear the database at the start of every load run rather than relying on Overwrite to clean up after itself. Always confirm the clear succeeded with a node count check before writing. The none() Predicate and Missing Properties This was the subtlest issue of the project. Our disruption query used: Cypher WHERE none(n IN nodes(path) WHERE n.disrupted = true) This returned zero results even when paths clearly existed, and the disrupted node was correctly flagged. In Neo4j, when a node doesn't have a disrupted property at all, n.disrupted = true evaluates to null rather than false. The none() predicate then treats every node as potentially disrupted and filters out all paths. This is exactly how Cypher's three-valued logic works. The solution is an explicit IS NOT NULL check: Cypher WHERE none(n IN nodes(path) WHERE n.disrupted IS NOT NULL AND n.disrupted = true) shortestPath() and Alternative Routes Initially, Neo4j's shortestPath() function was used to find alternative routes. It returned zero results. The reason is that shortestPath() finds the path with fewest hops first, then applies the WHERE none(...) filter. It computes a single shortest path rather than exploring alternative candidates, and filtering on disrupted nodes can eliminate that path without considering longer valid alternatives. The solution is to use a plain variable-length path match with an explicit hop limit instead. This lets the WHERE clause filter while still returning valid results: Cypher MATCH path = (s)-[:SHIPS_TO*..6]->(r) WHERE none(n IN nodes(path) WHERE ...) Guaranteed Connectivity in Generated Data With purely random route generation, it's possible for a single node to end up as the only connection between two layers - a so-called sole bridge. Disrupting that node severs the network completely and leaves no alternative routes to find. The solution is to generate routes with a guaranteed minimum connectivity. So, every source node gets at least two outbound routes, and every target node gets at least two inbound routes before random fill: Python def make_routes(sources, targets, n_routes, min_out=2, min_in=2): # Guarantee every source has at least min_out outbound routes for s in src_ids: sample = rng.choice(tgt_ids, size = min(min_out, len(tgt_ids)), replace = False) for t in sample: if (s, t) not in seen: rows.append(make_row(s, t)) # Guarantee every target has at least min_in inbound routes for t in tgt_ids: sample = rng.choice(src_ids, size = min(min_in, len(src_ids)), replace = False) for s in sample: if (s, t) not in seen: rows.append(make_row(s, t)) # Fill remaining routes randomly ... Cypher 25 Syntax If you are running Neo4j 2025.06 or later, the CALL { WITH n ... } subquery syntax used in batch deletes is deprecated. Use the new variable scope syntax instead: Cypher MATCH (n) CALL (n) { DETACH DELETE n } IN TRANSACTIONS OF 10000 ROWS Summary We've built a supply chain resilience analysis pipeline that models a supply chain as a graph, identifies its most critical nodes using betweenness centrality, simulates a real-world disruption, and finds alternative routes using native Cypher. Each tool did what it does best: Spark handled bulk data loading, Neo4j stored the graph and answered path queries, NetworkX computed the graph algorithm, and Plotly produced interactive visualizations at every stage. The gotchas section above contains several useful engineering lessons, which should save you time and effort on your projects. The full source code is available on GitHub.

By Akmal Chaudhri CORE

Retrieval-augmented generation (RAG) reduces hallucinations in generative AI chatbots by grounding each response in retrieved source data instead of relying only on what the model learned during training. Before the model writes a reply, the system fetches relevant passages from a trusted knowledge store and passes them in as context. The model then answers from that evidence, which shrinks the room it has to invent facts. This article looks at why hallucinations happen at the token level, how a RAG pipeline counters them, and the engineering choices that decide whether grounding actually holds up in production. Why Generative AI Chatbots Hallucinate A large language model predicts the next token from statistical patterns, not from a fact store it can look up. Ask it about something outside its training data or about a recent change, and it still returns fluent, confident text, sometimes wrong. That confident-but-wrong output is a hallucination. Three causes show up most often in conversational AI systems: Knowledge gaps. The training corpus has a cutoff, so newer facts are missing.Ambiguous prompts. Vague input pushes the model to guess.Pattern completion. The decoder prefers plausible phrasing over accurate phrasing when both fit. For a customer-facing bot, the cost is concrete: invented pricing, fictional policies, or wrong API behavior, all delivered in the same tone as a correct answer. What Retrieval-Augmented Generation Actually Does RAG connects the model to an external knowledge base at query time. Rather than answering from parameters alone, the chatbot searches a document store first, pulls the closest matches, and injects them into the prompt. The pipeline runs in three stages: Retrieve: embed the user query and run a similarity search against a vector index.Augment: place the top passages into the prompt as grounding context.Generate: the model composes an answer constrained by that context. Because the output is tied to retrieved text, the system can also return citations pointing at the exact source. How RAG Reduces Hallucinations RAG targets the root cause: missing or stale context. Supplying current, relevant evidence narrows the space where the model has to improvise. Grounding in approved sources The model reads from your documents, so answers reflect your data rather than internet averages. A well-built pipeline also instructs the model to reply "not found" when retrieval returns nothing useful, instead of filling the gap with a guess. Fresh data without retraining You update the index, not the weights. New policies or product details become answerable the moment they are ingested, which removes a major source of dated, wrong replies. Traceable answers Each response can carry a reference back to its source passage. For regulated domains, that audit trail is often the difference between a system people use and one nobody trusts. A Minimal RAG Loop The core retrieval-then-generate step looks like this in pseudocode: Python def answer(query, index, llm): q_vec = embed(query) passages = index.search(q_vec, top_k=5) if not passages: return "I don't have that information." context = "\n".join(p.text for p in passages) prompt = f"Answer using only this context:\n{context}\n\nQ: {query}" return llm.generate(prompt) The top_k cutoff, the "only this context" instruction, and the empty-result fallback are small details that carry most of the anti-hallucination weight. Where RAG Pipelines Break Retrieval quality, not model size, is where most accuracy is won or lost. Common failure points: Bad chunking. Segments too large dilute relevance; too small and they lose meaning.Weak embeddings. A mismatched embedding model returns passages that look related but aren't.No reranking. Top-k by cosine similarity alone often buries the best passage below near-duplicates.Silent context overflow. When retrieved text exceeds the window, passages get truncated, and the model fills the gaps on its own. 2026 Patterns Worth Knowing A few shifts are changing how teams build these systems this year. Agentic RAG. Instead of one lookup, the chatbot plans multi-step retrieval, calling tools and querying several sources before answering. This handles compound questions a single search cannot. GraphRAG. Pairing a knowledge graph with vector search captures relationships between entities, which improves answers over connected or multi-hop data. Continuous evaluation. Automated grounding checks score every answer for faithfulness to its sources, catching regressions before users report them. As enterprise adoption grows, this kind of automated eval is moving from nice-to-have to default. Decision Factors Before You Build If you are weighing RAG for a production bot, the factors that matter most: Data freshness and cleanliness beat any single model choice.Chunking and overlap shape retrieval accuracy more than people expect.Guardrails: confidence thresholds and fallback responses so the bot declines rather than fabricates.An eval pipeline that measures grounding rate, not just fluency.Latency budget: retrieval adds round trips, so cache common queries. FAQ Does RAG remove hallucinations completely? No. It reduces them sharply, but noisy data or poor retrieval can still produce errors, which is why evaluation and guardrails stay necessary. Is RAG better than fine-tuning? For fresh, factual answers, RAG usually wins because you update data without retraining. Fine-tuning suits tone and format. Many systems use both. What data does a RAG chatbot need? A curated knowledge base: documentation, FAQs, policies, or product data, cleaned and chunked for retrieval. A Final Word Hallucination is the line between a chatbot demo and a system a team can put in front of real users. RAG addresses it directly by grounding generative AI chatbots in current evidence rather than hoping the weights remember. The hard part lives in retrieval quality and evaluation, not in the model alone.

Then the search form grows, filters multiply, and nested criteria appear. Since using GET means placing the query inside the URI, a length limit problem emerges. Worse, placing sensitive query values in the URI increases the chance of exposure through access logs, browser history, proxies, and monitoring systems. Because the HTTP protocol does not forbid it, sending a body with GET may look like a way out, but building your design on behavior the standards leave undefined is not a recommended practice. Elasticsearch's GET-with-body search API is a well-known example, and Elastic's own documentation openly acknowledges the problem: "As a result, some HTTP servers allow it, and some—especially caching proxies—don't. [...] However, because GET with a request body is not universally supported, the search API also accepts POST requests." HTTP POST, on the other hand, carries the query in the request payload rather than the URI, which overcomes both the length limit and the data leakage problems. But POST is neither safe nor idempotent, since the protocol allows every invocation to change state on the server, and its response is not cached unless it carries explicit freshness information. This nature of POST also imposes a performance cost: results are recomputed and retransferred on every call, and a timed-out request cannot be safely retried. What is missing is clear: a method that is safe and idempotent like GET but carries content like POST. Until June 2026, HTTP did not have such a method in standardized form. The QUERY Method To address this need, the IETF introduced the QUERY method in RFC 10008. QUERY is the first new HTTP method since RFC 5789 was standardized in 2010. The core idea can be summarized as follows: a QUERY request asks the target resource to process the enclosed content in a safe and idempotent manner and to respond with the result. Everything else the RFC introduces either follows from this definition or builds practical machinery around it. Let's look at the key concepts one by one: Safe and Idempotent A QUERY is defined as a safe operation: it does not request a state change on the target resource. It can be retried, repeated, or restarted automatically without concern for partial side effects. This is the contract that separates it from POST. Meaning Comes From Content-Type RFC 10008 deliberately does not define a query language. The same endpoint may accept a JSON filter document, a form-encoded string, or any other query language defined by a media type; the media type of the request content defines how the server should interpret it. Servers are required to reject requests whose Content-Type is missing or inconsistent with the content. The RFC goes as far as forbidding content sniffing: a server is not allowed to infer a media type from the request content and use it to repair a missing or erroneous Content-Type. Explicitly Cacheable Unlike POST, QUERY introduces cacheability for body-carrying requests, with one crucial twist: the cache key must include the request content in addition to the URI, since two QUERY requests to the same URI with different bodies are different queries. Discovery via Accept-Query A server can advertise QUERY support with the Accept-Query response header, which lists the media types it accepts as query content. The Equivalent Resource A QUERY response may include a Location header pointing to a URI that represents the same query. A client can later re-fetch the result with a plain GET, no body required. The spec also gives 303 See Other a natural role for redirecting a query to a retrievable resource. The RFC's Security Considerations add one caveat here: when the query contains sensitive information that must not be logged, the URI assigned to such a resource should not include any sensitive portions of the original query content; otherwise, the exposure problem QUERY avoids would simply reappear one response later. Familiar Error Semantics The RFC recommends specific status codes for the failure cases: 400 when media type information is missing, 415 when the media type is not supported by the resource, and 422 when the content is well-formed but the query cannot be processed. A Decade in the Making The RFC had a long journey. The idea traces back to WebDAV's SEARCH method (RFC 5323, 2008), which demonstrated the demand for body-driven queries but remained confined to the XML-based WebDAV ecosystem. In 2021, the HTTP Working Group adopted the effort as a working group item, moving it from an individual proposal into the IETF standardization process. The method was later renamed from SEARCH to QUERY to avoid confusion with the existing WebDAV SEARCH method and to better reflect its purpose. The document was published as RFC 10008 in June 2026. Eleven years from the first draft to Proposed Standard is a useful reminder that even a seemingly simple addition to HTTP touches an enormous installed base and therefore receives extensive scrutiny. Where Ecosystem Support Stands Today As of July 2026, HTTP QUERY has completed the standardization phase with RFC 10008, but ecosystem adoption remains in its early stage. Many HTTP servers and proxies can forward QUERY requests without protocol changes, but native support across frameworks, browser APIs, caches, WAFs, and API tooling is still emerging. The primary barrier is no longer the protocol itself, but the large installed base of software that assumes a fixed set of HTTP methods. The Java ecosystem offers a useful snapshot of adoption in progress: Apache Tomcat A pull request adding QUERY support was merged on July 1, 2026 (apache/tomcat#1026). Support is available only in Tomcat 12 because it required Servlet API changes. Eclipse Jetty Eclipse Jetty has an open pull request (jetty/jetty.project#15316) implementing the core RFC 10008 semantics: method registration as safe and idempotent, the Accept-Query header, redirect behavior, and integration with compression and buffering handlers. It was initially aimed at Jetty 12.1 but has been retargeted to Jetty 13, aligning with a possible Jakarta Servlet 6.2 timeline. Jakarta Servlet There is an open issue (jakartaee/servlet#1068) proposing the addition of QUERY to the specification itself, so that HttpServlet gains first-class support and QUERY requests receive the same form parameter processing model currently defined for POST. This is arguably the most significant milestone for the broader Jakarta EE ecosystem, because it moves QUERY from container-specific support into the platform specification itself. Once Servlet defines QUERY, application servers such as WildFly, Payara, and Open Liberty can inherit support through their servlet containers as they move to the new specification level. As of this writing, none of them has shipped QUERY support ahead of the specification. What About Spring? Spring deserves its own section because of how request mapping is modeled. Spring MVC and WebFlux expose their annotation-based request mapping model through the RequestMethod enum, and that enum currently contains GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS, and TRACE. There is no RequestMethod.QUERY, which means you cannot declaratively map a QUERY request through Spring's annotation-based programming model today. The available workarounds are awkward and bypass Spring's normal request-mapping model: declare a generic mapping and inspect request.getMethod() manually, or implement a custom RequestMappingHandlerMapping. Unlike the Servlet case, this is not primarily a container problem; it is primarily a framework API and abstraction problem. The Spring team is aware. A community pull request adding QUERY support (spring-projects/spring-framework#34993) has been open since before RFC 10008 was published. It supersedes a feature request that had remained open for nearly two years, and maintainers have indicated an intention to target Spring Framework 7.1, currently expected in November 2026. There is even a naming collision to solve first: the obvious convenience annotation @QueryMapping is already used by Spring for GraphQL. Why Quarkus Can Do It Today This is where an underappreciated property of HTTP pays off: the request method is simply a token defined by the HTTP grammar. A server does not need to have built-in knowledge of every method to parse it. Quarkus builds its HTTP layer on Netty and Vert.x, and neither requires the method to be one of a predefined set; the request can reach the routing layer without requiring special handling for QUERY. On top of that, Jakarta REST has had a standard extension point for custom methods since JAX-RS 1.0: the @HttpMethod meta-annotation, the same mechanism that has enabled JAX-RS applications to expose WebDAV methods like PROPFIND for years. Put the two together and RFC 10008-compatible QUERY endpoints in Quarkus require no framework changes; they can be enabled through a single Jakarta REST extension point: Java @HttpMethod("QUERY") @Documented @Target(ElementType.METHOD) @Retention(RetentionPolicy.RUNTIME) public @interface QUERY { } The remaining work is implementing RFC 10008 semantics at the application layer, which is precisely what the example project demonstrates. The Example: A Product Catalog You Can QUERY The demo repository is available on GitHub: hakdogan/http-query-method. It is a small Quarkus application exposing a product catalog at /products, deliberately compact, with only a handful of classes, but each RFC 10008 concept has a concrete counterpart in the code. One Query, Two Media Types The resource accepts the same logical filter in two representations, demonstrating that the query semantics are determined by the Content-Type, not the URI: Java @QUERY @Consumes(MediaType.APPLICATION_JSON) public Response query(ProductFilter filter) { ... } @QUERY @Consumes(MediaType.APPLICATION_FORM_URLENCODED) public Response queryForm(String body) { ... } So both of these work, and mean the same thing: Shell curl -i -X QUERY http://localhost:8080/products \ -H 'Content-Type: application/json' \ -d '{"category":"laptop","maxPrice":2000}' curl -X QUERY http://localhost:8080/products \ -H 'Content-Type: application/x-www-form-urlencoded' \ -d 'category=laptop&maxPrice=2000' A request with an unsupported media type is rejected with 415, and a filter that is well-formed but self-contradictory, such as minPrice greater than maxPrice, returns 422. The second part is a design choice rather than an RFC requirement: Section 2.1 says 422 can be used when the content matches its media type, but the query cannot be processed due to its actual contents, and returning an empty result with 200 would be an equally valid reading. The demo treats the contradiction as a client error because an empty 200 response would be indistinguishable from a legitimately empty match, silently hiding what is almost certainly a bug in the caller. The Response Tells the Whole Story A successful QUERY comes back like this: Shell HTTP/1.1 200 OK Content-Type: application/json Accept-Query: application/json, application/x-www-form-urlencoded Location: http://localhost:8080/products?category=laptop&maxPrice=2000 Cache-Control: no-transform, max-age=60 ETag: "f675e29b" [{"category":"laptop","id":2,"name":"ThinkPad X1 Carbon","price":1899.00}, ...] Three headers carry the RFC's ideas: Accept-Query advertises which media types the resource accepts as query content. In the demo, it is added by a small response filter.Location points to the equivalent resource from Section 2.2 of the RFC: the same query expressed through the request URI. Fetch it with a plain GET, and you get the identical result, no body needed. One of the tests does exactly that round trip.Cache-Control and ETag make the cacheability promise concrete. The ETag is derived from the result, so repeating the query with If-None-Match returns 304 Not Modified without resending the result: Shell HTTP/1.1 304 Not Modified ETag: "f675e29b" This is the answer to "why not just POST": QUERY was designed to provide query semantics without giving up the cache-friendly properties associated with safe methods. Discovery Without Prior Knowledge How does a client discover that a resource supports QUERY? One OPTIONS request: Shell curl -i -X OPTIONS http://localhost:8080/products The response answers with two headers, one listing the methods the resource accepts and one listing the media types it accepts as query content: Shell HTTP/1.1 200 OK Allow: HEAD, QUERY, GET, OPTIONS Accept-Query: application/json, application/x-www-form-urlencoded In this case, Quarkus generated the Allow header automatically, including QUERY, simply because a resource method is bound to it. Proving Idempotency The demo's test suite covers the filtering logic, the media type handling, the error codes, the equivalent-resource round trip, the conditional request flow, and, fittingly for a method whose defining feature is repeatability, a test that repeats the same QUERY several times and verifies the operation remains safe and produces a consistent response. The key lesson from this example is not how QUERY was implemented, but why it was possible: the HTTP extension point already existed, and the framework did not need to invent a new abstraction. Conclusion QUERY is not a revolution; it is the standardization of a pattern that many systems have implemented through POST-based query endpoints for years. That is exactly why it matters. The gap between "works" and "works with the guarantees the protocol gives you" is where caching, idempotent retries, and better tooling become possible. Adoption is arriving unevenly: first in protocol implementations and servers, then in frameworks, gateways, and CDNs. But as the example shows, on a stack like Quarkus that treats the method as an extensible value rather than a hardcoded list, you do not have to wait to start experimenting. The protocol was ready for extension; the interesting question was whether the layers above it preserved that flexibility. The complete example, including all tests, is available on GitHub: hakdogan/http-query-method. References RFC 10008, The HTTP QUERY Method: https://www.rfc-editor.org/info/rfc10008/IETF Datatracker, document history: https://datatracker.ietf.org/doc/rfc10008/RFC 9110, HTTP Semantics: https://www.rfc-editor.org/info/rfc9110/RFC 4918, WebDAV: https://www.rfc-editor.org/info/rfc4918/RFC 5323, WebDAV SEARCH: https://www.rfc-editor.org/info/rfc5323/RFC 5789, PATCH: https://www.rfc-editor.org/info/rfc5789/

By Hüseyin Akdoğan CORE