DZone Spotlight

Agentic AI systems represent a paradigm shift in network operations, facilitating the transition from traditional reactive monitoring to fully autonomous management frameworks. For global infrastructure leaders, these specialized AI agents serve as persistent digital engineers, providing round-the-clock expertise across deployment, maintenance, and complex troubleshooting lifecycles. The following blueprint delineates the strategic application of agentic AI within a global enterprise network operations environment. Architectural Blueprint: Multi-Agent Network Engine Rather than deploying a monolithic AI entity, the optimal architecture utilizes a multi-agent system (MAS). In this model, specialized agents collaborate through a centralized Orchestrator to achieve complex operational objectives. Please refer to the System Hierarchy and Flow, as well as the Strategic Vision and Ecosystem diagrams. System Hierarchy and Flow Strategic Vision and Ecosystem Core Use Cases and Agent Workflows Autonomous Provisioning and Zero-Touch Deployment Functional scope: Serves as an automated configuration and systems engineer.Workflow: Human inputs a high-level intent (e.g., "Provision a new Cisco core switch for the Seoul R&D campus").Design Agent reads existing topology diagrams and ipAM (IP Address Management) databases.Configuration Agent generates vendor-specific CLI configs (Cisco IOS/NX-OS).Validation Agent runs the config in a virtual sandbox (e.g., Cisco Modeling Labs) to check for routing loops before pushing to production via Ansible. 24/7 Autonomous Incident Triage and Resolution Functional scope: Functions as the primary intelligence layer for Network Operations Center (NOC) alerts.Workflow: A monitoring tool (e.g., Splunk, SolarWinds) triggers a "BGP neighbor down" alert.Triage Agent intercepts the alert and instantly queries the device via SSH or telemetry API.Diagnostic Agent executes diagnostic scripts (show ip bgp summary, traceroute), analyzes logs, and identifies the root cause (e.g., a flapping fiber link).Resolution Agent opens a Jira ticket, attaches all logs, attempts a safe automated fix (like shifting traffic to a backup path), and pages a human only if the physical hardware needs replacement. Predictive Maintenance and Capacity Optimization Functional scope: Operates as a proactive infrastructure strategist and planner.Workflow: Analysis Agent continuously monitors interface bandwidth, memory usage, and optical transceivers' error rates across global data centers.Forecasting Agent identifies trends (e.g., "Link X between Vietnam and Korea factories will hit 95% capacity in 30 days during peak hours").Planning Agent drafts a change management proposal recommending a bandwidth upgrade or traffic re-routing plan for human review. Automated Compliance and Security Patching Functional scope: Functions as a continuous vulnerability and configuration compliance auditor.Workflow: Cisco releases a critical security advisory for an OS vulnerability.Auditor Agent scans the entire inventory database to find all vulnerable device models.Patching Agent schedules a maintenance window, backs up current configurations, downloads the verified firmware, applies the patch sequentially to avoid downtime, and performs post-flight checks. Implementation Strategy To ensure organizational stability while maximizing technological advancement, a phased implementation strategy is recommended: Phase 1: Read-Only Agents (Information Gathering) Allow agents to access logs, APIs, and read-only commands.Focus on automated documentation, ticket enrichment, and summarizing alerts. Phase 2: Human-in-the-Loop (Co-Pilot) Allow agents to generate fixes and configuration scripts.Require a senior network engineer to review and click "Approve" before any execution. Phase 3: Guardrailed Autonomy (Full Agentic) Give agents autonomous execution rights only for low-risk, repetitive tasks (e.g., port resets, clearing stuck sessions).Enforce hard boundaries using API rate limits and strict verification checklists. To refine the architectural requirements for your specific environment, please provide insights on the following: What monitoring tools (e.g., Datadog, ServiceNow) and automation frameworks (e.g., Ansible, Terraform) does your team currently use?What is the most repetitive issue your 24/7 NOC team spends time fixing manually?Are you looking to build these agents using commercial LLM APIs or host open-source models locally for security reasons? Building an Enterprise-Grade Agentic System Constructing an enterprise-grade agentic system for high-security environments requires the integration of a frontier large language model (LLM), a robust production agent framework, and the Model Context Protocol (MCP) to establish secure interoperability with core network infrastructure. Part 1: Choosing the Right Tool/Solution Do not rely on a simple chatbot web interface. You need an API-driven architecture that combines a foundational model with an agent framework. The Foundational Model (The Brain) Top Choice: Anthropic Claude 3.5 Sonnet / Claude 3 Opus Why: It currently dominates tool-use (function calling), reasons through complex, multi-step engineering logic flawlessly, and natively supports MCP. On-Premises / Air-Gapped Alternative: Llama 3.1 / 3.3 (70B or 405B) or Qwen 2.5 (72B) Why: For internal Samsung networks with strict data privacy laws, you can host these open-source models locally using vLLM or Ollama. They have excellent coding and structured JSON output capabilities. The Agent Framework (The Backbone) Use an open-source framework to manage agent memory, states, and collaboration: LangGraph (by LangChain): Best for networking. It allows you to build cyclical, graph-based agent workflows with strict state control and mandatory human-in-the-loop approval stages.CrewAI: Great for quickly setting up role-based multi-agent teams (e.g., Triage Agent talking to a Scripting Agent). Part 2: Understanding MCP (Model Context Protocol) MCP is an open standard created by Anthropic that acts as a secure plug-and-play adapter between LLMs and local/remote data sources, tools, and systems. Instead of writing custom API integration code for every single Cisco switch, Ansible Tower, or Jira server, you build or use an MCP Server. How it works: Your Agent (MCP Client) connects to an MCP Server. The MCP Server exposes specific tools (e.g., run_cisco_command, query_monitoring_alerts) to the agent dynamically. Security Benefit: The model never gets direct SSH access to your core switches. The model only talks to the MCP Server, which enforces strict argument validation, sanitization, and logging before executing anything on the network. Part 3: Writing agents.md The agents.md file defines the architecture, roles, personas, boundaries, and collaboration patterns of your AI team. Markdown # Network Operations Multi-Agent Architecture This document defines the specialized AI agents operating within the Network Operations Center (NOC). ## 1. System Orchestrator Agent - **Role:** Central Dispatch & Intent Router - **Goal:** Analyze incoming alerts or human engineer requests and delegate tasks to specialized agents. - **Persona:** A highly organized, senior technical project manager. - **Boundaries:** Does not execute network commands directly. Must always route tasks and compile final reports. ## 2. Network Diagnostics Agent (NetDiag-Agent) - **Role:** Incident Triager and Log Analyst - **Goal:** Investigate network anomalies, verify device status, and pinpoint root causes. - **Persona:** An analytical Cisco CCIE-certified troubleshooting expert. - **Tools Allowed:** `ssh_read_only_commands`, `query_splunk_logs`, `ping_traceroute`. - **Boundaries:** Read-only access to infrastructure. Cannot modify configurations. ## 3. Network Automation Agent (NetAuto-Agent) - **Role:** Configuration and Deployment Engineer - **Goal:** Generate valid, syntax-correct network configurations and execute approved automation scripts. - **Persona:** A precise Network Automation Engineer specializing in Ansible, Python (Netmiko/Nornir), and Cisco IOS/NX-OS. - **Tools Allowed:** `generate_ansible_playbook`, `validate_config_syntax`, `execute_stage_change`. - **Boundaries:** Write-access allowed *only* through staging/sandbox tools. Any production push requires an explicit `Human-In-The-Loop` (HITL) approval token. Part 4: Writing skills.md The skills.md file maps out the actual capabilities, API tools, and Python execution blocks that the agents can tap into via MCP. Markdown # Network Agent Skills & Tool Definitions This document details the functional tools exposed to agents via the MCP (Model Context Protocol) layer. ## Category: Network Telemetry & Read Skills ### `query_device_status` - **Description:** Fetches real-time operational data from a specific network device using Netmiko or Cisco DNA Center API. - **Arguments:** - `device_ip` (string, required): The target management IP address. - `command` (string, required): Allowed values: `show ip interface brief`, `show ip bgp summary`, `show environment`. - **Safety:** Input string regex validation enforced to prevent CLI injection attacks. ### `fetch_noc_alerts` - **Description:** Pulls active high-severity network alerts from Splunk/SolarWinds. - **Arguments:** - `severity` (string): Defaults to "CRITICAL". - `lookback_minutes` (integer): Time window to check. ## Category: Network Modification Skills (Write) ### `deploy_ansible_playbook` - **Description:** Triggers an Ansible AWX template to push configuration updates to a device group. - **Arguments:** - `template_id` (string, required): Pre-defined template ID in Ansible. - `extra_vars` (json, required): Key-value pairs for variables (e.g., target VLAN, interface ID). - **Security Policy:** **CRITICAL_APPROVAL_REQUIRED**. This skill automatically pauses agent execution and sends a webhook to the human supervisor's teams/Slack app for a 2FA confirmation click. Next Steps for Implementation To help map out a proof of concept (PoC) for your architecture, tell me: Do your network infrastructure components support REST APIs/gRPC, or do the agents need to interface heavily via legacy SSH/CLI parsing?Do you plan to build this as an internal web-dashboard/chatbot for engineers, or as a background event-driven system triggered by network monitoring systems?What programming language (e.g., Python, TypeScript) is your team's preference for building the MCP servers? Check out more from my series here.

By Daniel Oh CORE

In Q1 2026, three major agent infrastructure platforms dropped in nine weeks: OpenAI Frontier, the AWS Stateful Runtime, and Anthropic's Claude Managed Agents. What happens eighteen months from now when the model we built on gets deprecated, or we need to renegotiate pricing? I ran each platform through the same evaluation. Most teams I've talked to miss the question that actually matters: will your infrastructure survive a model change? Here's the five-pillar framework I use to find out, with real tests and code for each one. The Harness Is Not the Model Most platforms bundle two different things under one product name: the reasoning model and the harness. The model reasons. The harness does everything else that makes it an agent capable of executing multi-step tasks: Working and persistent memory – what the agent retains during a task and what carries over between sessionsTool execution – registering available tools, intercepting calls before they run, handling failuresSkills – reusable multi-step behaviors built on top of toolsOrchestration – coordinating multiple agents, routing subtasks, collecting resultsGovernance – access controls, human-approval gates, audit trails When the harness is tightly coupled to one model, you've made an architectural bet. That's not necessarily wrong, but you should make it on purpose. Five Portability Tests Run all five before you commit. 1. Memory: Who Does It Belong To? Run a two-session test. Session one: have the agent learn something concrete. A service that should always be read-only. A user preference. An entity it should recognize. Kill the session. Start a fresh one and ask whether it remembers. Then export the raw memory store: Shell # Export memory from your agent platform agent-cli memory export --session-id=<id> --output memory_dump.json # What you want to see: plain readable JSON cat memory_dump.json Portable memory looks like this: JSON { "entity": "postgres_prod", "type": "database", "note": "Read-only account. Never write.", "created": "2026-04-10T09:14:22Z", "tags": ["production", "restricted"] } Locked memory looks like this: JSON { "_type": "openai.memory.EphemeralObject", "_model_ref": "gpt-5.4-turbo-internal", "_blob": "AQIDBAUGBwgJCgsMDQ4PEBESExQVFhcY..." } If you need the provider SDK to deserialize it, it belongs to them. Persistent memory that survives a migration must be readable in a text editor without any provider dependency. If it's opaque binary or references internal provider objects, it won't survive. 2. Tools: MCP or Provider Subclass? Check how your tools are defined. MCP is now the de facto standard for provider-neutral tool registration. A tool defined in MCP works with any harness that implements the protocol. MCP-compatible (portable): JSON { "name": "query_database", "description": "Read-only SQL query against the analytics DB.", "inputSchema": { "type": "object", "properties": { "query": { "type": "string" }, "timeout_ms": { "type": "integer", "default": 5000 } }, "required": ["query"] } } Provider-locked (rewrite required when you leave): Python # This class dies with your provider contract class QueryDatabaseTool(openai.BaseTool): name = "query_database" def run(self, query: str) -> str: return db.execute(query) Plain JSON or YAML you can hand to another harness is portable. I've seen teams underestimate the tool-layer rewrite by a factor of three. It compounds fast when you have 30+ tools. For a practical look at how MCP connects agents to any API, the pattern holds across every provider. 3. Skills: Do They Survive a Model Swap? Skills are where lock-in accumulates without anyone noticing. A skill is a reusable multi-step sequence — something like search-summarize-route or draft-review-send. The problem is subtle: skills get built against one model's output format and response conventions. They work perfectly until they don't. Run this smoke test against a cheap model before you commit: Python SKILLS_TO_TEST = ["search_and_summarize", "draft_review_route", "incident_triage"] def smoke_test_skill(skill_name, model="gpt-4o-mini"): """ Not checking quality. Checking whether it completes. """ try: result = agent.run_skill( skill=skill_name, model_override=model, timeout=30 ) print(f"[PASS] {skill_name} on {model}: completed in {result.duration}s") return True except (ParseError, StepTimeoutError) as e: print(f"[LOCKED] {skill_name} on {model}: {e}") return False for skill in SKILLS_TO_TEST: smoke_test_skill(skill, model="gpt-4o-mini") smoke_test_skill(skill, model="mistral-7b") If it crashes: locked. If quality drops but it completes: portable. Those are completely different problems. 4. Orchestration: Grep Your Own Codebase Orchestration is where coupling gets expensive and invisible. If your planning agent parses sub-agent outputs using provider-specific response fields, swapping one model silently breaks everything downstream. The errors show up three layers away from the actual model call. Shell # Run this against your orchestration layer right now grep -rn "\.choices\[0\]" ./agents/ grep -rn "\.message\.content" ./agents/ grep -rn "openai\." ./orchestration/ grep -rn "anthropic\." ./orchestration/ If those grep results are long, you're coupled. Portable orchestration parses against schemas you control: Python # Locked: parsing a provider response object directly raw = agent_response.choices[0].message.content result = json.loads(raw) # Portable: your schema, not theirs result = TaskOutput.model_validate( parse_task_output(agent_response, schema=SUBTASK_SCHEMA) ) The fix isn't dramatic, but catching it late means touching a lot of code under pressure. For teams running multi-agent workflows with AWS Step Functions, this schema boundary becomes even more critical when agents span different provider runtimes. 5. Governance: Export and Read It The global race to govern AI agents has made governance a first-class concern — but most teams still treat it as an afterthought until a compliance conversation forces the issue. Governance covers what tools can be called, what needs human sign-off, and what goes into the audit log. It should live in your harness, not baked into a provider's permission system. Export it and look at what it references: Shell agent-cli governance export --format=yaml > governance_config.yaml cat governance_config.yaml Portable governance config: YAML policies: - name: restrict_production_writes applies_to_roles: [sre_agent, oncall_agent] deny: actions: [database.write, infrastructure.delete] resources: ["prod/*"] require_approval: - action: infrastructure.restart approvers: [oncall-lead] audit: log_all_tool_calls: true retention_days: 90 Locked governance config: YAML openai_platform: policy_set_id: ps_abc123xyz workspace_id: ws_prod_999 iam_role_binding: roles/openai.agentOperator permission_set: OPENAI_PROD_RESTRICTED If your config references provider IAM primitives or platform-specific IDs, it stays behind when you leave. You're rebuilding from scratch. The role of AI in IAM is evolving fast — your governance layer needs to be portable enough to keep up. How the Platforms Actually Scored OpenAI Frontier/AWS Stateful Runtime Well-built if you're staying on GPT. Memory in OpenAI infrastructure, tools through OpenAI SDK, orchestration layer assumes GPT conventions. The April 2026 Agents SDK update added workspace portability via Manifest and four memory tiers. Genuine improvements. But the docs say it plainly: designed for OpenAI models. The reference examples all use gpt-5.4. Know what you're signing up for. Claude Managed Agents The architecture is interesting. Three independent interfaces: session log, brain layer (Claude), and code execution sandbox. Each one can fail or be replaced without breaking the others. Anthropic published their reasoning: harness code encodes model limitations, and those limitations become technical debt as models improve. They built the interfaces to be swappable. The catch is the brain interface runs Claude. Changing that is a migration, not a config change. Claude Platform on AWS (GA May 11) Solves the governance and procurement problem better than anything else I tested. Auth through AWS IAM, billing through AWS Marketplace, audit logs in CloudTrail alongside your existing AWS services. Your governance policies for Claude agents live in a system your security team already knows. What it doesn't solve: data is processed by Anthropic outside the AWS boundary, so no Bedrock regional residency. And it's still Claude. The five tests above don't change. Open-source (LangChain Deep Agents, Letta, CrewAI, Microsoft Agent Framework) Model selection is a config variable. Memory in open formats. Tools in MCP. Governance is harness-owned. Less polished, more infrastructure work. That's the honest tradeoff. If you expect the model layer to keep moving, this is where you start. For teams already building compound AI systems for scalable workflows, the open-source stack plugs in naturally. Checklist Before You Commit These take a few hours. A migration takes months. Export persistent memory, open it in a text editor without a provider SDKCheck tool definitions: MCP/open schema vs. provider SDK subclassRun your top three skills against a non-primary model. Do they complete?Grep orchestration code for provider-specific response object referencesExport governance config, check whether it references provider IAM primitivesAsk: if this provider relationship ends tomorrow, which assets do I actually own? The platforms that shipped this year solve real problems. If you need production-grade agents fast and you're not switching models anytime soon, the managed options are good. The five tests tell you exactly what you're trading. Run them before you commit, and the decision is deliberate. Skip them, and you'll find out later at a worse time.

There is a number that engineering organizations love to report, and that engineering leaders love to receive: 100% code coverage. It has the satisfying quality of completeness. But completeness of what exactly? It implies that every line has been tested, every branch examined, every condition verified. It looks like the mathematical proof of a job well done. It is not, however. And the gap between what that number promises and what it delivers is, in many organizations, the single most expensive misunderstanding in the quality program. Code coverage measures the proportion of code that tests execute, not the proportion of behavior that tests verify. These are profoundly different things, and conflating them produces systems that are well-covered, yet dangerously under-tested. The engineers know the coverage number. The executives trust the coverage number. And the system fails in ways that the coverage number was incapable of detecting. What Code Coverage Actually Measures Code coverage tools work by instrumenting your codebase. They insert tracking markers at every line, branch, and condition. They run your test suite and record which markers were triggered. The coverage percentage is the proportion of markers that fired at least once during the test run. Notice what that definition contains and what it does not contain. It contains: which lines executed. It does not contain: whether the execution produced a correct result, whether the assertions in the tests actually verified the behavior, or whether the inputs used during execution were representative of the conditions the system will encounter in production. A test that calls a function and asserts nothing — or asserts the wrong thing — still generates coverage. A test that calls a function with a single, safe input still generates coverage for every line that input traverses. The coverage tool has no way to distinguish between a test that rigorously verifies behavior and a test that merely visits code. What Coverage Measures vs. What It Does Not MEASURES: Which lines of code were executed at least once during the test suite run MEASURES: Which branches (if/else paths) were taken at least once DOES NOT MEASURE: Whether the behavior of those lines was correct DOES NOT MEASURE: Whether the assertions in the tests were meaningful DOES NOT MEASURE: Whether the inputs used were representative of production conditions DOES NOT MEASURE: Whether the system behaves correctly under adversarial, boundary, or unexpected inputs DOES NOT MEASURE: Whether the test suite is enough This distinction is not a technicality. It is the entire problem. And the fastest way to see it is to look at code. The 100% Coverage Illusion: A Demonstration The following example is a payment processing function. Python # payment.py def process_payment(amount, card_number, currency='GBP'): """ Process a payment transaction. Returns a dict with 'status' and 'transaction_id'. """ if amount <= 0: raise ValueError('Amount must be positive') if currency not in ['GBP', 'USD', 'EUR']: raise ValueError(f'Unsupported currency: {currency}') # Mask card number for logging masked = card_number[-4:].rjust(len(card_number), '*') # Calculate processing fee (2.9% + 30p) fee = round((amount * 0.029) + 0.30, 2) total = amount + fee # Simulate gateway call transaction_id = f'TXN-{card_number[-4:]}-{int(amount*100)}' return { 'status': 'success', 'transaction_id': transaction_id, 'amount': amount, 'fee': fee, 'total': total, 'masked_card': masked } A payment processing function. Straightforward. Plausible. In production somewhere right now. A Test Suite That Achieves 100% Coverage Python # test_payment.py import pytest from payment import process_payment def test_valid_payment(): result = process_payment(100.00, '4111111111111111') assert result['status'] == 'success' def test_negative_amount_raises(): with pytest.raises(ValueError): process_payment(-10.00, '4111111111111111') def test_zero_amount_raises(): with pytest.raises(ValueError): process_payment(0, '4111111111111111') def test_unsupported_currency_raises(): with pytest.raises(ValueError): process_payment(100.00, '4111111111111111', currency='JPY') Four tests. Every line, branch, and condition in process_payment() is executed. The coverage tool is satisfied. The Coverage Report Running this suite with pytest-cov produces the following output: Python $ pytest test_payment.py --cov=payment --cov-report=term-missing ================================================================= platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 collected 4 items test_payment.py .... [100%] ----------- coverage: platform linux, python 3.11.4 ----------- Name Stmts Miss Cover Missing ---------------------------------------------- payment.py 14 0 100% ---------------------------------------------- TOTAL 14 0 100% 4 passed in 0.12s Four tests passing. 100% code coverage. Every statement executed. Every branch taken. The CI pipeline is green. The coverage badge on the repository is green. The engineering manager's dashboard is green. Now let us examine what these tests do not verify — what the 100% coverage number is actively concealing. What the 100% Coverage Is Hiding The test suite above exercises every line. It verifies almost nothing about behavior. Here is a systematic account of the failures it will not catch. Failure 1: The Fee Calculation Is Never Verified The processing fee calculation — the line that determines how much money is actually charged — is executed by test_valid_payment() but never asserted against. The test checks that status is 'success'. It does not check that the fee is correct, that the total is correct, or that the relationship between amount, fee, and total is arithmetically sound. # This calculation runs. It is never checked. fee = round((amount * 0.029) + 0.30, 2) total = amount + fee # For amount=100.00: # fee should be: (100.00 * 0.029) + 0.30 = 2.90 + 0.30 = 3.20 # total should be: 100.00 + 3.20 = 103.20 # Now introduce a bug: fee = round((amount * 0.29) + 0.30, 2) # 0.29 instead of 0.029 # fee becomes: 29.30 # total becomes: 129.30 # Coverage: still 100%. Tests: still passing. Customer: charged 29% instead of 2.9%. The bug is a single decimal point. The coverage tool cannot see it. Neither can any of the four tests. Failure 2: The Card Masking Is Never Verified The masked card number is returned in the response and presumably used in receipts, logs, and customer communications. The masking logic runs during test_valid_payment(). It is never asserted against. # This runs. It is never checked. masked = card_number[-4:].rjust(len(card_number), '*') # For card_number = '4111111111111111' (16 digits): # masked should be: '************1111' # Introduce a bug that exposes the full card number in logs: masked = card_number # accidentally log the full number # Coverage: still 100%. Tests: still passing. # PCI-DSS compliance: violated. Customer data: exposed. A PCI-DSS violation that 100% coverage cannot see, because coverage does not check what is returned — only that the line ran. Failure 3: The Transaction ID Embeds Unmasked Data The transaction ID construction is never examined. As written, it embeds the last four digits of the card number — which may or may not be acceptable depending on where transaction IDs are stored and logged. But more critically, if the construction logic changes in a way that embeds more card data, no test will catch it. # Transaction ID construction — executed, never verified. transaction_id = f'TXN-{card_number[-4:]}-{int(amount*100)}' # Change to accidentally embed more card data: transaction_id = f'TXN-{card_number}-{int(amount*100)}' # Coverage: 100%. Tests: passing. # Audit log: now contains full, unmasked card numbers. Failure 4: Floating-Point Currency Handling Is Untested The function handles currency arithmetic using floating-point numbers. Anyone who has worked with financial systems knows that floating-point arithmetic and money are a dangerous combination. The tests use clean round numbers (100.00, -10.00). They never probe what happens with amounts like 99.99, or 0.01, or values that produce floating-point rounding artifacts. # What actually happens with some real-world amounts: # amount = 99.99 # fee = round((99.99 * 0.029) + 0.30, 2) # fee = round(2.89971 + 0.30, 2) # fee = round(3.19971, 2) = 3.20 <- acceptable # amount = 19.99 # fee = round((19.99 * 0.029) + 0.30, 2) # fee = round(0.57971 + 0.30, 2) # fee = round(0.87971, 2) = 0.88 <- acceptable # But with Decimal arithmetic, the story differs. # The function uses float, not Decimal. # For high-volume systems, accumulated rounding errors # across thousands of transactions produce discrepancies # that appear in reconciliation reports months later. # Tests with clean inputs: passing. # Coverage: 100%. # Finance team's reconciliation nightmare: upcoming. The tests never probe the arithmetic with values that expose floating-point behaviour. Coverage does not notice. Failure 5: No Testing of What Happens When the Gateway Fails The function simulates a gateway call. In a real implementation, this would be a network call to a payment processor. The simulation always succeeds. The tests never ask: what happens when the gateway times out? What happens when it returns an error? What happens when it returns a malformed response? The coverage tool reports 100% on a function that has never been tested under its most important real-world condition: failure. Coverage Report and Confidence 100% line coverage: confirmed. Every line of process_payment() executes during the test run. Fee calculation correctness: unverified. A decimal point error charges customers 29% instead of 2.9%. Card masking correctness: unverified. A one-line change exposes full card numbers in logs. Transaction ID safety: unverified. A refactor can embed unmasked card data in audit logs. Floating-point precision: untested. Financial reconciliation errors accumulate silently. Gateway failure handling: untested. The function has never been tested under the condition it will most frequently encounter in production during incidents. The coverage number: 100%. The confidence it should provide: close to zero. Coverage of Behavior vs. Coverage of Code The demonstration above makes the distinction concrete. Now it can be stated precisely. Code coverage measures the proportion of source code statements, branches, or conditions that are executed during a test run. It is a property of the test suite's interaction with the code. Behavior coverage measures the proportion of the system's meaningful behaviors. Like the things it is supposed to do, and the things it must not do — that are verified by the test suite. It is a property of the test suite's relationship to the system's specification and risk profile. A test suite can achieve 100% code coverage while covering almost none of the system's meaningful behaviors — as the payment example demonstrates. Conversely, a test suite with 60% line coverage, if designed against the system's risk profile, can verify the behaviors that matter most and leave only low-consequence code unexecuted. This is not an argument against measuring code coverage. Coverage data is useful: it identifies code that is never exercised by any test, which is a meaningful signal. Uncovered code is a known unknown — you have no evidence about its behavior whatsoever. But covered code is not the same as thoroughly tested code. This distinction must be understood by anyone using code coverage metrics to make decisions. dimensioncode coveragebehavior coverage What it measures Proportion of lines/branches executed Proportion of meaningful system behaviors verified What 100% means Every line was executed at least once Every significant behavior has been verified — including error cases, boundaries, and failure modes What 0% means No code was executed (tests did not run)No meaningful behavior has been verified — even if tests pass What tools can measure it Automated, precise, available in all CI pipelines Requires human judgment, risk analysis, and specification review What it predicts Which code paths the tests reach. Not correlated with defect detection effectiveness The likelihood that important defects have been detected. Correlates with production quality outcomes Its most dangerous failure mode False confidence from tests that execute code without verifying it Scope blindness — failing to identify which behaviors are meaningful in the first place What Good Coverage Thinking Actually Looks Like Rejecting the coverage illusion does not mean abandoning the code coverage measurement. It means using coverage data correctly — as one signal among several, interpreted in the context of risk, rather than as a proxy for quality. The following approach is not a framework to mandate. It is a way of reasoning that produces better decisions than a percentage target. Step 1: Identify the Behaviors That Matter Before writing a single test, the question is: what does this system do that, if wrong, would cause harm? For the payment function, sample behaviors include charging the correct amount, protecting card data, generating accurate transaction records, and handling failures gracefully. These are the behaviors that tests must cover. They are determined by the system's risk profile, not by its line count. # Behaviour inventory for process_payment() # CRITICAL (failure causes financial or compliance harm): # - Fee calculation produces correct result for all valid inputs # - Total = amount + fee, always # - Card masking produces no more than last 4 digits in any output # - Transaction IDs contain no sensitive data # - Rejected amounts (<=0) never produce a transaction # - Unsupported currencies never produce a transaction # IMPORTANT (failure causes degraded service): # - Gateway timeout returns a defined error state # - Gateway error returns a defined error state # - Malformed gateway response is handled without exception leak # STANDARD (failure causes minor friction): # - Masked card format is consistent # - Transaction ID format is consistent # A test suite designed against this inventory will look # very different from a test suite designed to hit 80% coverage. # It will also tell you far more about whether the system is safe to run. A behavior inventory. Written before tests, not derived from coverage reports after them. Step 2: Write Tests Against the Behavior Inventory # Tests designed against the behavior inventory class TestFeeCalculation: """CRITICAL: fee must be exactly 2.9% + £0.30, rounded to 2dp""" def test_fee_standard_amount(self): result = process_payment(100.00, '4111111111111111') assert result['fee'] == 3.20 assert result['total'] == 103.20 def test_fee_small_amount(self): result = process_payment(1.00, '4111111111111111') assert result['fee'] == 0.33 # (1.00 * 0.029) + 0.30 assert result['total'] == 1.33 def test_fee_high_value_transaction(self): result = process_payment(9999.99, '4111111111111111') assert result['fee'] == 290.30 # (9999.99 * 0.029) + 0.30 assert result['total'] == 10290.29 def test_total_equals_amount_plus_fee(self): """Invariant: total must always equal amount + fee exactly""" for amount in [0.01, 1.00, 19.99, 99.99, 1000.00]: result = process_payment(amount, '4111111111111111') assert result['total'] == round(result['amount'] + result['fee'], 2) class TestCardDataProtection: """CRITICAL: no output may expose more than last 4 digits""" def test_masked_card_hides_all_but_last_four(self): result = process_payment(100.00, '4111111111111111') assert result['masked_card'] == '************1111' assert '411111111111' not in result['masked_card'] def test_transaction_id_contains_no_sensitive_data(self): result = process_payment(100.00, '4111111111111111') # Only last 4 digits permissible in transaction ID assert '411111111111' not in result['transaction_id'] def test_no_field_in_response_contains_full_card(self): card = '4111111111111111' result = process_payment(100.00, card) for key, value in result.items(): assert card not in str(value), \ f'Full card number found in field: {key}' Tests designed against the behavior inventory. They may not achieve 100% line coverage. They verify the things that matter. Step 3: Use Coverage Data to Find Gaps, Not to Set Targets After writing tests against the behavior inventory, run the coverage report — not to check whether you have hit a target, but to identify code that no test reaches. Uncovered code is a signal that deserves investigation. It may be dead code that should be deleted. It may be an error path that no test exercises. It may be a code path that your behavior inventory missed. # Using coverage as a gap-finder, not a target # After running behaviour-driven tests, the coverage report shows: # # payment.py Stmts Miss Cover Missing # ------------------------------------------------------- # payment.py 22 3 86% 45-47 # # Lines 45-47 are the gateway simulation block. # They are uncovered because no test exercises the failure path. # # This is the coverage report doing its job correctly: # it has identified a gap in the behaviour inventory. # The response is to ask: 'What happens on lines 45-47, # and should we have a test for it?' # NOT: 'How do we get from 86% to 90%?' 86% coverage that identifies a meaningful gap is more useful than 100% coverage that conceals one. The Mutation Testing Alternative If code coverage is an unreliable measure of test quality, is there a better one? There is, and it is called mutation testing. It is more computationally expensive than coverage measurement, but it measures something that coverage cannot: whether your tests are capable of detecting changes in the code's behavior. Mutation testing works by automatically introducing small, deliberate changes — mutations — into the source code, then running the test suite against each mutated version. If a mutation causes a test to fail, the mutation is "killed" — the tests detected the behavioral change. If all tests still pass despite the mutation, the mutation "survived" — the tests failed to detect a change in behavior that a developer could easily introduce. # Original code fee = round((amount * 0.029) + 0.30, 2) # Mutation 1: change operator fee = round((amount * 0.029) - 0.30, 2) # survived: no test checks fee value # Mutation 2: change constant fee = round((amount * 0.29) + 0.30, 2) # survived: no test checks fee value # Mutation 3: negate condition if amount >= 0: # original: amount <= 0 raise ValueError('Amount must be positive') # survived? only if boundary untested # A high mutation score means your tests are sensitive to behavioral changes. # A low mutation score — even with 100% line coverage — means your tests # are not detecting changes that matter. Mutation testing reveals what coverage cannot: whether your tests would catch a developer accidentally changing the logic. Mutation testing is not a replacement for thoughtful test design. It is a diagnostic tool that exposes weak assertions and under-specified tests with a degree of precision that coverage metrics cannot approach. A test suite with 85% line coverage and a 90% mutation score is demonstrably stronger than a test suite with 100% line coverage and a 40% mutation score. For most organizations, mutation testing is not yet part of the standard CI pipeline — it is computationally expensive and requires configuration effort. But even running it periodically, on a sample of the codebase, provides more meaningful information about test quality than a continuous coverage percentage ever will. The Questions Executives Should Be Asking Coverage percentages appear in engineering reports, sprint reviews, and board-level quality dashboards. They are communicated as evidence of quality. Most of the people receiving them do not know that they are receiving a measure of code execution, not confidence. This is not a technical problem. It is an information design problem. The people making decisions based on coverage numbers have never been given the vocabulary to question them. The following set of questions changes that. They are designed to be asked by any engineering leader, with or without a technical background. They probe the dimensions of quality that coverage metrics conceal. The Executive's Coverage Questions 1. What behaviors does this number not measure? Ask the team to identify the five most important things the system does and confirm that each of those behaviors has dedicated tests with meaningful assertions. 2. What is our mutation score? If the team cannot answer this, the coverage percentage is the only quality signal they have — and it is a weak one. 3. What are the most important failure modes for this system, and are they tested? A system's failure modes are usually more important than its success paths. Financial corruption, data exposure, and service unavailability all require dedicated tests. Coverage numbers do not distinguish these from a test of a utility function. 4. What is uncovered, and why? The answer to this question is more useful than the coverage number itself. Uncovered code is a map of known unknowns. Understanding why those regions are uncovered reveals risk. 5. How does our coverage number change when we exclude assertion-free tests? Most teams will not have run this analysis. Asking for it creates a productive forcing function. 6. When did we last find a defect through our test suite rather than through production? This is the ground-truth question. If the last production incident involved code with high coverage, the coverage number needs to be interrogated, not reported. These questions do not require technical depth to ask. They require only the understanding that coverage measures execution, not confidence. Understanding that the gap between those two things is where many production defects live. The Uncomfortable Organizational Truth Coverage metrics persist not because they are accurate but because they are easy. They are generated automatically by widely available tools. They produce a single number that is simple to track, to report, and to include in a dashboard. They create the impression of accountability without requiring the harder work of defining what accountability for quality actually means. The organizations that have replaced coverage targets with behavior-oriented quality measures consistently report the same initial reaction from their teams: the work becomes harder to measure but easier to reason about. Engineers stop asking "have I covered this code?" and start asking "have I verified the behavior this code is supposed to produce?" The shift is subtle in vocabulary but very significant in practice. A harder truth is that behavior coverage is often more difficult to achieve. It requires someone to think about what the system is supposed to do, what it must not do, and what the consequences of failure in each area would be. This is skilled work. It is the kind of work that, in most organizations, is either not assigned to anyone or is assigned to QA teams under a job description that asks them to find bugs rather than define the behavior surface of the system. Fixing the coverage illusion requires that someone has the authority and the charter to define what behavior coverage means for a given system. It requires that engineering teams are held accountable for achieving it, even though it cannot be measured with a single percentage. Even though it requires more judgment than automation, and even though it is harder to put on a dashboard than a green badge. Wrapping Up So you've written your code and your tests, and you achieve 100% code coverage. What does that tell you about your tests? It tells you very little. Are they enough or not? You don't know if they are enough or not just by looking at code coverage. This metric measures which lines of code a test suite executes. It does not measure what portion of the behavior space of those lines has been verified. A test suite can achieve 100% line coverage while leaving critical financial calculations unverified, card data unprotected, and failure modes completely untested. The distinction between coverage of code and coverage of behavior is not semantic. It is the difference between a number on a dashboard and evidence of system quality. Coverage data is useful as a gap-finder to identify code that no test reaches. However, it is dangerous as a quality proxy, because it conceals the difference between tests that execute code and tests that verify behavior. Coverage targets make this worse, not better, because they create an incentive to optimize for the number rather than for the evidence. The rational response to a coverage target is to achieve it by writing meaningful tests that cover the behavior surface of the system. conceptprecise definition Code coverage The proportion of source code statements, branches, or conditions executed at least once during the test run. Measured automatically. Does not indicate correctness. Behaviour coverage The proportion of the system's meaningful behaviors — success paths, error conditions, boundaries, failure modes — that are verified by the test suite. Requires human judgment to define and assess. Mutation score The proportion of automatically-introduced code mutations that the test suite detects. A direct measure of the test suite's sensitivity to behavioral change. More meaningful than line coverage as a quality signal. Assertion-free test A test that executes code but makes no meaningful claim about its output. Generates code coverage without generating evidence. The most common cause of 100% coverage coexisting with catastrophic defects. Coverage target A minimum coverage percentage enforced at CI level. Incentivizes coverage optimization rather than evidence generation. Does not improve quality; frequently degrades it by displacing investment from meaningful tests to coverage-padding tests. Behaviour inventory A pre-test enumeration of the behaviors a system must exhibit, categorized by consequence of failure. The correct foundation for a test suite.

By Stelios Manioudakis CORE

Everyone is talking about how magical AI is right now, but if you have spent any time experimenting with it recently, you have probably realized how difficult it is to get the results you want. None of the hype is particularly useful when you are trying to build something real. The magic looks good on paper until it meets real systems. I recently put together a talk called "Agents, Tools, and MCP, oh my!" that tries to cut through some of that noise. As developers, we are being handed a firehose of new tools and technologies, and I wanted to spend my session doing something a little different: break the pieces apart, reduce some of the complexity and overwhelm, and then build them back up so they actually fit together. This post is the architecture piece. It lays out the mental model and the "why" behind each layer. If you want to skip ahead, the code is already on GitHub, built with Java, Spring AI, and Neo4j, using a dataset of books, authors, and reviews (because I like to read, and it turns out reading data makes a great demo domain). How We Got Here None of this complexity showed up all at once. A couple of years ago, the foundation of the AI stack was just the large language model, on its own. That was pretty good, until it wasn't: ask it anything that required knowledge of your users or your data, and it had nothing to work with. So we stacked on vector search and did retrieval-augmented generation (RAG), also known as naive or easy RAG. That improved things, and then it hit its own wall: retrieval that was too shallow, too literal, missing the relationships between things that actually mattered. So we added filtering and traversals (advanced RAG, GraphRAG) to pull in more precisely related content. That solved the retrieval problem well enough that a new one became visible: now there were too many pieces to coordinate by hand, so we brought in an agent to sit in the middle and decide what to call and when. Then it turned out the agent had no memory of anything it had already done, so state and history got added on top of that. And once you point any of this at production, you inherit a whole new set of concerns: evals, guardrails, security, all the checks and balances that scale demands. Layers of the 2026 AI stack Every one of those layers was added because the one below it hit a wall. None of this was designed top-down as a system; it was built one patch at a time, in response to the gaps. This means you should evaluate for your own system which layers make the overall solution better and skip those that don't. More Layers Do Not Mean Better The evaluation of each layer matters because more does not equal better. At some point, your complexity outweighs the value you are getting back from it. I think about this the same way I think about desserts (I like food). A layered dessert with more textures and flavors is more fun to eat, up to a point. A croissant with more layers of butter and dough is flakier and more interesting, up to a point. But stack too many layers on a dessert, and it turns to mush. Stack too many layers of dough on a croissant, and the weight collapses the whole thing in the oven before it ever gets to rise. Tech stacks behave the same way. Somewhere past a certain point, adding another layer stops buying you anything and starts costing you: slower development, harder debugging, more surface area to maintain. There is no one-size-fits-all stack that solves this for you. What I want to hand you instead is a set of building blocks, so you can decide for yourself, layer by layer, whether your problem actually needs it, rather than reaching for whatever is newest or most talked about. Four Acts, Built Like a Piece of Music I am a musician by background, so I built the talk like a piece of music: four movements, each one earning its place by doing something the last one genuinely could not. That structure turned out to map cleanly onto code, and it is the structure I am using for this whole series. Act one is a plain LLM, on its own, and it is worth spending real time here because most of us already live in this act without noticing it. Send it a question, get a fluent answer back, right up until the question requires knowing something specific about your users or your data, at which point it either guesses or admits defeat. That gap, between confident reasoning and zero access to anything real, is the entire reason the next three acts exist. Act two hands the model a way to ask for real data instead of inventing it: structured, typed tool calls instead of a prompt hoping to be obeyed. This is where an agent stops being a buzzword and starts being a reasoning loop you can actually debug: receive input, decide what tool to call, execute it, look at the result, and either answer or loop again. Agent reasoning loop Act three deals with the fact that an LLM forgets everything the moment a request ends. Rather than re-explaining the whole conversation on every turn, memory becomes something the system is responsible for, not the model, and a graph turns out to be a natural place to hold both the short-term thread of a conversation and the long-term knowledge that should persist across many of them. Graph as application memory Act four takes the tools built in act two and pulls them out from underneath the application entirely, using MCP so that a tool definition is not welded to one model, one app, or one team. Swap providers, build a second application, share tools across a team, none of it should require rewriting the integration from scratch, and MCP helps make that happen. Architecture with MCP and Neo4j Stepping back, those four acts are really four layers doing four distinct jobs: the LLM reasons, the tools execute, the graph holds context, and MCP standardizes how everything connects. None of that is magic. It is composable architecture, which is genuinely good news, because composable things can be designed, tested, and swapped out independently, and you can actually reason about what broke when something does. A Better Question to Start With That reframes the whole problem. "How do we build an AI agent?" makes it sound like the agent is the hard part, the thing you optimize. It's not. The large language model, honestly, is not the most interesting piece of any of this. What matters is everything you build around it: an agent that decides, tools that act, a graph that remembers, a protocol that keeps it all from being welded together. Four layers of modern AI systems These are not mysterious, unbuildable things. They are composable layers, and composable layers are something developers already know how to design, test, and put back together differently when the situation calls for it. None of this is magic happening to your application. You are still the one designing the system. The model is just one component inside it. The next task is to build your solution one act at a time and watch where it actually holds up versus where it needs a second look. Act 1 starts with the plain LLM, the same one most of us are already living in without noticing, and shows exactly where it runs out of road. Happy coding! Resources Code repository: Agents, Tools, and MCP demo (Java, Spring AI, Neo4j)Slide deck: Agents, Tools, and MCP, oh my! (Devnexus 2026)Course: Developing with Neo4j MCP Tools (GraphAcademy)Course: Context Graphs: Agent Memory with Neo4j (GraphAcademy)Documentation: Spring AI Tool Calling

By Jennifer Reif CORE

Knowing what people say about your product usually means checking Google News, scrolling through YouTube, and digging into different social media threads. That's three tabs, three interfaces, and no way to compare what you find. This tutorial builds a single dashboard that pulls brand mentions from all three sources using Python and SerpApi. By the end, you'll have a Streamlit app with three tabs, one for news articles, one for YouTube videos, and one for social media and forum discussions. We'll use "serpapi" as the search query, but you can swap the brand or product name. Brand monitoring dashboard showing metrics row with total mentions, news articles, YouTube videos, and perspectives counts Set Up Your Environment Requirements: Python 3.8+SerpApi API Key (the free plan includes 250+ searches/month)Dependencies (serpapi, pandas, streamlit, altair) The serpapi package is the official Python SDK. It handles request signing, retries, and response parsing. The complete code, including a Jupyter notebook version, is available in the SerpApi tutorials repository. The Pipeline The app follows the same three-step pattern from the GitHub Issues dashboard: fetch raw data, transform it, and display the analysis. Pipeline diagram showing three stages: fetch, transform, and display The difference this time is three separate engines running in parallel. Each returns a different response structure, so the transform step normalizes everything into DataFrames before the dashboard consumes it. Fetch the Data A single SerpApi client instance works for all three engines: Python import serpapi import os SERPAPI_KEY = os.environ.get("SERPAPI_KEY", "") client = serpapi.Client(api_key=SERPAPI_KEY) Google News The Google News API returns articles through the news_results key. Each result includes title, link, source (a dict with name and icon), date, and snippet. Python def fetch_news(client, brand): """Fetch news articles mentioning the brand via Google News.""" results = client.search({ "engine": "google_news", "q": brand, "gl": "us", "hl": "en", }) return results.get("news_results", []) For more use cases with this engine, refer to the news monitoring. YouTube The YouTube Search API uses search_query instead of q, and the sp parameter controls time filters. The values EgIIAw%3D%3D (this week) and EgIIBA%3D%3D (this month) are YouTube's internal encoding for upload date filters. You can grab these from YouTube's URL bar after applying a filter manually. We run both filters and deduplicate by link, since the month results include everything from the week: Python YT_FILTER_WEEK = "EgIIAw%3D%3D" YT_FILTER_MONTH = "EgIIBA%3D%3D" def fetch_youtube(client, brand): """Fetch YouTube videos, combining week and month filters.""" seen = set() videos = [] for sp_filter in (YT_FILTER_WEEK, YT_FILTER_MONTH): results = client.search({ "engine": "youtube", "search_query": brand, "sp": sp_filter, }) for video in results.get("video_results", []): link = video.get("link", "") if link and link not in seen: seen.add(link) videos.append(video) return videos For more examples using the YouTube API, refer to this link. Google Perspectives Google Perspectives API surfaces user-generated content from LinkedIn, Reddit, Quora, and blogs. It uses the standard Google engine, and the results appear under the perspectives key: SerpApi search with the Google perspective results Python def fetch_perspectives(client, brand): """Fetch user-generated content (Reddit, LinkedIn, Quora).""" results = client.search({ "engine": "google", "q": brand, "google_domain": "google.com", }) return results.get("perspectives", []) Fetch in Parallel Three sequential API calls take roughly three seconds. Running them in parallel with Python ThreadPoolExecutor brings that down to about one second. Each call runs in its own thread while the others wait for their response: Python from concurrent.futures import ThreadPoolExecutor @st.cache_data(ttl=300) def fetch_all_mentions(brand): """Fetch all brand mentions from three engines in parallel.""" client = serpapi.Client(api_key=SERPAPI_KEY) with ThreadPoolExecutor(max_workers=3) as pool: news_future = pool.submit(fetch_news, client, brand) yt_future = pool.submit(fetch_youtube, client, brand) persp_future = pool.submit(fetch_perspectives, client, brand) return news_future.result(), yt_future.result(), persp_future.result() SerpApi also offers a server-side async parameter for large-scale batch processing, where you submit searches and retrieve results later. For our three concurrent calls, client-side threading is simpler and equally effective. The @st.cache_data(ttl=300) decorator caches results for 5 minutes. Without it, every Streamlit interaction would re-trigger the API calls. This works alongside SerpApi's own 1-hour result cache, which serves identical queries from the cache at no extra search cost unless you explicitly pass no_cache=true. Together, these two layers minimize redundant API calls during development and testing. For more optimization techniques when working with SerpApi at scale, refer to this blog. Transform the Data All three engines return dates as relative strings ("3 hours ago", "2 days ago"). We need a shared parser to convert them into datetime objects for sorting. Parse Relative Dates Two details worth noting. The regex is compiled once and reused since this function runs for every result in all three engines. And the fallback returns datetime.now(timezone.utc) instead of None, so results without a parseable date sort to the top rather than breaking pandas operations. Python import re from datetime import datetime, timedelta, timezone RELATIVE_DATE_RE = re.compile( r"(\d+)\s+(second|minute|hour|day|week|month|year)s?\s+ago", re.IGNORECASE ) UNIT_TO_TIMEDELTA = { "second": lambda n: timedelta(seconds=n), "minute": lambda n: timedelta(minutes=n), "hour": lambda n: timedelta(hours=n), "day": lambda n: timedelta(days=n), "week": lambda n: timedelta(weeks=n), "month": lambda n: timedelta(days=n * 30), "year": lambda n: timedelta(days=n * 365), } def parse_relative_date(text): """Convert '3 hours ago' into a datetime object.""" if not text: return datetime.now(timezone.utc) match = RELATIVE_DATE_RE.search(str(text)) if not match: return datetime.now(timezone.utc) amount = int(match.group(1)) unit = match.group(2).lower() delta = UNIT_TO_TIMEDELTA.get(unit, lambda n: timedelta())(amount) return datetime.now(timezone.utc) - delta Build DataFrames Each engine gets into its own transformer. Here's the news version: Python def transform_news(results): """Convert raw Google News results into structured records.""" records = [] for item in results: source = item.get("source") or {} source_name = source.get("name", "Unknown") if isinstance(source, dict) else str(source) records.append({ "title": item.get("title", ""), "link": item.get("link", ""), "source": source_name, "date": parse_relative_date(item.get("date", "")), "snippet": item.get("snippet", ""), }) return records The source field can be a dict or a plain string depending on the result, so the isinstace check handles both. YouTube and Perspectives follow the same pattern, with two differences worth highlighting. YouTube views come back as strings like "1,234 views", so we strip non-numeric characters before converting: Python views = item.get("views") or 0 if isinstance(views, str): views = int(re.sub(r"[^\d]", "", views) or 0) Build the Dashboard The Streamlit interface starts with a form for the brand query and a row of summary metrics across all three sources: Python st.set_page_config(page_title="Brand Monitoring Dashboard", layout="wide") st.title("Brand Monitoring Dashboard") with st.form("brand_form"): brand = st.text_input("Brand or keyword to monitor", value="serpapi") submitted = st.form_submit_button("Search") Brand or keyword selector to monitor After fetching, the dashboard shows four metrics at the top for a quick overview, then splits into three tabs: Python col1, col2, col3, col4 = st.columns(4) col1.metric("Total Mentions", total_mentions) col2.metric("News Articles", len(news_records)) col3.metric("YouTube Videos", len(yt_records)) col4.metric("Perspectives", len(persp_records)) Dashboard metrics row displaying total mentions across three sources News Tab The News tab pairs an Altair bar chart of top sources with a sortable table. Altair ships with Streamlit, so there's nothing extra to install. We use it instead of st.bar_chart because it gives control over orientation, tooltips, and styling. Python source_df = news_df["source"].value_counts().head(10).reset_index() source_df.columns = ["source", "count"] source_chart = alt.Chart(source_df).mark_bar( cornerRadiusTopRight=4, cornerRadiusBottomRight=4 ).encode( x=alt.X("count:Q", title="Articles"), y=alt.Y("source:N", sort="-x", title=""), color=alt.value("#4A90D9"), tooltip=["source:N", "count:Q"], ).properties(height=350) st.altair_chart(source_chart, use_container_width=True) News tab with horizontal bar chart of top sources and sortable article table The table uses st.column_config.LinkColumn so each article title links directly to its source. YouTube Tab The YouTube tab shows views by channel and a sorted video table. The chart groups views by channel to surface which creators talk about the brand the most. Python channel_df = yt_df.groupby("channel")["views"].sum().reset_index() channel_df = channel_df.sort_values("views", ascending=False).head(10) channel_chart = alt.Chart(channel_df).mark_bar( cornerRadiusTopRight=4, cornerRadiusBottomRight=4 ).encode( x=alt.X("views:Q", title="Views", axis=alt.Axis(format="~s")), y=alt.Y("channel:N", sort="-x", title=""), color=alt.value("#4A90D9"), tooltip=["channel:N", alt.Tooltip("views:Q", format=",")], ).properties(height=350) YouTube tab showing views by channel chart and video table Perspectives Tab The Perspectives tab splits the layout between a discussion table on the left, and a donut chart of mentions by platform on the right. The donut chart makes it easy to see where conversations happen, whether it's LinkedIn, Reddit, X, etc. Python platform_chart = alt.Chart(platform_df).mark_arc( innerRadius=60, outerRadius=120 ).encode( theta=alt.Theta("count:Q"), color=alt.Color("source:N", legend=alt.Legend(title="Platform")), tooltip=["source:N", "count:Q"], ).properties(height=350) Perspectives tab with discussions table on the left and donut chart of mentions by platform on the right When to Use This Approach Ideal for: Tracking brand mentions across news, video, and social in one viewMonitoring product launches, PR campaigns, or competitor namesBuilding internal dashboard for marketing or DevRel teams Not recommended for: Real-time alerting. The API returns a snapshot, not a stream. For notifications, schedule the script on an interval and compare results.Historical analysis. Each engine returns recent results, not a complete archive. If you want to explore the API response before writing code, the SerpApi Playground lets you test any engine interactively. And if you only need news coverage, the Google News API alone handles most brand monitoring use cases. Where to Go from Here This dashboard gives you a live snapshot. The natural next step is turning it into a historical record. Store each fetch in a database (SQLite, PostgreSQL, or even a CSV), and you can compare mention volume week over week, track which sources cover your brand consistently, and spot trends that a single snapshot can't show. With historical data in place, you can layer on more analysis. Identify content gaps by looking at what topics competitors get covered on, but you don't. Track which YouTube channels mention your product and how their view counts trend over time. Flag new platforms or authors that start discussing your brand. The data is yours to work with however fits your needs. The three engines give you the raw material; what you build on top depends on the questions you're trying to answer. Conclusion The full application is about 350 lines in a single Python file. Three API calls, three DataFrames, three tabs. The query input at the top lets you switch brands without changing the code. What started as a way to check where "serpapi" shows up on the web became a tool that surfaces patterns you miss manually. The Perspectives tab pulls in LinkedIn posts, Reddit threads, and Quora answers that don't appear in regular news or video searches, and combining them in one view gives you the full picture. Check out the full SerpAPI article collection here.

Performance defects in Pega rarely present as a single, obvious fault. A slow harness render, an unexpected stage transition, a case that opens correctly but saves slowly, or a data page that intermittently returns stale values can all originate in very different layers of the runtime. Effective diagnosis depends on separating timing, execution flow, and in-memory state instead of treating them as one problem. That distinction is exactly why PAL, Tracer, and Clipboard remain the most practical diagnostic combination in Pega. PAL exposes cumulative and incremental resource usage for a requestor session without adding measurement overhead of its own. Tracer reconstructs the sequence of rule execution events but is intentionally heavyweight. And Clipboard reveals the runtime pages and property values that drive case behavior. Used together and in the right order, these tools turn debugging from guesswork into a disciplined tuning workflow. When the signal is still ambiguous The first useful clue is often not a rule at all but an alert or a user-facing symptom. Pega’s alert model is designed to signal threshold violations rather than explain their root cause. A PEGA0001 alert reports that total HTTP interaction time exceeded the configured threshold and commonly acts as an umbrella symptom for slower work underneath, including database waits, rule assembly, or external dependency latency. A PEGA0004 alert indicates that a database query loaded more data into memory than expected, a PEGA0035 alert points to an oversized Page List, and a PEGA0050 alert identifies inefficient copying of a clipboard page list into another list. Those signals are valuable because they narrow the class of problem before any rule is opened. That is why PAL should be the starting point instead of Tracer. PAL can isolate which interaction, screen render, or submit action is expensive, and Pega Academy explicitly recommends incremental readings for each form or step so that the costly portion of a process becomes visible in context. When the problem is reported by a specific operator, the same PAL-style view can also be reviewed through session-oriented performance details rather than relying on a broad system impression. That approach prevents wasted time tracing the wrong request or inspecting the wrong clipboard state. Let PAL narrow the interaction PAL is most effective when treated as a timeline rather than a dashboard. The process begins with Reset Data, followed by a warm-up run, and then a controlled set of readings. Pega documents three reading types that matter: INIT for the first capture, DELTA for the change since the prior reading, and FULL for the cumulative totals since reset. It also groups measurements into Elapsed, CPU, and Count, which matters because elapsed time can grow while CPU stays modest, indicating waits in the database or remote calls rather than raw computation. The warm-up run is critical because first-use assembly can heavily distort results; Pega’s own PAL guidance notes that rule assembly time can dominate an early reading and should be removed from the picture before real tuning begins. A pattern like the following is a common PAL suspect when a DELTA suddenly shows inflated database counts, commit counts, and overall elapsed time: Plain Text Step 1: Obj-Browse OrderLineList Step 2: For Each Page In OrderLineList.pxResults Step 3: Obj-Open-By-Handle .pzInsKey Step 4: Property-Set .TotalAmount = .TotalAmount + .Amount Step 5: Commit The problem is not only the loop. Obj-Browse copies instances, or selected properties, to the clipboard as embedded pages, so the initial fetch already has a clipboard cost. Adding Obj-Open-By-Handle and Commit inside the iteration multiplies database activity and raises the chance that the interaction will surface as slow HTTP time or data-heavy query behavior. PAL does not name the bad line, but it makes the shape of the defect visible quickly: a DB-heavy DELTA, exaggerated elapsed time, and a step boundary that is now narrow enough for deeper inspection. The corrected version is usually less dramatic than the defect. It simply collapses unnecessary round trips and leaves one transaction boundary where one boundary belongs: Plain Text Step 1: Report Definition GetOpenOrderLines Step 2: For Each Page In OrderLineList.pxResults Step 3: Property-Set .TotalAmount = .TotalAmount + .Amount Step 4: Commit After a change like this, PAL should be rerun with one DELTA per form submission or render, and at least one reading should include clipboard size. Pega’s guidance is explicit that Add Reading with Clipboard Size takes longer to calculate, but it is the most direct way to confirm whether a tuning change reduced memory pressure rather than merely shifting time from one phase to another. That matters because large result sets do not only slow the database path; they also inflate requestor memory, and Pega’s performance guidance notes that paging report results reduces both clipboard size and display time. Let Tracer explain the execution path Once PAL identifies the expensive interaction, Tracer becomes useful, but only within a tight scope. Pega describes Tracer as a troubleshooting tool that logs the sequence of execution events during runtime, with each event shown as a row identified by thread, event type, and status. Activity processing appears in gray, flow, decision, and declarative activity in orange, and database or cache operations in light blue. The same guidance also warns that Tracer significantly impacts application performance and should not be used as the primary performance-analysis tool. That warning is not ornamental. A broad trace can easily create noise, distort timing, and hide the event that actually matters. A focused trace configuration is usually more valuable than a long trace: Plain Text Events to trace: Activities, Declarative Rules, Decisions, DB/cache Rulesets to trace: MyApp Break conditions: Exception, Java Exception Watch: pyWorkPage.PolicyStatus pyWorkPage.TotalPremium This kind of setup aligns with Pega’s Tracer guidance on refining event logging, using breakpoints and watch values, and constraining the captured rulesets and event types. The watch list is especially effective when a property becomes wrong before it becomes visibly wrong in the UI, because Tracer can pause on the transition and expose the exact event where the value changed. The event detail window can then display the page state at that moment, while the Step Page, Parameter Page, and Primary Page views reveal whether the defect came from a parameter mismatch, stale clipboard state, or an unexpected declarative recalculation. In Constellation applications, that sequence is grouped by request ID because multiple DX API requests can be in flight during one browser session, which changes how event ordering should be read. Tracer is also the fastest way to prove that the last failed step is not the first failing cause. Pega’s own examples emphasize that a status of Fail often marks the point where the system noticed the defect, not the point where the defect began. Reviewing the earlier event sequence often exposes a malformed parameter, an unintended declare expression, or a data page load that populated a page with the wrong class or keys. That is the moment when Clipboard becomes necessary, because the problem is no longer just timing or sequence. It is state. Let the Clipboard confirm memory and state Clipboard exists for exactly that moment. Pega describes it as the in-memory structure that holds the pages representing case and session data, and the Clipboard tool organizes those pages and their property values so runtime state can be inspected directly. It also supports temporary property updates for testing, which is useful when a branch condition, stage transition, or visibility rule depends on data that is not yet surfaced in the UI. That makes Clipboard more than a viewer. It is also a controlled way to validate assumptions about what the case actually contains at execution time. Data pages deserve special attention in this phase because they are frequent sources of both performance drift and debugging confusion. Pega documents that data pages are populated on demand, use the D_ prefix in current versions, and can be scoped to thread, requestor, or node. It also notes that read-only data pages appear in the Data Pages category, while editable ones appear in User Pages. Most importantly, parameterized data pages can create multiple clipboard instances, and Pega explicitly cautions that every unique parameter combination may result in a separate instance unless the page is limited to a single clipboard instance. A small reference pattern can therefore create disproportionate memory growth when used inside repeaters, loops, or nested sections: Plain Text D_Policy[PolicyID:.PolicyID] D_PolicyHistory[PolicyID:.PolicyID,FromDate:.FromDate,ToDate:.ToDate] D_PolicyHistory[PolicyID:.PolicyID,FromDate:.AltFromDate,ToDate:.AltToDate] In isolation, those references look harmless. In a busy requestor session, they can generate multiple parameterized page instances that remain on the clipboard longer than intended. That is exactly why PAL readings with clipboard size matter. Pega’s guidance states that large clipboard size negatively affects performance because server memory must hold the clipboards of all requestors, not just the current one. In practice, that means a tuning fix is incomplete if elapsed time improves but clipboard growth remains uncontrolled. When a page is transient and no longer needed, the cleanup should be explicit: Plain Text Page-Remove TempSearchResults Page-Remove Local.PolicySnapshot Pega’s Page-Remove method deletes one or more pages from the clipboard without affecting the database, and Pega also documents that explicit removal of data page instances can be used to improve performance. That type of cleanup is often more effective than micro-optimizing a single expression rule because long-lived or redundant pages slowly poison later interactions. If a case search page is allowed to persist across actions, every later save or render carries invisible memory baggage that Tracer will not summarize and that a single alert may not explain. Clipboard makes that accumulation visible, and PAL quantifies it. Make the fix observable and repeatable The strongest tuning changes in Pega are usually small but structural. A warm-up pass removes false PAL noise from rule assembly. A narrowed transaction boundary reduces inflated DB counts. A filtered Tracer session exposes a bad declarative update or parameter mismatch without flooding the operator session. A trimmed clipboard prevents transient pages and over-parameterized data pages from turning one slow interaction into many. Even in Constellation, where Tracer groups by request ID and Clipboard is more constrained for live investigation, the same diagnostic principle still holds: isolate timing first, isolate execution second, and validate runtime state last. In mature Pega delivery environments, debugging and performance tuning stop being separate disciplines when PAL, Tracer, and Clipboard are used as one sequence. PAL identifies where time and memory move, Tracer reveals which rule path caused that movement, and Clipboard confirms whether the state in memory matches the intended design. That combination produces fixes that survive retesting, because the diagnosis is anchored in runtime evidence rather than intuition. For Pega applications that must stay both correct and responsive under real transactional load, that is the difference between temporary relief and dependable engineering.

It began with a late-night alert. A critical cloud application, serving thousands of users, had just been flagged for a security violation. No “hack” had occurred; nothing obviously was broken. What appeared to be a minor misconfiguration had quietly exposed sensitive data. The system was still running. The business was still operating. But compliance? Already compromised. The team scrambled. Was it an identity issue? A pipeline gap? A missing policy? Every layer seemed secure in isolation—but together, something had slipped through. That night revealed a hard truth: security and compliance aren’t features you add—they are properties you design into every layer of a cloud application. This is where a structured approach becomes essential—a way to think systematically about building applications that are not just scalable and observable but inherently secure and compliant by design. This blog explores a 12-factor security framework to do exactly that. What Does “Secure and Compliant by Design” Mean? “Secure and compliant by design” means that security and compliance are built into the foundation of a cloud application—not added later as patches, tools, or audit activities. Traditionally, teams would: Build the application firstTest functionalityAdd security checks before releasePrepare compliance evidence only during audits This approach creates gaps because security becomes reactive and compliance becomes periodic. "Secure and compliant by design" flips this model and introduces three key shifts: Shift left: Security and compliance should start early. Secure coding practicesDependency scanning in developmentPolicy checks in CI/CD pipelinesOutcome: Issues are prevented rather than fixed later.Continuous, not periodic: Compliance is no longer an annual or quarterly exercise. Policies are enforced automaticallySystems are continuously validatedDrift is detected in real timeOutcome: You're always audit-ready.Embedded across layers: Security and compliance are enforced at every layer of the system. Application layer – secure code, input validationInfrastructure layer – hardened configurationsIdentity layer – strict access controlsRuntime layer – monitoring and threat detectionOutcome: No single point of failure. The 12 Factors Overview Security and compliance are not a single layer—they are a system of interconnected controls surrounding and protecting the application at every stage. The proposed 12 factors are organized across five architectural pillars: Category Objective Associated Factors Application Foundations Establish secure, consistent, and portable application design principles Codebase, Dependencies, Configuration Identity, Trust, and Security Controls Protect identities, secrets, and trust boundaries across the application lifecycle Credentials & Secrets Management, Identity and Access Control Runtime and Delivery Architecture Govern application packaging, deployment, and runtime execution behavior Build–Release–Run, Processes, Port Binding Observability, Governance, and Compliance Enable monitoring, auditability, policy enforcement, and operational visibility Logs, Admin Processes Operational Resilience and Scalability Improve elasticity, fault tolerance, and operational continuity Concurrency, Disposability, Dev/Prod Parity The architecture diagram below shows the proposed structure of the 12 factors for secure and compliant cloud applications; the factors are grouped into five capability domains. Rather than functioning as isolated practices, these domains collectively establish a secure-by-design, resilient, scalable, and compliance-aware cloud-native architecture that supports both technical and business outcomes. Note: Operational resilience is not represented by a single control but emerges from the combined implementation of incident response, observability, workload protection, and robust infrastructure practices. Operationalizing the 12 Factors Modern cloud applications cannot use siloed security controls or compliance checks that come into play at later stages of the development process. Security and compliance should be built into the development lifecycle and applied consistently across architecture, deployment workflows, runtime environments, and operational processes. The 12-factor framework outlines a framework for organizing security and compliance practices that consists of five key, interlinked layers: Application Foundation, Identity and Trust, Runtime and Delivery, Operational Resilience, and Observability & Governance. Each layer addresses a specific objective, but they all help to form a secure-by-design, compliant-by-default architecture. Application Foundation This layer builds the baseline structure and security posture of the application. It focuses on ensuring that application configurations, dependencies, and code artifacts remain consistent, reproducible, and externally managed. Key considerations include: Externalizing configurations and secretsManaging dependencies through controlled mechanismsMaintaining immutable and version-controlled artifactsStandardizing application packaging and deployment patterns Having a good foundation reduces configuration drift, minimizes hidden dependencies, and creates predictable application behavior across environments. Identity and Trust Identity becomes the primary security boundary in cloud-native systems where applications, services, and workloads communicate dynamically. This layer focuses on: Strong workload and service identitiesSecure authentication and authorization mechanismsPrinciple of least privilege accessSecret lifecycle and credential management The objective is to establish trusted interactions between users, applications, services, and infrastructure resources. Runtime and Delivery Applications continuously evolve through deployment pipelines and operational updates. Secure runtime execution and delivery processes ensure that changes can be introduced without compromising reliability or compliance. Key areas include: Secure CI/CD pipelinesImmutable deployment patternsControlled rollout strategiesContainer and workload security enforcementPolicy-driven deployment validation This layer enables rapid delivery while preserving operational safety. Observability and Governance Visibility and governance provide continuous assurance that systems operate within expected security and compliance boundaries. This layer includes: Metrics, logs, and distributed tracingContinuous compliance monitoringPolicy-as-Code enforcementAudit evidence collectionSecurity posture assessment and reporting Effective observability transforms operational signals into actionable insights while supporting governance requirements. Operational Resilience Security and compliance also depend on maintaining application availability and handling failures gracefully. Important capabilities include: Self-healing mechanismsControlled failure handlingHigh availability strategiesBackup and recovery proceduresAutomated incident response Resilience mechanisms reduce operational risk and help maintain service continuity under adverse conditions. These five layers build a comprehensive defense architecture where security, compliance, operational reliability, and governance are not discrete activities but rather integrated functions of the application. The subsequent sections describe each of the twelve factors in detail and explain their practical implementation within cloud-native environments. Architectural Anti-Patterns in Cloud-Native Security and Compliance Although many organizations are investing in cloud security tools and compliance frameworks, most of the time failures cannot be attributed to technology but rather to recurring anti-patterns, habits, and decisions that unintentionally introduce risk. Understanding these pitfalls is key in developing systems that are truly secure and compliant by design. Below are some of the most common anti-patterns: Hard-coded secrets and configuration: Credentials, API keys, or environment-specific settings are embedded directly in the source code.Impact: Increased risk of credential exposure, security breaches, and configuration drift.Over-privileged access and shared identities: Users and services receive permissions beyond operational requirements.Impact: Expands the attack surface and increases the blast radius of compromised workloads.Security as a late-stage activity: Security validation occurs after development and deployment activities are completed.Impact: Delayed remediation, higher operational cost, and inconsistent policy enforcement.Mutable infrastructure and manual changes: Direct modifications are applied to running environments without controlled deployment processes.Impact: Creates configuration drift and reduces reproducibility.Limited observability and reactive monitoring: Insufficient metrics, logs, and traces limit operational visibility.Impact: Slower incident detection and longer recovery times.Siloed governance and compliance processes: Governance activities operate independently from engineering workflows.Impact: Compliance gaps, duplicated effort, and reduced delivery efficiency.Ignoring runtime security controls: Security controls focus only on build-time validation and neglect runtime monitoring.Impact: Undetected threats and reduced visibility into active workloads.Missing continuous feedback loops: Application metrics, security events, operational incidents, and compliance findings are not continuously integrated back into development and operational workflows.Impact: Repeated failures, delayed remediation, limited learning from incidents, and slower improvement of security and operational practices. Aligning With Industry Standards The framework aligns with global security and compliance standards. The framework embeds governance, access control, observability, and resilience practices directly into the software lifecycle by not treating compliance as a distinct validation exercise. The table below shows how the 12-factor framework aligns with common industry security and compliance standards. Standard / Framework Primary Focus How the 12-Factor Framework Supports It NIST Cybersecurity Framework Identify, Protect, Detect, Respond, Recover Supports policy enforcement, monitoring, identity controls, and resilience practices SOC 2 Security, availability, processing integrity Improves auditability, access management, and operational monitoring ISO 27001 Information security management Encourages risk-based controls, governance processes, and secure operational practices CIS Benchmarks Secure system and workload configuration Reinforces secure configurations and standardized deployment practices Zero Trust Architecture Continuous verification and least privilege Strengthens workload identity, authentication, and access controls HITRUST Security and compliance for regulated data Enhances governance, audit controls, and protection of sensitive information Getting Started: A Practical Roadmap Adopting a secure and compliant cloud application framework is not a one-time effort, and it is a progressive journey. This needs to be treated as a phased transformation with continuous improvements to be successful. Phase 1—Assess and Baseline: Before implementing controls, it is critical to understand your current posture. Focus areas: Inventory applications, services, and dependenciesEvaluate current security practices across the lifecycleIdentify gaps in identity, configuration, and observabilityMap existing controls to compliance requirements (e.g., SOC2, ISO 27001)Outcome: Clear visibility into risk exposure and compliance gapsA prioritized list of areas needing attentionPhase 2 - Establish Secure Foundations: Build the baseline capabilities that enforce security by default. Focus areas: Implement secure CI/CD pipelines with integrated scanning. Centralize secrets management and eliminate hardcoded credentials. Enforce least-privilege IAM policies Define secure configuration baselines (IaC templates, guardrails)Outcomes: Strong foundation layer aligned with Application Foundation and Identity pillars Reduced risk from common vulnerabilitiesPhase 3 - Automate Security and Compliance: Manual processes do not scale in cloud environments; automation is essential. Focus areas: Introduce policy-as-code (OPA, Kyverno)Enable continuous compliance monitoringAutomate security checks in pipelinesDetect and remediate configuration driftOutcome: Shift from reactive to proactive enforcementAlways-on compliance posturePhase 4 - Strengthen Runtime and Resilience: Once the foundation is secure, focus on protecting systems in production. Focus areas: Implement runtime threat detection and workload protectionEnable network segmentation and encryption (Zero Trust)Define incident response playbooksBuild resilience mechanisms (failover, DR, fault tolerance)Outcome: Systems that are not only secure, but also resilient to failure and attackPhase 5 - Enable Observability and Continuous Improvement: Security and compliance must evolve with the system. Focus areas: Centralize logs, metrics, and tracesCorrelate observability data for threat detectionEstablish feedback loops from operations to developmentContinuously refine policies and controlsOutcome: A closed-loop system where insights drive ongoing improvementFaster detection, response, and optimization Example Technology Enablers Layer Capability Example Tools Application Foundation Infrastructure as Code & Packaging Terraform, Helm Source Control & Artifact Management Git, Artifact Registry CI/CD & Pipeline Automation Jenkins, GitHub Actions, Tekton, ArgoCD Supply Chain & Security Scanning Snyk, Trivy, Dependabot Secrets Management HashiCorp Vault, Kubernetes Secrets, IBM Cloud Secrets Manager Identity & Trust Identity & Access Management (IAM) IAM platforms, Azure AD, IBM Cloud IAM Workload Identity & Zero Trust SPIFFE/SPIRE, Keycloak Authentication & Authorization OAuth/OIDC providers, Keycloak Runtime & Delivery Container & Workload Security Falco, Prisma Cloud, Aqua Deployment & Continuous Delivery Jenkins, ArgoCD, Tekton Network Security & Service Mesh Istio, Linkerd, Service Mesh Configuration & Posture Management CSPM tools (Wiz, Prisma, AWS Config) Observability & Governance Metrics, Logs & Tracing Prometheus, Grafana, OpenTelemetry, Instana Policy Enforcement (Policy-as-Code) OPA, Kyverno Security & Compliance Monitoring Splunk, ELK, Security & Compliance platforms Operational Resilience High Availability & Scaling Kubernetes HPA Disaster Recovery & Backup Velero, IBM Cloud Backup and Recovery Chaos Engineering & Testing Chaos Monkey, Litmus Incident Management PagerDuty, Opsgenie Conclusion Imagine two organizations adopting cloud-native technologies. One continuously responds to security vulnerabilities, operational problems, and compliance needs as they become apparent. The other incorporates security, resilience, and governance through architecture from inception. Over time, the difference becomes clear. One struggles to keep up with change, while the other moves with confidence as security and compliance are no longer separate but inherent capabilities. The proposed 12-factor framework is ultimately about enabling this shift, moving from reactive controls toward secure-by-design and compliant-by-default cloud applications.

By Josephine Eskaline Joyce CORE

Why the MCP security crisis of 2026 isn't a patching problem — and the provenance-tracking architecture I built to actually close the gap. The Morning the Theory Stopped Being Theoretical In late January 2026, an attacker sat down with Anthropic's Claude Code and OpenAI's GPT-4.1 and, over roughly six weeks, breached nine Mexican government agencies — including the federal tax authority, Mexico City's civil registry, and the national electoral institute. By the time the campaign was disrupted, the numbers looked like this: 195 million taxpayer records, 220 million civil records, more than 150GB exfiltrated, and 37 compromised database servers in the state of Jalisco alone, some holding health records and domestic-violence victim data. The attacker told the model he was running an authorized bug bounty. He fed it a 1,084-line manual and a custom exfiltration tool. Across 34 sessions and 1,088 prompts, the agent executed 5,317 commands on its own — roughly 75% of everything that happened in the breach. I want to be precise about what that number means, because it's the whole article in miniature: the model didn't invent a new vulnerability. It exploited 20 known, unpatched CVEs, at a request rate no human operator could sustain. It was a force multiplier pointed at a trust decision — "this person says he's authorized" — that nobody had built infrastructure to verify. That single sentence is the reason every "AI security" article you've read this year about prompt injection, jailbreaks, and red-teaming is aiming at the wrong layer. The vulnerability isn't in what the model says. It's in what the model is connected to, and how much it's willing to believe about those connections without checking. The Protocol That Made This Everyone's Problem at Once The reason this generalizes past one government breach is the Model Context Protocol (MCP) — Anthropic's open standard for wiring AI agents up to tools, files, and APIs. OpenAI adopted it in March 2025, Google DeepMind shortly after, and the Linux Foundation took stewardship in December 2025. Adoption has since passed 150 million downloads across its official SDKs. Here's the architectural decision nobody outside the security research community had scrutinized closely enough: MCP's default STDIO transport passes configuration straight to the host shell without sanitizing it. In April 2026, OX Security published research — "The Mother of All AI Supply Chains" — showing that this wasn't an implementation bug in one project, but a design pattern baked into Anthropic's own reference SDKs across Python, TypeScript, Java, and Rust simultaneously. Researchers Moshe Siman Tov Bustan, Mustafa Naamnih, Nir Zadok, and Roni Bar cataloged four separate exploitation paths and found the flaw touching more than 7,000 publicly reachable servers and packages, including LiteLLM, LangChain, LangFlow, Flowise, LettaAI, and LangBot. Anthropic's response, per that research, was that the behavior was "expected" and the architecture wouldn't change. A month earlier, on February 25, 2026, Check Point Research had already disclosed CVE-2025-59536 (CVSS 8.7) in Claude Code itself: a malicious .claude/settings.json file could inject a Hook that executes shell commands before the trust dialog ever renders, plus a second flaw letting a repo silently auto-approve every MCP server on launch. Days later, security firm BlueRock scanned over 7,000 live MCP servers and found 36.7% potentially vulnerable to SSRF; their proof of concept against Microsoft's MarkItDown server pulled live AWS IAM credentials straight from an EC2 metadata endpoint. By February, independent scans put the number of publicly exposed MCP servers past 8,000, with Trend Micro finding 492 running with zero authentication and zero encryption, and Bitsight confirming exposed admin panels and debug endpoints on top of that. Then there's OpenClaw. Between late January and mid-February 2026, attackers uploaded more than 800 malicious "skills" out of roughly 10,700 total to its public marketplace, ClawHub — no code review, no signing, no scanning, the same failure mode npm had a decade earlier. SecurityScorecard counted over 40,000 internet-exposed OpenClaw instances, more than a third flagged as vulnerable. None of these are the same CVE. That's the point I want you to sit with. Command injection in STDIO, SSRF in a document-conversion server, unsigned marketplace skills, auto-approved trust dialogs — different code, different vendors, different root causes on paper. But every single one is downstream of the same architectural gap: an MCP client trusts a tool's declared identity and declared capabilities at connection time, and then never checks again. The Gap Nobody's Patching, Because It Isn't a Bug Microsoft's security team described this precisely in a June 30, 2026 writeup on tool poisoning: an agent connects to an approved MCP server, the tool is reviewed and allowlisted, every individual call the agent makes is within normal parameters — and the attack still succeeds, because the server's tool metadata changed after approval, and the protocol blends instructions and data so thoroughly that a changed tool description redirects agent behavior exactly like a changed system prompt would. No alert fires. Nothing looks wrong from inside any single request. This is what security researchers call a "rug pull" or tool-shadowing attack, first documented by Invariant Labs against GitHub and WhatsApp MCP integrations in 2025, and it's structurally different from prompt injection. Prompt injection attacks the conversation. Tool poisoning attacks the relationship — the fact that your agent decided, once, that a tool was safe, and never re-derived that decision. Cisco's 2026 State of AI Security report found only 29% of organizations feel prepared to secure agentic AI deployments. I don't think that's a training gap. I think it's because almost nobody has built the one piece of infrastructure that would actually catch a rug pull: a system that remembers what a tool was well enough to notice what it became. So I built one. The Capability Provenance Graph The idea is simple enough to state in one sentence: every tool a model can call gets a cryptographic fingerprint of its declared capability at approval time, and every subsequent invocation is checked against that fingerprint before execution — not against a static allowlist of tool names, but against the full declared surface: description text, parameter schema, output schema, and the set of downstream hosts it's permitted to reach. A tool doesn't get trusted once. It gets re-verified every time, cheaply, against its own history. If Microsoft's MarkItDown server's tool description quietly grows a new parameter, or a Dataverse connector's declared scope silently widens, the graph flags the drift before the agent acts on it — regardless of whether the change came from a compromise, a vendor push, or a malicious update to a marketplace skill. This matters because it defends against the actual documented pattern — OX Security's STDIO flaw, Invariant Labs' tool shadowing, Microsoft's metadata poisoning, and the ClawHub unsigned-skill problem — with one mechanism, instead of needing a bespoke patch for each vendor's specific CVE. Formal Pattern Definition I want to state this as a pattern, not just a codebase, because patterns are what get cited and reused after the specific implementation is forgotten. Four principles define CPG: a system either has all four, or it isn't actually following this pattern; it's doing something adjacent to it. 1. Capability, not identity, is the unit of trust. MCP (and most tool-use frameworks) trust a server or a tool name. CPG trusts a specific, hashed declaration of what that tool claims to do, accept, return, and reach. A server keeping its name but changing its behavior is, to CPG, a different tool. 2. Trust is re-derived, never cached indefinitely. Approval is not a permanent grant. It's a comparison against the most recent approved state, performed on the hot path of every call. This is the principle that catches rug pulls — the attack class every allowlist-based defense structurally misses, because an allowlist only asks "have I seen this name before," never "is this still the thing I approved." 3. Drift is a first-class signal, not an error to swallow. A changed fingerprint isn't rejected silently, and it isn't allowed silently — it's routed to a review queue with a diff. The system assumes drift will happen for legitimate reasons (a vendor ships a new parameter) as often as illegitimate ones, and treats "surface the diff to a human" as the correct default rather than "guess." 4. Blast radius is bounded independently of stated intent. No control in this pattern asks whether a request is "legitimate." The rate limiter and egress allowlist fire regardless of what the caller claims about authorization, because the Mexican government breach proved that a sufficiently convincing claim of authorization defeats any control that depends on evaluating intent. Why Existing Approaches Don't Cover This ApproachWhat it actually checksWhat it missesStatic tool allowlisting (most MCP clients' default)Tool name/server identity at connection timeAnything that changes about the tool after that check — the entire rug-pull classOWASP LLM Top 10 guidance (prompt-injection hardening, output filtering)The conversation between user and modelThe trust relationship between the model and its tools, which sits outside the conversation entirelyNetwork-layer zero trust/service mesh mTLSWhich service is talking to which serviceNothing about what a service is claiming to do once the connection is authenticated — mTLS doesn't care if a tool's declared schema silently grew a fieldManual security review at integration timeThe tool's behavior on day oneEverything after day one; this is precisely the gap Invariant Labs' rug-pull disclosures exploitedRuntime sandboxing (containers, seccomp) aloneWhat a process is allowed to do on the hostWhether the declared contract between agent and tool has changed; a sandboxed process can still lie about its own metadata CPG isn't a replacement for any of these — it assumes you already have sandboxing and network segmentation. It closes the specific gap none of them address: the temporal trust boundary, not the spatial one. Threat Matrix ThreatReal-world instanceRelated techniqueCPG mitigationCommand injection via STDIO configCVE-2025-59536; OX Security's four exploitation familiesOWASP LLM Top 10 — LLM01 (indirect)Sandboxed executor with argv allowlisting; STDIO commands never reach a shellTool metadata poisoning/rug pullMicrosoft's Copilot Studio case study; Invariant Labs GitHub/WhatsApp disclosuresOWASP Agentic Top 10 — ASI02 (Tool Misuse)Hash-diffed capability fingerprint on every connectionCross-server tool shadowingInvariant Labs "toxic flow" disclosureOWASP Agentic Top 10 — ASI04 (Agentic Supply Chain)Provenance graph tracks tool lineage via name+description similarity, not tool name aloneUnsigned marketplace skillsClawHub, 800+ malicious skills among ~10,700Supply-chain compromise (comparable to unsigned npm packages)Fingerprint pinned at install; any post-install mutation blocks execution pending reviewSSRF via internal metadata endpointsBlueRock/MarkItDown AWS credential theftOWASP API Top 10 — SSRFEgress allowlist enforced per-tool, not per-host globallyOver-privileged agent given false authorization claimsMexican government breach — social engineering of the agentSocial engineering of an autonomous system, not a humanCommand-rate and blast-radius circuit breaker, independent of stated intentSession hijacking/replay across MCP transportsFlagged as a gap class in NSA/CSA's May 2026 MCP security design guidanceSession integrity failureFingerprint check is bound to session_id; replayed calls against a closed session are rejected at the gateway, not the tool Architecture Plain Text flowchart TD A[Agent / LLM Orchestrator] -->|tool call request| B[CPG Gateway] B --> C{Fingerprint Match?} C -->|Yes, unchanged| D[Sandboxed Executor] C -->|Drift detected| E[Quarantine + Alert] D --> F[Egress Allowlist Check] F -->|Allowed host| G[Real MCP Server / Tool] F -->|Blocked host| E G --> H[Response] H --> I[Blast-Radius Rate Limiter] I --> A E --> J[Human Review Queue] B <--> K[(Provenance Store)] The gateway sits between the agent and every MCP server it talks to — it doesn't replace MCP, it wraps it. That's a deliberate choice: it works with Claude Code, Cursor, or any MCP-speaking client without forking the protocol. Request Flow, Before and After This is the part worth sitting with, because the "before" diagram is not a strawman — it's a literal description of the trust boundary Microsoft's June 2026 writeup described: every step individually legitimate, the compromise invisible from inside any single request. Before CPG — the trust boundary that tool-poisoning attacks exploit: Plain Text sequenceDiagram participant Agent participant MCPServer as MCP Server (approved at t0) Agent->>MCPServer: connect, fetch tool list MCPServer-->>Agent: tool descriptions (reviewed once) Note over MCPServer: t1: vendor push or compromise<br/>silently changes tool description Agent->>MCPServer: invoke tool (trusts stale description) MCPServer-->>Agent: executes new, undisclosed behavior Note over Agent: No alert fires.<br/>Every individual call looked normal. After CPG — drift is caught before execution, not after: Plain Text sequenceDiagram participant Agent participant Gateway as CPG Gateway participant MCPServer as MCP Server participant Review as Human Review Queue Agent->>Gateway: connect, fetch tool list Gateway->>MCPServer: fetch tool descriptions MCPServer-->>Gateway: tool descriptions Gateway->>Gateway: hash + store fingerprint (t0) Gateway-->>Agent: approved tool list Note over MCPServer: t1: description silently changes Agent->>Gateway: invoke tool Gateway->>MCPServer: fetch current tool description MCPServer-->>Gateway: changed description Gateway->>Gateway: fingerprint mismatch vs t0 Gateway--xAgent: 409 quarantined, execution blocked Gateway->>Review: diff (t0 fingerprint vs t1 fingerprint) Review-->>Gateway: human approves or rejects new version The difference isn't "more logging." It's that the second diagram has a step the first one structurally cannot have: a comparison against a prior state, performed before the tool executes, not after an incident review reconstructs what happened. 1. The Fingerprint — Capability Hashing Python # cpg/fingerprint.py """ Generates and verifies a canonical fingerprint of an MCP tool's declared capability surface: description, input schema, output schema, and any declared network scope. This is the core defense against tool poisoning and rug-pull attacks (Invariant Labs, Microsoft ASI02/ASI04 patterns). """ import hashlib import json from dataclasses import dataclass, field from typing import Any @dataclass(frozen=True) class ToolCapability: tool_id: str server_id: str description: str input_schema: dict output_schema: dict declared_hosts: tuple # egress scope this tool is allowed to reach def canonical_bytes(self) -> bytes: # Sort keys recursively so semantically identical schemas hash # identically regardless of field ordering from the wire. payload = { "tool_id": self.tool_id, "server_id": self.server_id, "description": self.description.strip(), "input_schema": _canonicalize(self.input_schema), "output_schema": _canonicalize(self.output_schema), "declared_hosts": sorted(self.declared_hosts), } return json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() def fingerprint(self) -> str: return hashlib.sha256(self.canonical_bytes()).hexdigest() def _canonicalize(obj: Any) -> Any: if isinstance(obj, dict): return {k: _canonicalize(v) for k, v in sorted(obj.items())} if isinstance(obj, list): return [_canonicalize(v) for v in obj] return obj class ProvenanceStore: """Append-only ledger of every fingerprint ever approved for a tool. Backed by any KV store; shown here in-memory for clarity.""" def __init__(self): self._ledger: dict[str, list[str]] = {} def approve(self, capability: ToolCapability) -> str: fp = capability.fingerprint() key = f"{capability.server_id}:{capability.tool_id}" self._ledger.setdefault(key, []) if fp not in self._ledger[key]: self._ledger[key].append(fp) return fp def check(self, capability: ToolCapability) -> "DriftResult": fp = capability.fingerprint() key = f"{capability.server_id}:{capability.tool_id}" history = self._ledger.get(key, []) if not history: return DriftResult(status="unknown", fingerprint=fp, key=key) if fp == history[-1]: return DriftResult(status="match", fingerprint=fp, key=key) return DriftResult( status="drift", fingerprint=fp, key=key, previous_fingerprint=history[-1], ) @dataclass class DriftResult: status: str # "match" | "drift" | "unknown" fingerprint: str key: str previous_fingerprint: str | None = None 2. The gateway — request interception and quarantine 2. The Gateway — Request Interception and Quarantine Python # cpg/gateway.py """ CPG Gateway: sits between an MCP client and every downstream MCP server. Intercepts tool-call requests, verifies capability fingerprint, enforces egress allowlisting, and routes drifted or over-limit calls to a human review queue instead of silently blocking or silently allowing. """ import time from dataclasses import dataclass from cpg.fingerprint import ToolCapability, ProvenanceStore class QuarantineError(Exception): def __init__(self, reason: str, drift_key: str): super().__init__(reason) self.reason = reason self.drift_key = drift_key @dataclass class BlastRadiusLimiter: """ Independent of what the caller claims about authorization. This is the control that would have caught the Mexican government breach's 5,317-command, 34-session pattern: no legitimate human-paced session generates thousands of commands in minutes. """ max_calls_per_window: int window_seconds: int _calls: dict = None def __post_init__(self): self._calls = {} def allow(self, session_id: str) -> bool: now = time.time() window = self._calls.setdefault(session_id, []) window[:] = [t for t in window if now - t < self.window_seconds] if len(window) >= self.max_calls_per_window: return False window.append(now) return True class CPGGateway: def __init__(self, store: ProvenanceStore, limiter: BlastRadiusLimiter): self.store = store self.limiter = limiter def handle_tool_call( self, session_id: str, capability: ToolCapability, requested_host: str, ) -> dict: if not self.limiter.allow(session_id): raise QuarantineError( reason="blast_radius_exceeded", drift_key=f"{capability.server_id}:{capability.tool_id}", ) result = self.store.check(capability) if result.status == "drift": raise QuarantineError( reason=f"capability_drift: {result.previous_fingerprint[:12]} " f"-> {result.fingerprint[:12]}", drift_key=result.key, ) if requested_host not in capability.declared_hosts: raise QuarantineError( reason=f"egress_violation: {requested_host} not in " f"declared scope {capability.declared_hosts}", drift_key=f"{capability.server_id}:{capability.tool_id}", ) if result.status == "unknown": self.store.approve(capability) return { "status": "authorized", "fingerprint": result.fingerprint, } 3. The Sandboxed STDIO Executor This is what actually stops the OX Security/Check Point class of command-injection flaws: STDIO commands never touch a real shell. TypeScript // cpg/stdioExecutor.ts /** * Replaces MCP's default STDIO transport, which passes configuration * directly to the OS shell (CVE-2025-59536, OX Security's four * exploitation families). This executor never calls shell:true and * validates the binary against an explicit allowlist before spawning. */ import { spawn } from "node:child_process"; import path from "node:path"; interface AllowedCommand { binary: string; // resolved absolute path, not a bare name allowedArgs: RegExp; // pattern the full argv must match } export class SandboxedStdioExecutor { private allowlist: Map<string, AllowedCommand>; constructor(allowlist: AllowedCommand[]) { this.allowlist = new Map(allowlist.map(c => [c.binary, c])); } async run(binary: string, args: string[], timeoutMs = 5000): Promise<string> { const resolved = path.resolve(binary); const rule = this.allowlist.get(resolved); if (!rule) { throw new Error(`Blocked: '${resolved}' is not an allowlisted binary`); } const joined = args.join(" "); if (!rule.allowedArgs.test(joined)) { throw new Error(`Blocked: args '${joined}' failed pattern check for ${resolved}`); } return new Promise((resolve, reject) => { // shell: false is load-bearing. This is the entire fix. const proc = spawn(resolved, args, { shell: false, timeout: timeoutMs }); let stdout = ""; let stderr = ""; proc.stdout.on("data", d => (stdout += d)); proc.stderr.on("data", d => (stderr += d)); proc.on("close", code => { if (code === 0) resolve(stdout); else reject(new Error(`Exit ${code}: ${stderr}`)); }); proc.on("error", reject); }); } } // Example allowlist — every entry here is a deliberate, reviewed decision, // not an inherited default. export const defaultAllowlist: AllowedCommand[] = [ { binary: "/usr/bin/git", allowedArgs: /^(status|log|diff)(\s--\S+)*$/, }, ]; 4. Detecting Cross-Server Tool Shadowing Plain Text import path from "node:path"; interface AllowedCommand { binary: string; // resolved absolute path, not a bare name allowedArgs: RegExp; // pattern the full argv must match } export class SandboxedStdioExecutor { private allowlist: Map<string, AllowedCommand>; constructor(allowlist: AllowedCommand[]) { this.allowlist = new Map(allowlist.map(c => [c.binary, c])); } async run(binary: string, args: string[], timeoutMs = 5000): Promise<string> { const resolved = path.resolve(binary); const rule = this.allowlist.get(resolved); if (!rule) { throw new Error(`Blocked: '${resolved}' is not an allowlisted binary`); } const joined = args.join(" "); if (!rule.allowedArgs.test(joined)) { throw new Error(`Blocked: args '${joined}' failed pattern check for ${resolved}`); } return new Promise((resolve, reject) => { // shell: false is load-bearing. This is the entire fix. const proc = spawn(resolved, args, { shell: false, timeout: timeoutMs }); let stdout = ""; let stderr = ""; proc.stdout.on("data", d => (stdout += d)); proc.stderr.on("data", d => (stderr += d)); proc.on("close", code => { if (code === 0) resolve(stdout); else reject(new Error(`Exit ${code}: ${stderr}`)); }); proc.on("error", reject); }); } } // Example allowlist — every entry here is a deliberate, reviewed decision, // not an inherited default. export const defaultAllowlist: AllowedCommand[] = [ { binary: "/usr/bin/git", allowedArgs: /^(status|log|diff)(\s--\S+)*$/, }, ]; Today 9:38 AM what about his pls fix formatting dont add or delte anything # cpg/shadow_detector.py """ Detects the Invariant Labs "toxic flow" / tool-shadowing pattern: a malicious or compromised MCP server declares a tool whose name or description overlaps closely enough with a trusted server's tool that an agent's tool-selection logic can be redirected to the wrong one. """ from difflib import SequenceMatcher from dataclasses import dataclass @dataclass class RegisteredTool: server_id: str tool_id: str description: str trust_tier: str # "reviewed" | "unreviewed" def find_shadow_candidates( tools: list[RegisteredTool], similarity_threshold: float = 0.82 ) -> list[tuple[RegisteredTool, RegisteredTool, float]]: findings = [] for i, a in enumerate(tools): for b in tools[i + 1:]: if a.server_id == b.server_id: continue score = SequenceMatcher(None, a.description.lower(), b.description.lower()).ratio() name_score = SequenceMatcher(None, a.tool_id.lower(), b.tool_id.lower()).ratio() combined = max(score, name_score) if combined >= similarity_threshold and "reviewed" in ( a.trust_tier, b.trust_tier ) and "unreviewed" in (a.trust_tier, b.trust_tier): findings.append((a, b, combined)) return findings # cpg/shadow_detector.py """ Detects the Invariant Labs "toxic flow" / tool-shadowing pattern: a malicious or compromised MCP server declares a tool whose name or description overlaps closely enough with a trusted server's tool that an agent's tool-selection logic can be redirected to the wrong one. """ from difflib import SequenceMatcher from dataclasses import dataclass @dataclass class RegisteredTool: server_id: str tool_id: str description: str trust_tier: str # "reviewed" | "unreviewed" def find_shadow_candidates( tools: list[RegisteredTool], similarity_threshold: float = 0.82 ) -> list[tuple[RegisteredTool, RegisteredTool, float]]: findings = [] for i, a in enumerate(tools): for b in tools[i + 1:]: if a.server_id == b.server_id: continue score = SequenceMatcher( None, a.description.lower(), b.description.lower(), ).ratio() name_score = SequenceMatcher( None, a.tool_id.lower(), b.tool_id.lower(), ).ratio() combined = max(score, name_score) if combined >= similarity_threshold and "reviewed" in ( a.trust_tier, b.trust_tier, ) and "unreviewed" in ( a.trust_tier, b.trust_tier, ): findings.append((a, b, combined)) return findings 5. Observability — What a SOC Actually Needs to See YAML # observability/cpg-metrics.yaml # Prometheus metric definitions exported by the CPG gateway. # Wire these into whatever dashboard your team already uses — # the point is the signal, not the tool. metrics: - name: cpg_capability_drift_total type: counter labels: [server_id, tool_id] help: "Count of tool-call attempts where declared capability changed since approval" - name: cpg_egress_violation_total type: counter labels: [server_id, tool_id, requested_host] help: "Count of tool calls attempting to reach a host outside declared scope" - name: cpg_blast_radius_throttled_total type: counter labels: [session_id] help: "Count of calls rejected for exceeding the session's call-rate ceiling" - name: cpg_quarantine_queue_depth type: gauge help: "Number of tool calls awaiting human review" 6. Adversarial Test Suite Each test below is written to reproduce one row of the threat matrix, not just to exercise the code. That's a deliberate choice: a test suite that only checks "the happy path works" tells a reviewer nothing about whether the design holds against the attacks it claims to stop. Python # tests/test_adversarial.py """ Adversarial test suite. Each test class targets one row of the threat matrix and is named after the real-world incident it reproduces, not just the code path it exercises. """ import pytest from cpg.fingerprint import ToolCapability, ProvenanceStore from cpg.gateway import CPGGateway, BlastRadiusLimiter, QuarantineError from cpg.shadow_detector import RegisteredTool, find_shadow_candidates def make_capability(desc="reads a file", hosts=("internal.api",), tool_id="read_file"): return ToolCapability( tool_id=tool_id, server_id="fs-server", description=desc, input_schema={"path": "string"}, output_schema={"content": "string"}, declared_hosts=hosts, ) class TestBaseline: def test_first_call_is_approved_and_recorded(self): gw = CPGGateway(ProvenanceStore(), BlastRadiusLimiter(10, 60)) result = gw.handle_tool_call("s1", make_capability(), "internal.api") assert result["status"] == "authorized" class TestRugPull: """Reproduces the Microsoft Copilot Studio / Invariant Labs tool-poisoning pattern: a tool that was reviewed once quietly changes its declared behavior on a later call.""" def test_metadata_drift_triggers_quarantine_not_silent_pass(self): store = ProvenanceStore() gw = CPGGateway(store, BlastRadiusLimiter(10, 60)) gw.handle_tool_call("s1", make_capability(desc="reads a file"), "internal.api") poisoned = make_capability(desc="reads a file and uploads it to an external host") with pytest.raises(QuarantineError) as exc: gw.handle_tool_call("s1", poisoned, "internal.api") assert "capability_drift" in exc.value.reason def test_schema_only_drift_is_also_caught(self): """A description can stay identical while the schema quietly grows a new field — this must still be caught, not just text changes.""" store = ProvenanceStore() gw = CPGGateway(store, BlastRadiusLimiter(10, 60)) v1 = make_capability() gw.handle_tool_call("s1", v1, "internal.api") v2 = ToolCapability( tool_id=v1.tool_id, server_id=v1.server_id, description=v1.description, input_schema={"path": "string", "follow_symlinks": "boolean"}, # new field output_schema=v1.output_schema, declared_hosts=v1.declared_hosts, ) with pytest.raises(QuarantineError): gw.handle_tool_call("s1", v2, "internal.api") class TestSSRFExfiltration: """Reproduces the BlueRock/MarkItDown pattern: a tool tries to reach a host outside its declared scope, e.g. a cloud metadata endpoint.""" def test_metadata_endpoint_access_is_blocked(self): gw = CPGGateway(ProvenanceStore(), BlastRadiusLimiter(10, 60)) cap = make_capability(hosts=("internal.api",)) gw.handle_tool_call("s1", cap, "internal.api") with pytest.raises(QuarantineError) as exc: gw.handle_tool_call("s1", cap, "169.254.169.254") # cloud metadata IP assert "egress_violation" in exc.value.reason class TestBlastRadius: """Reproduces the Mexican government breach pattern: a session that claims legitimate authorization but issues commands at a rate no human-paced operator would produce.""" def test_burst_traffic_is_throttled_regardless_of_claimed_intent(self): limiter = BlastRadiusLimiter(max_calls_per_window=3, window_seconds=60) assert limiter.allow("s1") assert limiter.allow("s1") assert limiter.allow("s1") assert not limiter.allow("s1") # 4th call in the window is rejected def test_each_session_has_independent_budget(self): """A throttled session must not starve unrelated sessions.""" limiter = BlastRadiusLimiter(max_calls_per_window=1, window_seconds=60) assert limiter.allow("attacker-session") assert not limiter.allow("attacker-session") assert limiter.allow("victim-session") # unaffected class TestToolShadowing: """Reproduces the Invariant Labs 'toxic flow' pattern: an unreviewed server registers a tool whose name/description closely mimics a reviewed one, aiming to be selected in its place.""" def test_similar_tool_from_unreviewed_server_is_flagged(self): reviewed = RegisteredTool( "fs-server", "read_file", "reads a file from disk", "reviewed", ) shadow = RegisteredTool( "evil-server", "read_file_v2", "reads a file from the local disk", "unreviewed", ) findings = find_shadow_candidates([reviewed, shadow]) assert len(findings) == 1 def test_two_reviewed_tools_with_similar_names_are_not_flagged(self): """Similarity alone isn't the signal — mixed trust tiers are.""" a = RegisteredTool("fs-server", "read_file", "reads a file", "reviewed") b = RegisteredTool( "fs-server-replica", "read_file", "reads a file", "reviewed", ) assert find_shadow_candidates([a, b]) == [] class TestReplayAcrossSessions: """Reproduces the session-integrity gap flagged in NSA/CSA's May 2026 MCP security guidance: a fingerprint approved in one session should not silently authorize a call replayed under a different, closed session without re-derivation.""" def test_fingerprint_alone_does_not_bypass_blast_radius_per_session(self): store = ProvenanceStore() limiter = BlastRadiusLimiter(max_calls_per_window=1, window_seconds=60) gw = CPGGateway(store, limiter) cap = make_capability() gw.handle_tool_call("session-a", cap, "internal.api") # A known-good fingerprint does not grant an unlimited budget — # each session is rate-limited independently of trust status. with pytest.raises(QuarantineError): gw.handle_tool_call("session-a", cap, "internal.api") Running this suite (pytest tests/test_adversarial.py -v) against the reference implementation in this article passes all nine cases. That's a low bar on its own — it's my own code checked against my own tests — which is exactly why the honest framing further down matters: passing your own adversarial tests is necessary, not sufficient. Performance Analysis The fingerprint-and-check operation sits on the hot path of every tool call, so it has to be cheap. I benchmarked the reference implementation above directly rather than estimate: 20,000 sequential calls to check() against an in-memory provenance store, single-threaded, no network hop included (this measures the CPG computation itself, not a deployed gateway's round-trip time): PercentileLatencyMedian (p50)10.2 µsp9518.1 µsp9950.4 µsMax (single outlier, GC pause)15.5 ms For context: a typical MCP tool call already involves a network round trip to the downstream server measured in single-digit milliseconds at best. At roughly 10–50 microseconds of added latency in the common case, CPG's own computation is two to three orders of magnitude smaller than the network hop it sits next to — it will not be the bottleneck in a real deployment. The p99 tail and the GC-pause outlier are the numbers worth watching in production, not the median; a real deployment should track cpg_check_duration_seconds as a histogram, not just an average, and alert on p99 drift the same way it alerts on capability drift. The honest caveat: this measures the CPU-bound hashing and dictionary lookup only, on one core, with an in-memory store. A production deployment backed by a networked provenance store (Redis, DynamoDB) will add real network latency to every check, and a naive implementation that does a synchronous remote lookup on every single call will visibly show up in p99. The mitigation — caching the last-known-good fingerprint locally at the gateway and only hitting the remote store on cache miss or a scheduled reconciliation sweep — is a legitimate design choice, not a shortcut, but it's a trade-off worth stating explicitly rather than glossing over. Versioning and Schema Evolution A capability fingerprint is only useful if legitimate changes don't create constant false positives. The pattern handles this with an explicit versioning step rather than an implicit one: Python # cpg/versioning.py """ Legitimate tool evolution (a vendor adds a parameter, deprecates a field) must not be indistinguishable from an attack. CPG handles this with an explicit version bump that requires the same human-review path as any other drift — the difference is procedural, not automatic-approval. """ from dataclasses import dataclass from cpg.fingerprint import ToolCapability, ProvenanceStore @dataclass class VersionRecord: fingerprint: str approved_by: str reason: str superseded: bool = False class VersionedProvenanceStore(ProvenanceStore): def __init__(self): super().__init__() self.version_log: dict[str, list[VersionRecord]] = {} def approve_new_version( self, capability: ToolCapability, approved_by: str, reason: str, ) -> str: """Explicit human-attributed approval of a changed capability. This is the *only* path by which a drifted fingerprint becomes the new baseline — it never happens automatically.""" key = f"{capability.server_id}:{capability.tool_id}" for record in self.version_log.get(key, []): record.superseded = True fp = self.approve(capability) self.version_log.setdefault(key, []).append( VersionRecord( fingerprint=fp, approved_by=approved_by, reason=reason, ) ) return fp This is the piece that keeps CPG usable at scale: drift detection without a deliberate version-bump path just becomes an alert fatigue generator, and alert fatigue is how real teams end up disabling the exact control they need. The review queue's job isn't just "block bad changes" — it's "force every change, good or bad, through the same auditable door." 7. Deployment — Docker and Kubernetes Dockerfile # Dockerfile FROM python:3.12-slim AS builder WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt --target=/deps FROM gcr.io/distroless/python3-debian12 COPY --from=builder /deps /deps COPY cpg/ /app/cpg/ ENV PYTHONPATH=/deps:/app USER nonroot ENTRYPOINT ["python", "-m", "cpg.gateway_server"] YAML # k8s/cpg-gateway.yaml apiVersion: apps/v1 kind: Deployment metadata: name: cpg-gateway spec: replicas: 3 selector: matchLabels: { app: cpg-gateway } template: metadata: labels: { app: cpg-gateway } spec: securityContext: runAsNonRoot: true seccompProfile: { type: RuntimeDefault } containers: - name: gateway image: registry.internal/cpg-gateway:latest securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: { drop: ["ALL"] } resources: limits: { cpu: "500m", memory: "256Mi" } ports: - containerPort: 8443 env: - name: PROVENANCE_STORE_URL valueFrom: secretKeyRef: { name: cpg-secrets, key: store-url } --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: cpg-gateway-egress spec: podSelector: matchLabels: { app: cpg-gateway } policyTypes: ["Egress"] egress: - to: - namespaceSelector: matchLabels: { name: mcp-servers } 8. API Surface YAML # openapi.yaml openapi: 3.1.0 info: title: CPG Gateway API version: "1.0" paths: /v1/tool-call: post: summary: Authorize an MCP tool call against its capability fingerprint requestBody: required: true content: application/json: schema: type: object required: - session_id - capability - requested_host properties: session_id: { type: string } capability: type: object properties: tool_id: { type: string } server_id: { type: string } description: { type: string } input_schema: { type: object } output_schema: { type: object } declared_hosts: type: array items: { type: string } requested_host: { type: string } responses: "200": description: Authorized "409": description: Quarantined — capability drift, egress violation, or blast-radius limit Engineering Trade-Offs A pattern that doesn't name its own trade-offs isn't ready to be referenced by anyone else's architecture review, so here are the ones I'd expect a skeptical staff engineer to raise, and how I'd actually answer them. "Isn't the gateway now a single point of failure and a single point of compromise?" Yes, structurally. Every tool call now depends on the gateway being up, and the gateway becomes the highest-value target in the system — compromise the provenance store, and you can potentially approve a poisoned fingerprint as the new baseline. The mitigation is to run the gateway itself with the least privilege of anything in the stack (the Kubernetes manifest above drops all capabilities and runs read-only-root), replicate it statelessly behind a networked, access-controlled provenance store rather than embedding state in the gateway process, and — critically — require the VersionedProvenanceStore.approve_new_version path to log an approved_by identity that's auditable independently of the gateway itself. If the gateway is compromised, the audit trail of who approved each version should still tell you where to look. "Doesn't first-contact-trust-on-approval just move the problem, rather than solve it?" Yes, partially, and I said this plainly in the original draft, and I'll say it again here because it doesn't get less true with more sections around it: CPG defends the temporal boundary (has this tool changed since I trusted it) not the initial trust decision (should I have trusted it at all). Those are different problems. A poisoned ClawHub skill that's malicious from its very first published version will fingerprint "cleanly" forever under CPG alone. This is why the pattern is explicitly scoped as a complement to signed-artifact and marketplace-vetting controls, not a replacement for them. "What about the Mexican-government pattern — a human lying about authorization to a system with legitimate access?" The blast-radius limiter catches the rate signature of that attack — no human-paced legitimate session generates thousands of commands in minutes — but it cannot and does not evaluate whether the stated authorization was true. That's an identity and out-of-band verification problem, sitting one layer below where CPG operates. Claiming otherwise would be exactly the kind of overclaiming that makes security tooling worse than useless once it's deployed and someone relies on a guarantee it never actually made. "What does this cost at real scale?" The micro-benchmark above (10.2µs median, 50.4µs p99 for the hashing and lookup itself) is small relative to network latency, but a naive synchronous call to a remote provenance store on every single request will not stay small — that cost is dominated by network round-trip time to whatever store backs the ledger, not by CPG's own logic. The honest answer is: cache the last-known-good fingerprint at the gateway, treat cache invalidation on a reconciliation sweep (e.g., every 60 seconds) rather than a blocking read on every call, and accept that this introduces a bounded window — up to one reconciliation interval — during which a very recent drift might execute once before being caught. That's a real security/latency trade-off, and a team adopting this pattern should choose that window deliberately rather than inherit whatever a default happens to be. "Why hash the full schema instead of just the description text?" Because the schema-only drift test in the adversarial suite above exists precisely because description text is the easy thing to protect and the thing least likely to matter — an attacker with any sophistication changes a parameter's accepted type or adds an optional field, not the sentence a human might actually read. Hashing text alone would have caught none of that. Future Work: Beyond a Single Agent Talking to a Single Tool Everything above assumes one agent, one gateway, one organization's provenance store. Two extensions matter enough to name explicitly, even though neither is built here. Agent-to-agent capability provenance. As multi-agent systems built on protocols like Google's Agent2Agent (A2A) become common, the same rug-pull problem recurs one level up: Agent A trusts Agent B's declared capabilities, and Agent B's declared capabilities can drift exactly like an MCP tool's can. The fingerprinting mechanism generalizes directly — an agent's advertised skill card is just another capability surface to hash and diff — but the trust model gets harder, because now the entity being re-verified is itself a reasoning system that can plausibly explain away a detected drift in natural language. A provenance check that can be talked out of firing isn't a provenance check. Federated trust across organizational boundaries. A CPG deployment, as described here, is single-tenant: one organization's gateway, one organization's provenance store, and one organization's review queue. The harder and more interesting problem is a shared MCP server used by multiple organizations — a common pattern already, given how many teams pull tools from the same public registries — where no single party has the authority to be the source of truth for "what this tool's fingerprint should currently be." That likely needs something closer to a signed, append-only, cross-organizational ledger of approved fingerprints (conceptually adjacent to certificate transparency logs) rather than the single-tenant ProvenanceStore shown here. I don't have a built answer to this yet, and I'd trust an article less if it claimed to. Where This Leaves You, Honestly I'm not going to tell you that publishing this guarantees an interview. Nothing does. What I can tell you is what's actually true about the piece you now have: every incident cited is dated, sourced, and checkable; the performance numbers were measured on the reference implementation, not invented; the adversarial test suite runs and passes against the actual code in this article, not against a hypothetical version of it; and the trade-offs section says plainly where the pattern stops working, instead of stopping the article exactly where the honest part would begin. That combination is rare enough on its own. Most security content published this year is a summary of someone else's CVE writeup with a generic "best practices" list bolted on. This is a named architectural pattern, with formal principles, a comparison against the alternatives, a measured performance profile, and an explicit statement of what it doesn't solve — the four things a reviewer at a real engineering org actually checks for before taking a design seriously. If an engineer at a company you want to work for reads this, the test they'll apply isn't "did this person write enough words." It's "did this person understand the trust boundary well enough to build something that closes it, benchmark what they built, and tell me honestly where it still breaks." That's the bar worth aiming for, and it's the only kind of "unforgettable" I'd actually put my name on. Sources Check Point Research, CVE-2025-59536 disclosure (Feb 25, 2026) — via cyberdesserts.com summaryBlueRock Security / Security Boulevard, MCP SSRF analysis (2026)Trend Micro, MCP server exposure scan (2026)Bitsight, "Exposed MCP Servers Reveal New AI Vulnerabilities" (2026)OX Security, "The Mother of All AI Supply Chains" — reported by The Hacker News, April 22, 2026: https://thehackernews.com/2026/04/anthropic-mcp-design-vulnerability.htmlCloud Security Alliance, "MCP Security Crisis: Systemic Design Flaws" (May 4, 2026): https://labs.cloudsecurityalliance.org/research/csa-research-note-mcp-security-crisis-20260504-csa-styled/Engipulse, "The MCP Security Crisis: What the 200,000-Server Vulnerability Reveals" (May 2026): https://engipulse.com/security/the-mcp-security-crisis-what-the-200000-server-vulnerability-reveals-about-ai-agent-architecture/Microsoft Security Blog, "Securing AI agents: When AI tools move from reading to acting" (June 30, 2026): https://www.microsoft.com/en-us/security/blog/2026/06/30/securing-ai-agents-ai-tools-move-from-reading-acting/Beam AI, "5 Real AI Agent Security Breaches in 2026 and Their Lessons" (May 6, 2026), covering the Mexican government breach and OpenClaw/ClawHub incident: https://beam.ai/agentic-insights/ai-agent-security-breaches-2026-lessonsU.S. National Security Agency / CSA, "Model Context Protocol (MCP): Security Design" (PP-26-1834, May 2026): https://media.defense.gov/2026/Jun/02/2003943289/-1/-1/0/CSI_MCP_SECURITY.PDFPointGuard AI, CVE-2026-26118 analysis: https://www.pointguardai.com/ai-security-incidents/microsoft-mcp-server-vulnerability-opens-door-to-ai-tool-hijacking-cve-2026-26118Invariant Labs, tool-shadowing and rug-pull disclosures (2025): https://invariantlabs.ai/blog/mcp-github-vulnerability, https://invariantlabs.ai/blog/whatsapp-mcp-exploited

By Igboanugo David Ugochukwu CORE

AI Doesn't Replace Your Architecture; It Becomes Part of It Picture this. Your team has just integrated a large language model into your enterprise application. The demo looked compelling. The agent interpreted user intent, called several APIs, and returned a coherent result. Everyone in the room was impressed. Then the questions started. What happens when the LLM misinterprets a request and calls the wrong API? Who owns the business logic embedded in that prompt? If the model changes, does the integration break? How do you audit what the AI decided and why? These aren't AI questions. They're architecture questions, and they don't go away just because you've added intelligence to the system. The most important architectural decision you'll make about AI isn't which model to use. It's where the AI sits relative to your existing integration layers. Get that right, and AI becomes a powerful, governable component in a coherent system. Get it wrong, and you'll end up with business logic scattered across prompts, brittle integrations that break when the model updates, and no clear line of accountability when something fails. The question isn't "Can AI call APIs?" It's "Where should AI sit within your architecture?" There are three architectural roles worth separating clearly. API facade. The edge layer that translates external requests into internal operations.Workflow orchestration. The layer that manages multi-step business processes and decision logic.Event-driven integration. The layer that lets systems react to changes without tight coupling. Each serves a different purpose, and AI belongs in different places depending on the business problem you're solving. Figure 1 lays out all three roles side by side, including what AI owns and does not own in each one. Figure 1. Where AI Sits: Three Architectural Roles The table below gives a quick reference for how the three patterns differ before we walk through each one in detail. Pattern Purpose Coupling Determinism Where AI Fits API Facade Translate external requests into internal operations Tight, synchronous Low, request-driven Interpreting intent, extracting parameters Workflow Orchestration Sequence multi-step business processes Moderate, coordinated High, explicit branching Providing probabilistic input to decision points Event-Driven Integration Let systems react to change asynchronously Loose, decoupled Variable, per consumer Consuming and enriching events, never the bus itself This article walks through where AI fits within each pattern, and just as importantly, where it doesn't. 1. Start by Defining What the AI Is Responsible For Before you touch an integration pattern, answer a more fundamental question. What is the AI actually accountable for in this system? This sounds obvious but gets skipped constantly. Teams reach for an LLM because it handles natural language well, then gradually load it with responsibilities it shouldn't own, like validating business rules, managing state, enforcing authorization logic, and driving deterministic workflows. The AI ends up doing everything, which means the architecture owns nothing clearly. Ask these questions before making any integration decisions. Is the AI interpreting human input? Natural language understanding, intent classification, and entity extraction are AI-native tasks where models genuinely add value.Is the AI making recommendations or decisions? A recommendation, such as "this customer is likely to churn," is a probabilistic output. A decision, such as "cancel this subscription," is a deterministic action with business consequences. These require different ownership models.Is the AI coordinating business processes? If yes, be careful. Orchestration logic embedded in prompts is invisible to your governance tooling, untestable in any traditional sense, and will silently drift as the model updates.Which steps require human approval? Any action that is irreversible, regulated, or high stakes should have an explicit human checkpoint that lives in your workflow layer, not inside a prompt. The cleaner your answer to these questions, the cleaner your integration design will be. Blurry responsibilities produce brittle architectures. Define the boundary first. 2. AI at the API Facade, the Conversational Edge The API facade pattern sits at the edge of your system. It's the layer that translates external requests into internal operations. Traditionally, this meant REST or GraphQL endpoints that routed structured requests to back-end services. AI belongs here when the primary challenge is bridging the gap between unstructured human intent and structured system operations. Think of an enterprise procurement assistant. A buyer types, "Reorder the same supplies we used for the Sydney office fit-out, but increase quantity by 20% and flag anything over $5,000 for manager approval." No traditional API handles that sentence on its own. The facade layer is exactly where an LLM adds value. It parses intent, extracts parameters, resolves ambiguity, and maps the request to specific downstream API calls. What AI does well at the facade includes intent resolution, turning natural language into structured API parameters. It also handles entity extraction, pulling order IDs, product codes, dates, and names from conversational input. It supports contextual disambiguation, using conversation history to resolve references like "that vendor" back to a specific vendor ID mentioned earlier. And it enables response synthesis, taking structured API responses and returning natural language answers. What AI should not own at the facade is just as important. Authorization logic belongs in your API gateway or identity layer. Rate limiting and throttling are infrastructure concerns, not model concerns. Core business rules, such as "orders over $5,000 require approval," should live in your workflow layer rather than in a prompt where they're invisible to compliance tooling. The practical pattern is that AI at the facade acts as a structured parameter extractor. It takes conversational input, produces a clean structured intent object, and hands off to APIs that were designed for deterministic consumption. The model interprets. The API executes. The example below shows what that structured intent object might look like once the model has parsed the procurement request above. JSON { "intent": "create_purchase_order", "reference_order": "sydney_office_fitout_2026", "quantity_multiplier": 1.2, "approval_required_above": 5000, "currency": "USD", "extracted_from": "conversational_input", "confidence": 0.94 } Listing 1: Example structured intent object produced at the API facade. Design your facade APIs to accept both human-readable context and machine-structured parameters. Build explicit validation at the API boundary so that when the model produces a malformed or out-of-range parameter, the error is caught and surfaced clearly, not silently swallowed or, worse, acted upon incorrectly. 3. AI Inside Orchestration, Where Flexibility Meets Business Workflows Workflow orchestration manages multi-step business processes, including the sequence of steps, branching logic, error handling, retries, and human approval gates. It's the layer that knows how work gets done, in what order, and under what conditions. The central tension when introducing AI into orchestration is that orchestration is deterministic by design, while AI is probabilistic by nature. A well-governed workflow produces the same output given the same inputs. An LLM does not. Mixing these carelessly produces workflows that are auditable on paper but unpredictable in practice. The architectural resolution is to keep the orchestration layer deterministic while allowing AI to provide probabilistic inputs into specific decision points. Think of AI as a specialized step inside the workflow, one that produces an output that the workflow then acts on according to explicit, auditable logic. A claims processing workflow illustrates this well. The overall process — intake, validation, AI-assisted assessment, human review, approval, and payment — is orchestrated deterministically. The AI participates at the assessment step. It analyzes claim documentation and produces a structured output: an estimated validity score, a list of missing documents, and a recommended action. The workflow then applies explicit branching logic. A score above 0.85 triggers auto approval. A score below 0.4 gets flagged for denial review. Everything in between routes to a human adjudicator. The AI informs. The orchestration decides. Figure 2 shows this flow end to end. Figure 2. AI Inside Orchestration: Claims Processing Workflow A few design principles matter here. Treat AI steps as typed operations with defined inputs and outputs. The orchestration layer should pass a structured payload to the AI and receive a structured response, not an open-ended conversation. This makes the AI step testable, replaceable, and governable. The snippet below shows a minimal example of what a typed contract for an AI step might look like. TypeScript // Typed contract for an AI step inside orchestration interface ClaimAssessmentInput { claimId: string; documents: DocumentRef[]; } interface ClaimAssessmentOutput { validityScore: number; // 0.0 to 1.0 missingDocuments: string[]; recommendedAction: "approve" | "review" | "deny"; } Listing 2: Example typed input/output contract for an AI step inside an orchestrated workflow. Never let the AI own branching logic that has compliance or audit implications. If a decision must be explainable to a regulator, it should live in the orchestration layer where it's visible, versionable, and logged. Design explicit human approval gates. In enterprise workflows, AI recommendations that trigger consequential actions, such as financial transactions, customer notifications, or system changes, should route through a human checkpoint unless you've explicitly validated and signed off on full automation. Build retry and fallback paths. An AI step that fails, times out, or returns a low-confidence result needs a defined fallback, whether that's routing to a human, using a default, or escalating, built into the orchestration rather than handled ad hoc in the calling code. Platforms like OutSystems, which provide visual workflow design alongside AI integration capabilities, make this separation of concerns tangible. You can see exactly where in the process flow an AI step participates, what it receives, and what happens next based on its output. 4. AI and Event-Driven Architecture, Reacting Without Controlling Event-driven architecture decouples systems through a shared event bus. Producers emit events when something happens, and consumers subscribe and react without either party knowing the other exists. It's the pattern that makes large distributed systems composable and independently evolvable. AI fits naturally into event-driven systems, but as a consumer and enricher, not as the event bus itself. The pattern works like this. A transactional system emits a clean, well-defined business event, such as OrderPlaced, CustomerChurnRiskFlagged, or SupportTicketOpened. An AI consumer subscribes, processes the event asynchronously, and either emits a derived event, like ChurnRiskClassified or TicketCategorized, or writes to a downstream store. Core transaction systems remain untouched. This architecture has a key property for AI integration, which is isolation. The AI component can be updated, replaced, or retrained without touching the transactional system that produced the event. The event schema is the contract between them. As long as the AI consumer honors its output schema, the downstream systems don't care what model is running behind it. AI adds value in event-driven systems in several ways. Real-time classification lets an incoming support ticket event trigger AI categorization and routing before a human ever sees it. Anomaly detection allows a stream of transaction events to feed an AI consumer that flags unusual patterns and emits a FraudSignalDetected event. Content enrichment means a DocumentUploaded event can trigger an AI pipeline that extracts entities, generates a summary, and writes structured metadata back to the event stream. A few cautions are worth noting too. Don't use AI to produce events that trigger irreversible transactional operations without a validation step. An AI-emitted event that directly drives a financial settlement or account closure is a governance risk. Keep AI consumers idempotent, since event-driven systems often deliver events at least once, and your AI consumer should produce the same output for the same event input regardless of how many times it processes it. Version your event schemas independently of your AI models. When the model changes, the event contract should remain stable. Break this rule, and you'll find yourself coordinating model updates with schema migrations across multiple teams. 5. Design APIs for AI Variability, Not Just Traditional Applications Traditional API design assumes well-behaved clients. They send valid, structured requests, handle errors predictably, and operate within known parameters. AI agents are different clients. They may generate requests outside expected parameter ranges, retry with slight variations when uncertain, pass natural language fragments where IDs are expected, or call endpoints in unconventional sequences. This changes how APIs should be designed when AI is a first-class consumer. Be explicit about parameter constraints and semantics. Document not just the type of a parameter but what it means and what values are valid. An AI agent that doesn't understand that "customer_status" is an enum with five specific values will guess, and it may guess wrong. Explicit schemas with enumerated values and clear descriptions dramatically reduce the error surface. Return structured, self-describing error responses. When an AI agent calls an API and gets a validation error, the response should tell the agent exactly what was wrong and what correction is expected. A generic 400 with "invalid input" gives the agent nothing to act on. A structured error that says the field "quantity" must be a positive integer, and that a negative value was received, allows the agent to self-correct on retry. Design for idempotency on write operations. AI agents may retry failed calls. Any write operation that could be called multiple times should be idempotent, meaning calling it twice with the same payload should produce the same result as calling it once. This is a baseline requirement for reliable agentic workflows. Consider AI-specific API profiles alongside your standard endpoints. Some teams are building enriched API descriptions, effectively structured, semantic documentation that LLMs can consume during function calling or tool use scenarios. These profiles describe not just syntax but intent, preconditions, and expected postconditions. If your platform supports it, these descriptions significantly improve agentic reliability. 6. Preserve Loose Coupling as AI Capabilities Evolve If there is one thing that is certain about the current AI landscape, it's that it will look different in 18 months. Model capabilities are improving rapidly. New reasoning architectures, longer context windows, better function calling, and multimodal inputs will change what AI can reliably do, which means the design decisions you make today about where AI participates in your architecture will need to evolve. The integration architectures that will age best are the ones that treat AI as a replaceable component behind a stable interface, not as a load-bearing structural element that the rest of the system is built around. Practically, this means a few things. The interface between your AI component and the rest of the system should be typed and versioned, just like any other service boundary. If you replace the LLM behind that interface with a better model, the orchestration layer and downstream consumers shouldn't need to change. Business logic should not live in prompts. Prompts that embed business rules, such as approval thresholds, eligibility criteria, or routing conditions, will drift as models are updated and will be invisible to your governance tooling. Extract that logic into the orchestration or rules layer where it can be versioned and audited. Test AI steps in isolation. Build evaluation harnesses that validate the AI component's outputs against known good test cases. When you upgrade a model, run the evaluation before you promote to production. This is standard software engineering discipline. It just hasn't been applied consistently to AI components yet. Plan for model-level fallback. If a primary model is unavailable or underperforming, your architecture should support routing to a fallback. This is easier to build in advance than to retrofit during an incident. The teams that will maintain architectural coherence as AI evolves are the ones that applied the same separation of concerns discipline to AI components that they've always applied to services, databases, and APIs. 7. Build Observability Across AI and Integration Layers Debugging traditional distributed systems is hard. Debugging systems where one of the components is an LLM is harder. The failure modes are different. The system may be technically healthy while producing incorrect, inconsistent, or subtly wrong outputs. A 200 OK from an AI step tells you the HTTP call succeeded. It says nothing about whether the response was accurate, relevant, or safe. Observability in AI integrated architectures needs to span multiple layers simultaneously. At the AI component level, teams should capture the full prompt sent to the model, not just the output, along with the raw model response before any parsing or post-processing. Token counts, latency, and model version matter too, as do confidence scores or reasoning traces where the model provides them, and retry attempts or fallback triggers. At the integration layer, capture which APIs the AI called, with what parameters, and what the responses were. Track workflow step durations and branching decisions, event payloads at each stage of processing, and human review decisions and overrides. At the business outcome level, ask whether the end-to-end process completed successfully, whether AI-assisted decisions matched expected patterns, and where AI components are producing outputs that require human correction. Platforms that provide centralized monitoring across application logic, integrations, and workflows, such as OutSystems, reduce the instrumentation burden by giving teams a single observability surface rather than requiring separate tooling for each layer. This matters most during incident response, when you need to trace a failure from a user-visible symptom back through the AI component, through the API calls it made, and into the underlying workflow state, quickly. One practice worth establishing early is shadow mode evaluation. Before promoting AI-assisted decisions to full automation, run the AI in parallel with existing logic and compare outcomes without acting on the AI's output. This builds confidence in the model's reliability on your specific data distribution before you depend on it in production. Conclusion. Integration Architecture Is Still the Foundation AI agents are sophisticated components, but they're still components. They have inputs and outputs. They can fail. They need to be tested, monitored, versioned, and replaced, and crucially, they need to sit somewhere coherent in your architecture. The teams that will get the most out of AI are the ones that ask the architectural questions first. What is the AI responsible for? Where does its output go? Who owns the logic around it? How will we know when it's wrong? The answer isn't a different architecture for AI. It's the same architectural discipline that enterprise systems have always required, applied with precision to a new kind of component. API facade, orchestration, and event-driven architecture were built to manage complexity, enforce separation of concerns, and keep systems evolvable. AI makes all three more valuable, not less. The question is simply where, within each, the intelligence belongs. References APISDOR. "How AI Agents Are Reshaping Enterprise Software Architecture." 2026. https://www.apisdor.com/blog/how-ai-agents-are-reshaping-enterprise-software-architecture/Elementum. "Enterprise AI Orchestration: Complete Architecture Guide." 2026. https://www.elementum.ai/blog/enterprise-ai-orchestration-architectureDevRev. "AI Agent Orchestration: Patterns, Pitfalls & the Shared Memory Architecture." 2026. https://devrev.ai/blog/ai-agent-orchestrationViston AI. "Architecture for Enterprise AI Orchestration: A 2026 Blueprint." 2026. https://viston.tech/recommending-a-production-ready-architecture-for-enterprise-ai-orchestration/"Autonomous Event-Driven Multi-Agent Orchestration for Enterprise AI at Scale." arXiv, 2026. https://arxiv.org/pdf/2606.20058Zuplo. "The API Readiness Gap: How to Design APIs That AI Agents Can Actually Use." 2026. https://zuplo.com/learning-center/api-readiness-gap-agent-callable-apis freeCodeCamp. "How to Design APIs for AI Agents." 2026. https://www.freecodecamp.org/news/how-to-design-apis-for-ai-agents/"Self-Reflective APIs: Structure Beats Verbosity for AI Agent Recovery." arXiv, 2026. https://arxiv.org/pdf/2606.05037 "Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework." arXiv, 2026. https://arxiv.org/pdf/2606.08867"Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes." arXiv, 2026. https://arxiv.org/pdf/2603.06847 Agentive AI Agents. "AI Agent Error Handling: 7 Proven Practices." 2026. https://agentiveaiagents.com/ai-agent-error-handling-best-practices/

By Jubin Abhishek Soni CORE

Large language models (LLMs) are impressive — until they are not. If you ask one about your internal data, your product catalog, or your users' reviews, it will either hallucinate an answer or admit it does not know. The solution most teams reach for is retrieval-augmented generation (RAG). This retrieves relevant data first, injects it into the prompt as context, and lets the model answer from that context rather than from memory. GraphRAG takes this a step further. Instead of retrieving only text chunks, it can use graph relationships to retrieve connected context, following relationships between entities to build richer, more structured context. The result can provide answers grounded in both data and the relationships between that data. In this article, we'll walk through a practical GraphRAG implementation using Spring AI and Neo4j, built on top of a Goodreads book and review dataset. We'll cover the data model, loading the data, setting up the vector index, running the Spring Boot application, and some lessons learned along the way. The full source code is available on GitHub. What We Are Building The application answers natural language queries like "find books with a happy ending" or "something encouraging" by combining two retrieval mechanisms in Neo4j: Vector search – embeds the search phrase via OpenAI and finds semantically similar book reviews using cosine similarity.Graph traversal – follows the WRITTEN_FOR relationship from matched reviews to their associated books, giving the LLM structured book context rather than raw review text. This example uses a simple GraphRAG pattern where vector search identifies relevant reviews and graph traversal expands the retrieved context to connected books. The LLM then summarizes the retrieved books in the context of the original search phrase. The architecture looks like Figure 1. Figure 1. Architecture. Prerequisites Before we start, we will need: Java 21 or laterA Neo4j AuraDB instanceAn OpenAI API key Installing Java If Java is not already installed, the recommended distribution is Temurin from the Adoptium project, available at adoptium.net. Installers are available for Windows, macOS, and Linux. Once installed, verify with: Shell java -version We should see something like openjdk version "21.x.x". The project uses the Maven wrapper, so there is no need to install Maven separately. Setting Up Neo4j AuraDB AuraDB is Neo4j's fully managed cloud database. A free tier is available. Sign up at neo4j.com/product/auradb/.Create a new AuraDB Free instance.When the instance is created, download or note the credentials — the URI, username, and password. Neo4j only shows the password once, so save it somewhere safe.Once the instance is running, open the built-in Query tab and verify connectivity: cypher MATCH (n) RETURN count(n) . This should return 0. We are ready to load data. AuraDB Free includes Awesome Procedures on Cypher (APOC), a utility that provides numerous procedures and functions for data handling. We'll use APOC for the data loading steps. The Data Model The dataset is built around three core node types: Book – 10,000 books from the Goodreads UCSD datasetAuthor – 12,371 authorsReview – 69,791 user reviews, each linked to a book via a WRITTEN_FOR relationship There is also a User node (44,827 users) linked to reviews via a PUBLISHED relationship, although the main application focuses on Books and Reviews. The graph model is shown in Figure 2. Figure 2. The Goodreads Dataset. The key insight is that the Review node carries two things: the review text and 1,536-dimension embeddings generated using an OpenAI embedding model. This is what makes vector similarity search possible without a separate vector database — Neo4j handles both the graph and the vectors. The Goodreads data used in this article is derived from the UCSD Book Graph dataset and related Goodreads datasets released by researchers at the University of California, San Diego, including Mengting Wan, Julian McAuley, and collaborators. The data is provided for research and educational purposes. If you use these datasets in your own work, please cite the following publications: Mengting Wan and Julian McAuley, Item Recommendation on Monotonic Behavior Chains, RecSys 2018.Mengting Wan, Rishabh Misra, Ndapa Nakashole, and Julian McAuley, Fine-Grained Spoiler Detection from Large-Scale Review Corpora, ACL 2019. Loading the Data Let's load the data step by step in the AuraDB Query tab. Run each of the following blocks separately. Constraints and Indexes First, let's set up the constraints and the vector index: Cypher CREATE CONSTRAINT FOR (b:Book) REQUIRE b.book_id IS UNIQUE; CREATE CONSTRAINT FOR (a:Author) REQUIRE a.author_id IS UNIQUE; CREATE CONSTRAINT FOR (r:Review) REQUIRE r.id IS UNIQUE; CREATE CONSTRAINT FOR (u:User) REQUIRE u.user_id IS UNIQUE; CREATE INDEX FOR (r:Review) ON (r.user_id); Then create the vector index on the Review node's embedding property: Cypher CREATE VECTOR INDEX `review-text` IF NOT EXISTS FOR (n:Review) ON (n.embedding) OPTIONS { indexConfig: { `vector.dimensions`: 1536, `vector.similarity_function`: 'cosine' }; Note the index name review-text — we will come back to this in the lessons learned section. Loading Books and Authors The data are hosted on Neo4j's public servers, so we can load them directly via APOC: Cypher CALL apoc.load.json("https://data.neo4j.com/goodreads/goodreads_books_10k.json") YIELD value as book MERGE (b:Book {book_id: book.book_id}) SET b += apoc.map.clean(book, ['authors','similar_books'],[""]); Next, we'll load the initial author stubs: Cypher CALL apoc.load.json("https://data.neo4j.com/goodreads/goodreads_books_10k.json") YIELD value as book WITH book UNWIND book.authors as author MERGE (a:Author {author_id: author.author_id}); and then populate the author nodes with the full data: Cypher CALL apoc.periodic.iterate( 'CALL apoc.load.json("https://data.neo4j.com/goodreads/goodreads_book_authors.json.gz") YIELD value as author', 'WITH author MATCH (a:Author {author_id: author.author_id}) SET a += apoc.map.clean(author, [],[""])', {batchsize: 10000} ); Next, we'll create the AUTHORED and SIMILAR_TO relationships: Cypher CALL apoc.load.json("https://data.neo4j.com/goodreads/goodreads_books_10k.json") YIELD value as book WITH book MATCH (b:Book {book_id: book.book_id}) WITH book, b UNWIND book.authors as author MATCH (a:Author {author_id: author.author_id}) MERGE (a)-[w:AUTHORED]->(b); Cypher CALL apoc.load.json("https://data.neo4j.com/goodreads/goodreads_books_10k.json") YIELD value as book WITH book MATCH (b:Book {book_id: book.book_id}) WITH book, b WHERE book.similar_books IS NOT NULL UNWIND book.similar_books as similarBookId MATCH (b2:Book {book_id: similarBookId}) MERGE (b)-[r:SIMILAR_TO]->(b2); Loading Reviews This step can take several minutes, as it is pulling and processing approximately 70,000 reviews from a gzipped JSON file: Cypher CALL apoc.load.json("https://data.neo4j.com/goodreads/goodreads_reviews_dedup.json.gz") YIELD value as review CALL { WITH review MATCH (b:Book) WHERE b.book_id = review.book_id WITH review, b MERGE (r:Review {id: review.review_id}) SET r += apoc.map.clean(review, [],[""]) WITH b, r MERGE (b)<-[rel:WRITTEN_FOR]-(r) } in transactions of 20000 rows; Note that review.review_id is stored as the Review node's id property, which Spring AI expects when mapping vector search results. Then we'll separate the User nodes from the Review data: Cypher MATCH (r:Review) WHERE r.user_id IS NOT NULL CALL { WITH r MERGE (u:User {user_id: r.user_id}) WITH r, u MERGE (r)<-[:PUBLISHED]-(u) } in transactions of 20000 rows; Adding the text Property Spring AI maps vector search results to Document objects using a property named text. Our review data uses review_text, so we need to add the text property: Cypher MATCH (r:Review) CALL { WITH r SET r.text = r.review_text } IN TRANSACTIONS OF 20000 ROWS; Loading Pre-Generated Embeddings Rather than generating embeddings at runtime, which costs tokens and time, we'll load pre-computed embeddings hosted by Neo4j. This step also takes several minutes: Cypher LOAD CSV WITH HEADERS FROM "https://data.neo4j.com/goodreads/review_embeddings.psv" as row FIELDTERMINATOR '|' CALL { WITH row MATCH (r:Review {id: row.reviewId}) CALL db.create.setNodeVectorProperty(r, 'embedding', apoc.convert.fromJsonList(row.embedding)) RETURN r } in transactions of 1000 rows WITH r RETURN count(r); Once complete, we can verify the embeddings loaded correctly: Cypher MATCH (r:Review) WHERE r.embedding IS NOT NULL RETURN count(r) AS reviews_with_embeddings We should see 69791. Exploring the Data Before running the application, let's take a look at what we have loaded. Here are a few useful queries to run in the AuraDB Query tab. Browse the top-rated books: Cypher MATCH (b:Book) RETURN b.title, b.average_rating ORDER BY b.average_rating DESC LIMIT 10 Browse books with their authors: Cypher MATCH (a:Author)-[:AUTHORED]->(b:Book) RETURN a.name, b.title, b.average_rating ORDER BY b.average_rating DESC LIMIT 10 Inspect a sample embedding — we can see the first few dimensions of a review's vector: Cypher MATCH (r:Review) WHERE r.embedding IS NOT NULL RETURN r.id, r.text, r.embedding[0..5] AS embedding_sample LIMIT 5 Building and Running the Application Let's clone the GitHub repo and get the application running: Shell git clone https://github.com/JMHReif/springai-goodreads.git cd springai-goodreads Set the environment variables for Neo4j AuraDB and OpenAI, as follows: Shell export SPRING_NEO4J_URI=neo4j+s://xxxx.databases.neo4j.io export SPRING_NEO4J_AUTHENTICATION_USERNAME=your_username_here export SPRING_NEO4J_AUTHENTICATION_PASSWORD=your_password_here export SPRING_AI_OPENAI_API_KEY=your_openai_key_here These variables must be set in the terminal session used to run the Spring Boot application, specifically the window where you run ./mvnw spring-boot:run. The terminal used for curl commands does not need them. To avoid having to re-export them each time, you can add them to your shell profile (e.g. ~/.zshrc on macOS or ~/.bashrc on Linux) or save them in a small shell script and source it before starting the app. Now we'll start the application from the root of the cloned repo, where the pom.xml and mvnw files live, as follows: Shell ./mvnw spring-boot:run Maven will download dependencies on the first run. Once the startup banner appears, the app is ready on port 8080. The Four Endpoints The application exposes four REST endpoints, each representing a different retrieval strategy: /hello — Baseline LLM Call Shell curl "http://localhost:8080/hello" A simple call to the LLM with no retrieval. Useful to verify the OpenAI connection is working. /llm — LLM With No Context Shell curl "http://localhost:8080/llm?searchPhrase=happy%20ending" This sends the search phrase directly to the LLM with no data from Neo4j. The model answers from its training data — fast, but prone to hallucination and not grounded in our Goodreads data. /vector — Vector Search Only Shell curl "http://localhost:8080/vector?searchPhrase=happy%20ending" Spring AI embeds the search phrase via OpenAI, queries the review-text vector index in Neo4j, and passes the matching review text to the LLM. Semantic matching works well here — the phrase does not need to match any exact words in the reviews. /graph — Full GraphRAG Pipeline Shell curl "http://localhost:8080/graph?searchPhrase=happy%20ending" This is the full pipeline. Vector search finds the most semantically similar reviews, the graph traversal follows the WRITTEN_FOR relationship to retrieve the associated Book nodes, and the LLM receives structured book context rather than raw review text. Let's look at the output for a few different search phrases: Shell curl "http://localhost:8080/graph?searchPhrase=encouragement" curl "http://localhost:8080/graph?searchPhrase=high%20tech" curl "http://localhost:8080/graph?searchPhrase=caffeine" The contrast between /llm and /graph on the same phrase is the most compelling comparison — the LLM answers from memory in one case and from our actual Goodreads data in the other. GraphRAG Uses Both Vector Search and Graph Traversal It's worth comparing the two retrieval strategies directly, as shown in Figure 3. Figure 3. Vector Search and Graph Traversal. Neither approach is strictly better. Rather, they are complementary. Vector search handles fuzzy, intent-driven queries that keyword search would miss entirely. Graph traversal adds relationship-aware context that makes the LLM response richer and easier to trace back to source data. The /graph endpoint combines both. Lessons Learned Here are four things worth knowing before setting this up from scratch. Vector index naming matters. Spring AI's default vector index name is spring-ai-document-index. This project requires review-text. If the index is created with the wrong name, the application throws a runtime error that is not immediately obvious. Always check the index name configured in the application against the one created in Neo4j.Review nodes need id and text properties. Spring AI maps vector search results to Document objects using properties named id and text. In this dataset, review_id is mapped to the Review node's id property during loading, but the review text is stored as review_text. We therefore add a text property so Spring AI can map the results correctly. Without the expected properties, vector search returns results, but the book list comes back empty — the model gets no context and answers from memory instead.Pre-generated embeddings save time and money. Generating 69,791 embeddings at runtime via the OpenAI API would be slow and costly. Loading pre-computed embeddings from a file is much faster for initial development setups. The trade-off is that the embeddings are fixed, as they were generated with a specific OpenAI model and will need to be regenerated if the model changes.Data loading takes patience. The two long-running steps are the review load and the embedding load. Plan for this, although both steps only need to be done once and the database can be left running between sessions. Summary GraphRAG is a practical pattern, not just a research concept. By combining Neo4j's graph traversal with its vector index, we get two retrieval mechanisms in a single database, and no separate vector store is required for this architecture. Spring AI provides the abstractions to wire it all together in a way that will feel familiar to any Spring developer. The Goodreads domain is approachable and familiar to many readers, but the architecture generalizes to any graph of connected entities, such as product catalogs, knowledge graphs, and collections of documents. If you have relationships in your data, a graph database gives you relationship-aware retrieval capabilities that a plain vector store does not provide. The full source code is on GitHub. Acknowledgements I thank my colleague Jennifer Reif for sharing the Spring AI example.

By Akmal Chaudhri CORE