Ops Narrator

• 

  deck

Inspiration

A SOC drowns in alerts. A single "encoded PowerShell ran" alert is hours of analyst work: decode the payload, figure out what it does, check whether it spread, find the privilege escalation and lateral movement, scope the blast radius, and write it up. Most alerts never get that treatment — they're triaged in seconds and closed. We wanted an agent that does the full investigation automatically and shows its work, so an analyst gets a finished brief to verify instead of a raw alert to chase.

## What it does

A Splunk saved search fires a webhook the moment it sees encoded PowerShell. Instead of paging a human, the alert is handed to an autonomous agent (Claude Opus 4.7) that:

• decodes the malicious payload (base64 → UTF-16LE, nested layers) and extracts indicators.
• runs its own follow-up SPL queries through the official Splunk MCP Server to pivot through the data,
• reconstructs the kill chain across hosts — process ancestry, UAC-bypass privilege escalation, WMI lateral movement, and C2 — and
• writes a SOC-grade incident brief (severity, findings, MITRE mapping, IOCs, timeline, scope, recommended containment).

Every run also emits a step-by-step JSONL reasoning trace — one event per thinking step, tool call (with latency + row count), tool result, and detected hypothesis revision — rendered in a single-page trace viewer. You see how it reasoned, not just its conclusion.

## How we built it

The core is a tool-using agent loop built on the Anthropic SDK with claude-opus-4-7, adaptive extended thinking, and a hard iteration + wall-clock budget. The model gets one triggering alert and a menu of eight investigative tools (decode_payload, splunk_search, find_process_ancestry, find_pattern_across_hosts, check_unusual_parents, find_lateral_execution, trace_account_activity, finalize_brief) and decides turn by turn what to look at next.

All Splunk-backed tools reach Splunk through the official Splunk MCP Server (Splunkbase app 7931) over streamable HTTP via the Model Context Protocol — so the agent operates over real Splunk data, not hardcoded answers. A FastAPI endpoint receives the webhook, acks in 200ms, runs the agent in the background, and serves the trace viewer + a runs dashboard.

A deliberate design choice: the model-facing tool descriptions and system prompt are generic — they never name the dataset, threat family, hosts, or expected outcome. The agent has to discover the incident from the data on every run.

## Challenges we ran into

• The MCP token was the hardest part. The official Splunk MCP Server rejects a normal Splunk token with 403 Invalid token audience — it requires audience mcp and an RSA-encrypted token. We had to mint it from the app's own /services/mcp_token endpoint.

• Opus 4.7's thinking API changed. The old budget_tokens knob 400s on 4.7; we switched to adaptive thinking + an effort setting, and had to turn on summarized thinking so the trace logger could actually capture the model's reasoning.

• Real-world data is messy. Timezone skew between the data (CST) and the incident wall-clock (EDT), multivalue alert fields arriving as JSON arrays with "-" null markers, and account names that differ between process and auth events all had to be handled.

• Cost discipline. Opus runs aren't free, so we added rolling prompt-caching breakpoints that cut prompt-token cost ~42% on a multi-turn run, and built a pluggable provider layer to iterate on free backends without burning credits.

## What we learned

Giving an agent generic tools and forcing it to discover the incident rather than spoon-feeding it the answer produces investigations that genuinely reason, including real hypothesis pivots mid-run. And capturing the reasoning trace turned out to matter as much as the verdict: a brief you can't audit isn't trustworthy in a SOC.

## What's next

Persistent MCP connection reuse for sub-30s investigations, tuning the tool outputs so the hypothesis pivot is reliable across runs, and expanding beyond the encoded-PowerShell detection to a broader library of alert types.

Built With

• anthropic
• claude
• claude-opus-4.7
• fastapi
• mcp
• model-context-protocol
• python
• splunk
• splunk-mcp-server
• uvicorn