Sentinel Brief

Splunk multi-agent war room: a fired detection becomes an auditable incident brief plus a backtested SPL fix that cuts false alarms 75%, in under 3 minutes.

Comment

Inspiration

Every SOC runs on the same painful loop: a Splunk detection fires, and a human spends 30–60 minutes pivoting across indexes to answer four questions — is it real, how far did it spread, what do we do, and the one nobody ever gets to: why did our own rule over-fire? That last question is detection drift — the same noisy rule pages the SOC at 2am on a legitimate service account, week after week, forever. Splunk's own Agentic SOC ships a Triage Agent that answers the first question. We asked: what if an agent closed all four — and got smarter at detection every time it ran?

What it does

When a detection fires, Sentinel Brief dispatches a supervisor agent that orchestrates four specialist subagents and returns one typed, auditable incident brief:

  • Verdict + confidence + MITRE ATT&CK mapping (true/false positive, severity)
  • Multi-index timeline reconstructing the kill chain across the auth and edr indexes
  • Ranked blast radius — every affected entity, ordered by exposure, origin called out
  • Human-approved containment — SOP-grounded actions, every one dry-run and gated on a human
  • ⭐ A self-improving SPL detection fix — the agent rewrites the over-firing rule, keeping the logic that caught the real attack and adding an EDR correlation so it only fires on a genuine signal. This removes the entire false-positive class while preserving the true positive.
  • ⭐⭐ A measured, backtested proof of that fix — Sentinel Brief doesn't just suggest the new rule, it replays both the old and the new rule over the live historical data and reports the result deterministically: old rule 4 alerts (3 false) → new rule 1 alert (0 false) = 75% fewer alerts, true positive retained. The three eliminated alerts are genuine false positives — legitimate svc_monitor / svc_patch / svc_vuln service accounts that fan out across admin shares but carry no correlated attack-tool EDR signal. That correlation is the honest discriminator; we make no claim of an external ground-truth oracle. The alert that paged you at 2am never pages you again — and now we can prove it.

The output is a typed Pydantic object, not a chat transcript — every field, including the backtest metric, is auditable.

How we built it

  • Splunk splunklib.ai — the agentic spine: a supervisor delegating to four subagents, each on its own model tier (the SDK's context-bloat-mitigation pattern used at full depth).
  • Splunk MCP Server — the Correlator subagent queries Splunk through the MCP Server, invoking splunk_run_query against live indexes. The encrypted MCP token is minted automatically by splunklib.ai through the admin Service — no manual token wiring.
  • Splunk Developer Tools — local SPL tools registered with @registry.tool() and executed via ToolContext.service.jobs.oneshot.
  • Splunk Enterprise 10.4 — real auth + edr indexes seeded with a lateral-movement scenario (a compromised svc_backup account pivoting from WKS-014 across admin shares after a PsExec exec).
  • Google Vertex AI / Gemini — gemini-2.5-pro for the reasoning agents (Supervisor, Adjudicator), gemini-2.5-flash for the high-throughput ones (Correlator, Responder, Detection Engineer).
  • Backtest engine — a deterministic helper that replays the old and new detection rules over the live indexes and computes the false-positive reduction from real query results (not LLM prose).
  • Pydantic for the typed IncidentBrief; Python 3.14; 40 offline tests.

See ARCHITECTURE.md for the full data-flow diagram and MCP-EVIDENCE.md for the captured proof of the live MCP tool calls.

Challenges we ran into

  • splunklib.ai ships on the SDK's develop branch and needs Python ≥3.13 — a Day-0 spike pinned the exact install + import shape before we built anything.
  • Making the agent genuinely call the MCP Server (not just local SPL) meant tracing the SDK's connect_remote_mcp path to learn it auto-mints an encrypted token through the Service — then allowlisting the remote tools so the LLM could use them.
  • Keeping the live integration fail-safe: if MCP is unavailable the agent degrades cleanly to local SPL, so the brief never breaks.

Accomplishments we're proud of

  • The self-improving detection loop — a capability no first-party triage agent ships.
  • We measured it, we didn't just claim it. A deterministic backtest replays the old and new rules over real Splunk history and proves a 75% false-positive reduction (4→1 alerts, 3 FPs eliminated) with the true positive retained — a hard number computed from real query results, not LLM prose.
  • A real, end-to-end run on live Splunk with the agent calling splunk_run_query through the MCP Server — including the backtest's own OLD-rule query, routed through MCP (evidence captured).
  • One scannable, typed brief — auditable, not a chatbot.

What we learned

How to drive Splunk's agentic stack at depth: splunklib.ai supervisor/subagent orchestration, the MCP Server's token + tool model, and SPL tool execution through ToolContext — and how to keep a multi-agent system demo-safe with graceful degradation.

What's next

Promote the Responder's dry-run actions to real SOAR playbooks behind approval; expand the detection-engineering loop to push the fixed rule straight into savedsearches.conf as a PR; and swap the Adjudicator onto Splunk's hosted Foundation-Sec model when available on the platform.

Built with

splunk · splunklib.ai · splunk-mcp-server · spl · python · google-vertex-ai · gemini · pydantic · model-context-protocol · react · vite · tailwindcss · framer-motion · fastapi

The console (UI)

A single-screen SOC Detection-to-Decision console renders the typed IncidentBrief as a security-ops war room: the fired-alert queue, an animated supervisor + 4-subagent fan-out, then the adjudicated brief with the backtested 75% false-positive reduction as the hero element. Vite + React + Tailwind + Framer Motion over a FastAPI backend; it bundles a real captured run so the demo renders instantly and never depends on a live connection on camera.

Built With

Share this project:

Updates