-
-
SIFTGuard architecture: five orchestrators on one typed MCP server. Red trust boundary separates untrusted agent reasoning.
-
Real F1 across 3 datasets. Native Claude is the only orchestrator scoreable on all three (mean 0.867). Orchestration is the only variable.
-
Spoliation suite: 15/15 evidence-destruction attacks blocked at the MCP layer by architecture. rm, dd, mkfs, path traversal all denied.
-
Live run, Native Claude loop: agent self-corrects through failed tool calls (analyzeMFT, log2timeline) and keeps reasoning toward a verdict.
-
Completed run: 12 IOCs mapped. usbclient.exe dropper, license_ctrl/ftusbsrvc, C2 at xxx.10, MITRE techniques — force-directed graph.
-
Auto-generated incident report: executive summary, event timeline, IOCs, MITRE mapping. Every traces to an audit-DB tool-execution row.
Inspiration
In November 2025, Anthropic's security team published findings on GTG-1002 — a Chinese state-sponsored operation where attackers used Claude Code to run autonomous reconnaissance, exploitation, and lateral movement at 80-90% autonomy. The AI handled everything at request rates described as "physically impossible" for human operators.
That was the offensive side.
The SIFT Workstation is the defensive platform. 18 years old. 200+ tools. Trusted by every serious incident responder on the planet. And defenders using it still look up command-line flags during active incidents.
The gap is real: adversaries move at machine speed. Defenders don't. SIFTGuard closes that gap.
What it does
SIFTGuard is an autonomous DFIR agent that runs five orchestration paradigms — Anthropic native loop, LangGraph, OpenAI function-calling, Gemini 3 Pro, and Claude Code headless CLI — against a single typed MCP server of forensic tools. The same model API surface and the same Pydantic-validated tools are held fixed across all five adapters; orchestration is the only variable. We measure what that variable buys across three forensics datasets — memory APT, NTFS disk, and a live IP-theft investigation — and publish the F1 numbers.
Headline — 5 orchestrators × 3 datasets
| Orchestrator | TEST-001 (memory) | TEST-002 (disk) | TEST-003 (ROCBA) | Cross-dataset mean |
|---|---|---|---|---|
| Native Loop (claude-sonnet-4-6) | 1.000 | 0.600 | 1.000 | 0.867 |
| OpenAI FC (gpt-5.5) | 1.000 | 0.800 | — † | 0.900 (2/3) |
| Claude Code (headless, Sonnet 4.6) | 1.000 | 0.000 ‡ | — † | 0.500 (2/3) |
| LangGraph (Sonnet 4.6) | 0.750 | 0.000 ‡ | 0.015 | 0.255 |
| Gemini 3 Pro | 0.250 | 0.400 | — † | 0.325 (2/3) |
Scorer: applicability-aware F1 (siftguard.eval.score, GT v1.1.0). TEST-001 = SRL-2018 APT memory image, 4 applicable IOCs. TEST-002 = NIST CFReDS Hacking Case (Greg Schardt / Mr. Evil), NTFS disk image, 5 applicable IOCs. TEST-003 = SANS ROCBA Standard Forensic Case (Fred Rocba IP theft), NTFS C: drive with a broken backup boot sector, 12 applicable IOCs.
† Returned no scoreable verdict on TEST-003 — agent terminated before surfacing disk artifacts in report text.
‡ Tool-applicability failure on raw disk evidence — documented in docs/LIMITATIONS.md.
Native Loop is the only orchestrator that produces scoreable verdicts across all three datasets — memory APT, NTFS disk forensics, and live IP-theft with anti-forensic counterplay — at a cross-dataset mean F1 of 0.867. Same model API surface, same typed MCP server, same prompts across all five adapters. Orchestration is what differs.
How we built it
Why orchestration is the only variable (verbatim, ADR-006 §1):
A Digital Forensics and Incident Response (DFIR) agent that ships behind a Security Operations Center (SOC) perimeter cannot be coupled to a single LLM vendor or a single orchestration framework. The coupling is not an aesthetic concern; it is an operational and regulatory liability.
Outage risk (Anthropic 2025-05; OpenAI 2025-06; Google 2025-09), regulatory diversity (BaFin-supervised banks, KRITIS infrastructure, FedRAMP-Moderate, HIPAA), model deprecation cycles, and on-prem deployment requirements all push the same direction: the orchestration layer has to be indifferent to which model is reasoning behind it. Five live adapters on one typed MCP surface is how SIFTGuard treats vendor neutrality as a property of the architecture, not a marketing claim.
What the multi-orchestrator design surfaced (verbatim, ADR-006 §5.2):
Lowest-to-highest cost ratio on the same evidence file: $0.1949 (OpenAI FC) → $0.5293 (Claude Code), a 2.72× spread. This is not measurement noise. Median seeded variance for the canonical native-loop baseline is σ = 0.000 across n = 6 seeds (TEST-001, F1 = 0.909, recorded in ADR-001 §4 D5). A 2.72× delta with σ ≈ 0 on the baseline is structural and explainable: OpenAI FC's four iterations reflect aggressive parallel tool-call batching driving cost down; Claude Code's eighteen iterations reflect headless MCP-RPC round-trip overhead — the design tradeoff named in §3.4 — driving cost up. The three direct-API adapters in between ($0.2289–$0.2591) cluster tightly because they pay neither extreme. The framework would have been blind to all of this under any single-orchestrator design (A1) or single-framework design (A2).
The architectural claim. A SIFTGuard agent cannot alter, delete, or fabricate evidence, and we prove it with automated tests rather than a policy document — 15/15 spoliation suite, run on every push to main. Four hard boundaries make that claim mechanical, not aspirational:
- Typed MCP boundary. Every forensic tool is a Pydantic-validated function with a frozen schema. The agent never sees raw shell; it sees structured findings with provenance.
- Instrumented agent loop. Every iteration writes a structured snapshot — tokens, cost, confidence vector, hypothesis state, self-correction events — immutable once written.
- Append-only audit DB. SQLite with insert-only access enforced at the data layer. Migrations versioned and verified at startup.
- Versioned methodology. Every report stamped with the methodology version and SHA-256 of
EVAL_FRAMEWORK.md. Change the scoring rules and the version bumps; prior results stay attributable to the methodology that produced them.
Architectural rationale and rejected alternatives: ADR-001 (evaluation framework), ADR-006 (multi-orchestrator + vendor lock-in), ADR-007 (spoliation moat). Full ADR index at docs/adr/.
Challenges we ran into
Single-variable isolation across five paradigms. LangGraph state graphs, OpenAI's function-calling loop, Gemini's tool-use surface, Anthropic's native Messages API, and Claude Code's headless CLI each carry different assumptions about state, retries, parallelism, and trace shape. Getting all five to consume the same Pydantic MCP server with the same model API surface and the same prompts — so that orchestration becomes the only variable — was the bulk of Phase B engineering.
Generalization across evidence types is where orchestrators separate. Three of five score F1 = 1.000 on memory (TEST-001). The dataset shift to NTFS disk (TEST-002) and the live IP-theft case (TEST-003) is what pulls them apart: LangGraph and Claude Code fail on raw disk because the memory-focused Volatility 3 surface does not match the evidence type — iteration-budget exhaustion, not hallucination; both kept reasoning correctly about a tool surface that could not return findings. Only Native Loop produces a scoreable verdict on all three. We treat that asymmetry as a result, not a bug — it is exactly what a single-orchestrator design would have hidden. Documented in docs/LIMITATIONS.md.
Applicability-aware scoring. The original text-match scorer punished correct verdicts on the wrong evidence type. We built an applicability layer into the ground truth (GT v1.1.0) so that an IOC counts as scoreable only when the tool surface could plausibly produce it. Specification: docs/EVAL_FRAMEWORK.md.
Safe execution without a sandbox. SIFT runs real forensic tools against real evidence. The 15/15 spoliation suite actively attempts to destroy evidence and verifies all fifteen attacks are blocked at the MCP layer — by architecture, not by prompt.
Accomplishments that we're proud of
- Five orchestrators live on the same typed MCP server with F1 measured per dataset, across three distinct evidence types
- Native Loop generalizes across all three datasets at cross-dataset mean F1 = 0.867 — the only orchestrator scoreable on memory, disk, and the live IP-theft case
- Three of five score F1 = 1.000 on TEST-001; OpenAI FC clears F1 ≥ 0.80 on both datasets it scored
- Spoliation test suite: 15/15 attacks blocked architecturally at the MCP layer — not by prompt
- Applicability-aware F1 scorer with versioned methodology and SHA-256 stamping
- Append-only audit DB — every finding in every report traces to a tool-execution row
- Live FastAPI/SSE dashboard streams tool calls, IOC detection, and hypothesis state in real time
- SBOM signed with Sigstore keyless; SLSA Level 3 build provenance
What we learned
Single-variable testing changes what the project is allowed to claim. We started assuming orchestration was incidental — pick any framework, the model does the work. The 2.72× cost spread with σ ≈ 0 on the baseline, and the fact that three orchestrators tie at F1 = 1.000 on memory but only one survives the shift to disk and live IP-theft evidence, showed orchestration is the variable that decides whether a DFIR agent is deployable in a regulated SOC. That is not a result you can produce with one orchestrator and a hypothesis.
Architecture Decision Records are not internal paperwork. ADR-006 §1 and §5.2 became the strongest paragraphs in the README hero and this Devpost. Writing the architectural rationale in the form a judge can read pays compounding returns across every submission artifact.
What's next
- Registry, MFT, and filesystem tools end-to-end on live disk images (closes the TEST-002 / TEST-003 tool-applicability gap so all five orchestrators generalize)
- Multi-source correlation — cross-reference memory and disk findings from the same system, flag discrepancies
- Cryptographic row chaining on the audit DB — move append-only from application-layer enforcement to a tamper-evident hash chain
- Benchmark expansion — more public datasets, more orchestrator × dataset cells
- Analyst training mode — agent explains each tool choice and what it expected to find
Log in or sign up for Devpost to join the conversation.