Agent Canary

CI for AI-agent security: Canary attacks every candidate release, compares it with an accepted baseline, and blocks unsafe PRs with reproducible evidence.

Comment

Inspiration

Traditional CI is excellent at catching code regressions, but it cannot tell us when an AI agent becomes less secure.

A small prompt, tool-permission, or retrieval change can make an agent reveal sensitive information or perform an unsafe action while every normal unit test still passes. We wanted to build the missing security layer for AI-agent releases.

Agent Canary evolved from the existing Canary red-team engine into a CI security gate for AI-agent behavior.

What we built

Canary attacks an AI-agent candidate over HTTP before it ships. It then replays equivalent attacks against an accepted baseline and compares the actual behavior.

The result is simple:

  • PASS — no new security regression
  • WARN — non-blocking issue or incomplete coverage
  • BLOCK — a new critical or high-severity regression

The developer does not need to understand every attack technique. They receive a clear release decision with the exact prompts, responses, evaluator reasoning, evidence, and telemetry behind it.

For the demonstration, CompanyAgent is deployed separately on AWS. The accepted baseline and vulnerable candidate run side by side, allowing Canary to compare both versions during a real pull-request workflow.

How it works

A pull request triggers the GitHub Action. The action submits the candidate release to the Canary FastAPI backend.

Canary creates a durable release job and runs a LangGraph red-team workflow:

  1. An LLM-backed Strategist selects and plans attack techniques.
  2. Parallel LLM-backed Attackers generate adversarial prompts.
  3. Canary sends those prompts to the target agent through an HTTP adapter.
  4. The target response is recorded with status, latency, hashes, and evidence.
  5. An LLM-backed Evaluator judges whether the attack actually succeeded.
  6. Deterministic detectors provide concrete signals such as PII or credential exposure.
  7. The same attack cases are replayed against the accepted baseline.
  8. Canary classifies each case as regression, known, resolved, or clean.
  9. The gate produces PASS, WARN, or BLOCK.
  10. GitHub receives the security check and job summary.

The attackers never declare their own success. The evaluator is the authority.

All orchestration roles—Strategist, Attacker, Evaluator, and Reporter—are LLM-backed through Backboard.io using the Luna 5.6 model route. The system does not use hardcoded vulnerability results or deterministic attack payloads.

What we learned

The biggest lesson was that AI-agent security must be evaluated behaviorally.

Comparing finding counts is not enough. A vulnerability that already existed in the baseline should not be treated as a newly introduced regression. This led us to build stable attack-case identities and paired baseline-versus-candidate execution records.

We also learned that the evidence is as important as the verdict. A useful security check must show what Canary asked, what the agent returned, which detector signals fired, how the evaluator reached its conclusion, and how confident it was.

Telemetry became essential as well. Every model call records provider, model, token counts, latency, retry state, status, hashes, and—when authorized—the prompt and completion text.

Challenges

The hardest part was evolving an existing autonomous red-team engine without replacing its working capabilities. We had to connect the original campaign system to a new project, release, baseline, and regression model while preserving parallel LangGraph execution and HTTP target support.

Other challenges included:

  • pairing newly generated LLM attack cases across two releases;
  • preventing false PASS results when an evaluator call fails;
  • validating target ownership and rejecting unsafe SSRF destinations;
  • keeping CI credentials away from browser JavaScript;
  • making the release job survive beyond a single API request;
  • controlling LLM cost and latency on small AWS instances;
  • presenting complex security evidence through a simple developer-facing UI.

The final system is intentionally focused:

PR → attack → compare → evidence → decision.

Limitations

The hackathon demo uses a pre-deployed candidate CompanyAgent endpoint on AWS rather than creating a new preview container for every PR. Canary still evaluates the real deployed candidate over HTTP and compares it with the trusted baseline.

A production deployment would add automated preview provisioning, HTTPS termination, stronger worker infrastructure, and centralized persistent storage.

Why it matters

Agent Canary evaluates every candidate version pushed to a pull request and compares it with the accepted baseline. It detects vulnerabilities introduced by that specific change, while distinguishing them from vulnerabilities that already existed.

It answers one question before release:

Is the AI agent we are about to ship less secure than the one we already trust?

That is the security regression traditional CI cannot see.

Built With

Share this project:

Updates