🛡️ LLM Agent Security Testbed

Modern LLM-powered agents execute privileged actions: querying internal databases, reading file systems, and interacting with backend APIs. Every action is a boundary where an attacker’s prompt can trigger unauthorized execution.

⚠️ Key Architectural Takeaway:
The vulnerability rarely lives inside the LLM weights alone. It thrives in the trust boundary between the model's intent request and the application backend executing it without validation.

Much like SQL Injection stemmed from unparameterized string concatenation rather than the database engine itself, LLM Confused-Deputy Flaws occur when application code blindly trusts an agent's tool arguments.

flowchart TD
    subgraph Adversary["Adversarial Inputs"]
        A1["Direct Override Prompt"]
        A2["Role Authority Claim"]
        A3["Indirect Data Injection"]
        A4["Boundary Bypass Hinting"]
    end

    subgraph AgenticLoop["LLM Agent Runtime (Gemini 3.6 Flash)"]
        LLM["Agent Reasoning Core"]
        FC["Tool Call Declaration: get_user(username)"]
    end

    subgraph DefenseLayer["Evaluation Defense Layers"]
        direction TB
        subgraph Naive["Naive Backend (Insecure)"]
            N1["Zero Validation"]
            N2["Returns ALL fields (inc. Password)"]
            N3["Ignores restricted=True"]
        end
        
        subgraph Hardened["Hardened Backend (Secure)"]
            H1["Access Control Enforcement"]
            H2["Refuses restricted=True rows"]
            H3["Password Field Stripped by Design"]
        end
    end

    subgraph Evaluation["Inspection & Grading Engine"]
        G1["Tool Output Interception"]
        G2["Target Secret Inspection ('s3cr3t-fake-admin-pw')"]
        G3["Verdict: LEAKED | BLOCKED | UNCLEAR"]
    end

    Adversary --> LLM
    LLM --> FC
    FC -.->|Test Run A| Naive
    FC -.->|Test Run B| Hardened
    Naive --> G1
    Hardened --> G1
    G1 --> G2 --> G3

Both tools expose an identical function signature to the LLM agent (get_user(username: str)). The model cannot determine which tool version it is interacting with.

The testbed exercises 5 key vulnerability categories defined in attacks.py:

The environment is backed by pure, deterministic Python dataclasses in testbed/fake_data.py:

💡 Why Alice's bio is poisoned: This models a realistic indirect prompt injection scenario where an attacker doesn't need elevated privileges. They only need to control data a tool retrieves (e.g. public profile bio), waiting for an agent to read it during a routine lookup.

Grading free-text LLM responses is fundamentally non-deterministic. A model might hedge, partially disclose information, or decline to call a tool entirely.

Distinguishing UNCLEAR from BLOCKED is crucial: it prevents falsely claiming that a tool backend is secure when the attack simply failed to reach the tool layer.

llm-agent-testbed/
├── testbed/
│   ├── __init__.py               # Package initializer
│   ├── attacks.py                # Structured attack checklist (5 categories)
│   ├── display.py                # Formatted terminal display & verdict styling
│   ├── fake_data.py              # Mock backend storage & seeded injection payloads
│   ├── models.py                 # Pure dataclass shapes: FakeUser, AttackAttempt, AttackResult
│   ├── runner.py                 # Multi-turn attack execution engine & grading logic
│   ├── tools_hardened.py         # Hardened implementation with boundary defenses
│   └── tools_naive.py            # Baseline unvalidated lookup implementation
├── diagrams/
│   ├── 01-architecture-overview.svg
│   ├── 02-naive-vs-hardened-flow.svg
│   ├── 03-attack1-direct-override.svg
│   ├── 04-attack2-role-authority.svg
│   ├── 05-attack3-indirect-injection.svg
│   ├── 06-attack4-boundary-bypass.svg
│   ├── 07-attack5-chained-request.svg
│   ├── 08-summary-table.svg
│   └── 09-summary-chart.png
├── .env                          # Local API keys (ignored by git)
├── .gitignore                    # Standard exclusion rules
├── BUILD-JOURNAL.md              # Engineering decision log & architectural evolution
├── LICENSE                       # MIT License
├── NOTES.md                      # Project notes & phase progress tracker
├── PHASE-6-REPORT.md             # In-depth test report, API quotas & failure analysis
├── README.md                     # Main project overview & documentation
├── V1-RESULTS.md                 # Full detailed walk-through of all 5 attack results
├── pyproject.toml                # Project metadata & dependencies
└── uv.lock                       # Deterministic dependency lockfile

Clone the repository and set up dependencies with uv:

git clone https://github.com/pie-script/llm-agent-testbed.git
cd llm-agent-testbed
uv sync

Create a .env file in the root directory:

GEMINI_API_KEY="your_gemini_api_key_here"

Execute attacks against either tool version through the test harness:

# Run Attack 1 against the Naive tool (vulnerable baseline)
uv run python -c "from testbed.attacks import ATTACKS; from testbed.runner import run_attack; print(run_attack(ATTACKS[0], 'naive'))"

# Run Attack 1 against the Hardened tool (access-controlled defense)
uv run python -c "from testbed.attacks import ATTACKS; from testbed.runner import run_attack; print(run_attack(ATTACKS[0], 'hardened'))"

• 📖 V1-RESULTS.md — Comprehensive breakdown of all 5 attacks with result diagrams, prompt iterations, and security takeaways.
• 🔬 PHASE-6-REPORT.md — In-depth report on test harness validation, API constraints, and model behavior.
• 📓 BUILD-JOURNAL.md — Step-by-step engineering decision log and thought process.

• Mocked Backend by Design: Pure Python dataclasses avoid complex Docker/sandboxing setups to keep the focus strictly on agentic tool security.
• Prompt Testing vs. Model Internals: Evaluates external prompt behavior and tool authorization, not model weight fine-tuning.
• Empirical Exploration: Serves as a disciplined educational prototype rather than a heavy enterprise red-teaming scanner.

• Phase 0 — Verified Gemini 3.6 Flash function calling loop end-to-end.
• Phase 1 — Defined attack success metrics, ground-truth secrets, and mocked backend scope.
• Phase 2 — Implemented immutable data models (FakeUser, AttackAttempt, AttackResult).
• Phase 3 — Formulated naive and hardened defense rules.
• Phase 4 — Wired tools to live LLM API loop and confirmed baseline behavior.
• Phase 5 — Authored multi-category attack suite with planted indirect injection vectors.
• Phase 6 — Automated batch runner execution, multi-turn support, and response grading.
• Phase 7 — Spot-checking ambiguous outcomes (unclear classification review).
• Phase 8 — Generated comprehensive evaluation writeups, summary tables, and visual charts.