SentinelDS

• 

  Project Title

• 

  Dashboard - Observability

• 

  Describes how the Sentinel service protects the agents

• 

  Architecture

• 

  Dashboard - Agent Console

• 

  Dynatrace Dashboard

Inspiration

AI agents are about to run production workloads: building models, ingesting data, fetching research, with real tool access and real consequences. But today, almost no one is watching what those agents actually do. A poisoned CSV reaches training without raising an alarm. A malicious webpage hijacks a research agent mid-summary. The model stays well-behaved; the architecture is wide open.

We asked: what if the workspace itself had an immune system? One that observed every agent action, learned normal behavior, and halted the next dangerous tool call before damage spread. Not by tuning the model, but by defending at the architectural layer where the actual exposure lives. We combined OpenTelemetry's full-pipeline instrumentation with Dynatrace's Davis AI anomaly engine and a deterministic Sentinel Agent to build a runtime security layer for agentic data-science.

What it does

SentinelDS is an agentic data-science workspace defended by Dynatrace. Three Gemini-powered specialist agents collaborate on a real mission: (eg. drowsiness-detection)

• Research Agent: surveys papers and blog posts; summarizes approaches.
• Feature Engineering Agent: pulls datasets, profiles them, builds features (metrics, find correlations, find patterns).
• Modelling Agent: trains XGBoost and CatBoost classifiers with Optuna tuning; reports accuracy and false-positive rate.

Every LLM call, tool call, and dataset I/O is wrapped in an OpenTelemetry span and shipped to Dynatrace SaaS over OTLP. Davis AI auto-baselines the workspace and raises Problems on anomalous behavior. A separate Sentinel Agent queries Dynatrace over the MCP before each risky tool call and returns one of three verdicts (ALLOW / WARN / HALT), a deterministic, fail-closed gate that catches what model-layer safety tuning cannot.

The demo shows two attacks, end-to-end, with detection and Sentinel response:

How we built it

OpenTelemetry instrumentation layer: every agent tool call is wrapped with a @trace_tool decorator that emits a span with structured attributes (tool.name, tool.args.hash, dataset.uri, egress.host, agent.name). LLM calls are auto-instrumented via OpenLLMetry/Traceloop. Spans ship to Dynatrace over OTLP/HTTP:

@trace_tool("fetch_url")
@sentinel_gate
async def fetch_url(url: str, *, session: SentinelSession) -> str:
    with tool_span("fetch_url", {"egress.host": urlparse(url).netloc}):
        content = await _http_get(url)
    injection_detector.scan(content, session=session)
    return content

Injection detector (A1 path): on every fetch_url call, a local regex + heuristic scanner checks the fetched content for injection signatures (imperative override patterns, exfil instructions, structural anomalies). When triggered, it flips SentinelSession.compromised = True and POSTs a custom event to Dynatrace. The decision is O(1) local, with no MCP round-trip in the critical path:

@dataclass
class SentinelSession:
    session_id: str
    compromised: bool = False
    halt_reason: str | None = None

Sentinel pre-flight (A2 path): before the Modelling Agent calls its training tool, preflight() queries the Dynatrace Remote MCP for open Problems on the workspace entity. If Davis AI has flagged dataset drift, the pre-flight returns HALT and the orchestrator skips the tool call:

async def preflight(tool_name: str, session: SentinelSession) -> Decision:
    if session.compromised:
        return Decision.HALT
    if is_risky(tool_name):
        problems = await dynatrace_mcp.list_open_problems(entity=WORKSPACE_ENTITY)
        if problems:
            return Decision.HALT
    return Decision.ALLOW

Two detection paths, same enforcement: A1 uses a local flag (deterministic, O(1)); A2 uses a Davis AI Problem surfaced over MCP. Both terminate at the same @sentinel_gate decorator that raises PermissionError on HALT:

A1: fetch_url → injection_detector → session.compromised=True → @sentinel_gate raises
A2: csv_ingest → OTel metric emit → Davis AI → MCP Problem → preflight() HALT → @sentinel_gate raises

ADK multi-agent orchestration: three sub-agents are composed into a SequentialAgent via Google ADK. The Sentinel is a cross-cutting supervisor, not a fourth agent in the sequence:

root_agent = SequentialAgent(
    name="sentinelds_pipeline",
    sub_agents=[research_agent, feature_agent, modeling_agent],
)

A2A deployment: each agent is also packaged as an Agent-to-Agent (A2A) service with its own Dockerfile and deployment script, enabling Cloud Run deployment for the production path.

Challenges we ran into

• Indirect injection subtlety: crude IGNORE PREVIOUS INSTRUCTIONS payloads are caught by Gemini's safety filters. A convincing A1 payload must look like legitimate research metadata: malicious URLs embedded as supplementary_data_url or references[] fields, which the agent follows because its own prompt instructs it to enrich findings via cited sources. Getting this to reliably work against Gemini 2.5 Flash without triggering refusals required multiple payload iterations.
• Two detection paths with different latencies: A1's local injection detector is O(1); A2's Davis AI problem detection requires a metric baseline to form (minutes to hours of normal traffic). For the demo, we seed a baseline snapshot so Davis has a reference distribution before the attack runs. This forced us to build a seed_baseline.py script and explicit baseline_snapshots/ management.
• Fail-closed under MCP unavailability: the Sentinel must HALT on MCP errors for A2, not silently ALLOW. Implementing correct fallback semantics (distinguish "no problems found" from "MCP unreachable") required careful exception handling in dynatrace_mcp.py.
• OpenTelemetry + Google GenAI SDK wiring: OpenLLMetry auto-instruments the GenAI SDK, but the TracerProvider must be initialized before the SDK is imported. Import order bugs caused spans to go missing silently; a single-init guard in instrumentation.py resolved this.
• google.genai / google.adk schema split: running agents through the get_fast_api_app(web=True) interface switches the SDK into Gemini Enterprise Agent Platform mode, which enforces a strict conversation-history schema. Any part_metadata field — valid in raw stateless Gemini Developer API calls — causes a hard ValueError when the ADK event loop wraps it in the enterprise tracking envelope. The ADK framework also appends its own routing variables to the parts collection, so if any model config or middleware hook injects part_metadata before the payload leaves the container, the Enterprise validation layer rejects it. The fix was to strip part_metadata blocks from every part right before the message is sent to Google's servers, keeping the raw-API and ADK-UI paths compatible without forking the agent code.

Accomplishments that we're proud of

• Two attacks demonstrated end-to-end against a hardened Gemini model, exploiting the agent architecture rather than the model's alignment
• A deterministic, fail-closed Sentinel gate that halts tool execution in O(1) (A1) or via a Davis AI MCP query (A2), not an LLM judgment
• Full OTel instrumentation across LLM calls, tool calls, dataset I/O, and MCP calls, all visible as structured traces in Dynatrace
• The four-phase defense loop (EMIT → DETECT → DECIDE → ENFORCE) working end-to-end for both attack classes
• @sentinel_gate and @trace_tool decorators that compose cleanly on any ADK tool function without changing its signature
• Mapping both attacks to MITRE ATLAS (AML.T0051, AML.T0020) and SANS AISMM Stage 3 → 4 capabilities

What we learned

The elegance of the Confused Deputy framing: a well-aligned model in a well-designed agent is still vulnerable when its legitimate permissions are weaponized by attacker-controlled content. The Research Agent does not misbehave — it follows its own instructions exactly. The attack works precisely because the agent was told to fetch and enrich from cited sources. Model-layer safety tuning is necessary but insufficient; the attack surface is the architecture.

Davis AI's value for A2 is not just detection; it's the baseline. Without a reference distribution for dataset.stats.label_flip_rate, there is no anomaly to detect. Investing in metric seeding (establishing what "normal" looks like) is as important as the anomaly rule itself.

The two-path defense (local flag for A1, MCP query for A2) taught us that decision latency is a first-class constraint: a Sentinel that adds a 500ms MCP round-trip to every fetch_url call would be rejected by any real team. The right answer is to match detection mechanism to attack speed: local for content-layer attacks, async observability for data-pipeline attacks.

What's next for SentinelDS

• Production A2A deployment on Cloud Run with per-agent GCP service accounts and Dynatrace API tokens (non-human identity lifecycle)
• Davis AI custom anomaly detectors for the five non-demoed threats: tool/MCP abuse, model supply-chain poisoning, resource exhaustion, secret exfiltration, recursive agent loops
• Sentinel policy externalized from code into a Dynatrace-managed configuration, enabling policy updates without redeployment
• Quantitative risk methodology (FAIR-based) for Sentinel decision thresholds, replacing hardcoded is_risky() with a probability-of-harm weighting
• MLSecOps training-data validation as a production pipeline stage (not a compensating control)
• SANS AISMM Stage 5 trajectory: self-improving defenses that feed Sentinel HALT events back into Davis AI as labeled anomalies, closing the detection feedback loop.

Built With

• a2a
• catboost
• cloud-run
• davis-ai
• docker
• dynatrace
• dynatrace-mcp
• gemini
• google-adk
• mitre-atlas
• opentelemetry
• optuna
• otlp
• pandas
• python
• sans-aismm
• scikit-learn
• traceloop
• vertex-ai
• xgboost