Skip to content

feat: three-mode safety policy (strict / balanced / autonomous) with native shell classification - #3835

Merged
dgageot merged 1 commit into
docker:mainfrom
trungutt:feat/safety-modes
Jul 27, 2026
Merged

feat: three-mode safety policy (strict / balanced / autonomous) with native shell classification#3835
dgageot merged 1 commit into
docker:mainfrom
trungutt:feat/safety-modes

Conversation

@trungutt

Copy link
Copy Markdown
Contributor

Why

Tool approval today is a coarse two-option choice. When a call needs confirmation, the user can approve it once — or approve everything for the rest of the session. Prompt fatigue pushes people to the blanket option without them realising it also covers destructive commands (rm -rf, docker volume rm, …). There is no middle ground, and no obvious way back.

The shell classifier (safer: true) was also entangled with the approval verdict: it read the session policy and decided whether to ask, which made its judgement impossible to reuse and every new approval behavior an edit to the classifier.

What this PR does

Introduce three explicit safety modes and split the classifier into a pure labeller so the mode is what actually gates the call:

               safe     destructive   unknown
  strict       ask      ask           ask
  balanced     ALLOW    ask           ask
  autonomous   ALLOW    ALLOW         ALLOW
  • strict — ask before every tool call, read-only ones included.
  • balanced — the new middle tier: positively-recognised safe calls (safe-listed shell commands, read-only-annotated tools) run silently; destructive and unrecognised calls still ask.
  • autonomous — the legacy --yolo: everything runs.

Every call gets a safe / destructive / unknown label from a new dependency-free pkg/safety package (shell commands via the pattern taxonomy, other tools via their MCP annotations), and the pipeline becomes:

preempt hooks  →  custom rules (deny/allow/ask)  →  mode × label  →  default hooks  →  ask user

Custom rules always win over the mode — a deny: rule blocks even autonomous, an allow: rule silences even strict. One deliberate asymmetry: session-scoped ask: rules (direct user intent) always prompt, while agent-author ask: rules yield to a user-chosen auto-approving mode.

Escalation is guided and reversible: confirmation prompts offer the next rung ([B]alanced on safe calls, approve-all on risky ones), a new --safety flag and the existing PATCH /sessions/:id/safety-policy endpoint set or undo the mode at any time, and downgrading genuinely revokes the blanket approval.

Compatibility

  • Legacy policy values (unsafe / safer / safe-auto) and resume verbs (approve-session, approve-safe, approve-safer) are accepted and normalized, so existing API consumers keep working. One deliberate nuance: safer maps to balanced, which is narrower — old safer waved unrecognised commands through silently, balanced prompts for them. The mapping errs toward asking; it never widens.
  • Sessions that never choose a mode keep the historical default: read-only tools auto-approve, everything else asks. Explicit strict is stricter than that on purpose.
  • The shell toolset's safer: flag stays parseable but is ignored — classification now always runs. YAMLs pinning the safer_shell builtin keep working; it degrades to a metadata-only labeller.
  • Confirmation events and the on_tool_approval_decision hook now carry safety_label, so host applications can render risk tiers without re-implementing the taxonomy.
  • Deliberate behavior change: the safety mode is session-scoped state, so resuming a session stored as autonomous keeps it autonomous without re-passing --yolo. On main, blanket approval had to be re-requested on every run. An explicit --safety on resume still overrides the stored mode.

Supersedes #3806 (same goal; this version keeps the decision table authoritative — no hidden read-only fast path under explicit strict — moves the classifier out of the hook lane into a runtime component, and preserves wire compatibility for configs, resume verbs, and persisted sessions).

…native shell classification

Tool approval today is a coarse two-option choice: approve a call once,
or approve everything for the rest of the session. Prompt fatigue
pushes people to the blanket option without them realising it also
covers destructive commands. There is no middle ground, and no obvious
way back. The shell classifier (safer: true) was also entangled with
the approval verdict, which made its judgement impossible to reuse.

Introduce three explicit safety modes and split the classifier into a
pure labeller so the mode is what actually gates the call:

               safe     destructive   unknown
  strict       ask      ask           ask
  balanced     ALLOW    ask           ask
  autonomous   ALLOW    ALLOW         ALLOW

Every call gets a safe / destructive / unknown label from a new
dependency-free pkg/safety package (shell commands via the pattern
taxonomy, other tools via their MCP annotations), and the pipeline
becomes: preempt hooks -> custom rules -> mode x label -> default
hooks -> ask user. Custom rules always win over the mode; session
ask rules always prompt while agent-author ask rules yield to a
user-chosen auto-approving mode.

Escalation is guided and reversible: confirmation prompts offer the
next rung, a new --safety flag and the PATCH safety-policy endpoint
set or undo the mode at any time, and downgrading genuinely revokes
the blanket approval (the yolo toggle restores the pre-escalation
mode, surviving branch and fork).

Compatibility: legacy policy values (unsafe / safer / safe-auto) and
resume verbs are accepted and normalized; safer maps to balanced,
which is deliberately narrower (unknown commands now prompt). Sessions
that never choose a mode keep the historical default. The shell
toolset's safer: flag stays parseable but is ignored; the safer_shell
builtin degrades to a metadata-only labeller and warns at config build
time. Confirmation events and the on_tool_approval_decision hook now
carry safety_label. Resuming a session stored as autonomous keeps
autonomous without re-passing --yolo; an explicit --safety on resume
overrides.

@docker-agent docker-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessment: 🟢 APPROVE

The three-mode safety policy implementation (strict / balanced / autonomous) is well-structured. The new pkg/safety package provides a clean, dependency-free classifier with thread-safe pattern compilation via sync.OnceValues. Legacy resume verbs (approve-session, approve-safe, approve-safer) are properly normalized through NormalizeResumeType before the policy switch, so backward compatibility is maintained. The safer_shell deprecation path is intentional per its documentation. No bugs introduced by this PR were confirmed.

@aheritier aheritier added area/a2a Agent-to-Agent protocol, A2A server, inter-agent communication area/agent For work that has to do with the general agent loop/agentic features of the app area/api For features/issues/fixes related to the usage of the cagent API area/cli CLI commands, flags, output formatting area/config For configuration parsing, YAML, environment variables area/core Core agent runtime, session management area/docs Documentation changes area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection area/sessions For features/issues/fixes related to session lifecycle (resume, persistence, export) area/testing Test infrastructure, CI/CD, test runners, evaluation area/tui For features/issues/fixes related to the TUI kind/feat PR adds a new feature (maps to feat:). Use on PRs only. labels Jul 27, 2026
docker-agent

This comment was marked as low quality.

@trungutt
trungutt marked this pull request as ready for review July 27, 2026 08:38
@trungutt
trungutt requested a review from a team as a code owner July 27, 2026 08:38
docker-agent

This comment was marked as low quality.

@docker-agent docker-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessment: 🟢 APPROVE

The three-mode safety policy (strict / balanced / autonomous) is well-implemented. The (mode × label) decision table in pkg/runtime/toolexec/permissions.go is correct and exhaustively tested. The classifier in pkg/safety/ is conservative by design (unknown = ClassUnknown, compound shell commands blocked, deny-flag list). Session state transitions are mutex-guarded. Legacy wire compatibility (unsafe/safer/safe-auto normalization) is correctly implemented throughout the pipeline.

The permission pipeline change — where team-tier ForceAsk rules yield to a user-chosen Autonomous mode while session-tier ForceAsk always prompts — is consistent with the pre-PR behavior (old yolo bypassed all ForceAsk), and the PR actually improves it by preserving session-tier ForceAsk even under Autonomous. This design decision is correctly documented in code comments.

@Sayt-0

Sayt-0 commented Jul 27, 2026

Copy link
Copy Markdown
Member

Don't merge yet there are still a few bugs. I am currently correcting it

@dgageot
dgageot merged commit 1ed0590 into docker:main Jul 27, 2026
26 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/a2a Agent-to-Agent protocol, A2A server, inter-agent communication area/agent For work that has to do with the general agent loop/agentic features of the app area/api For features/issues/fixes related to the usage of the cagent API area/cli CLI commands, flags, output formatting area/config For configuration parsing, YAML, environment variables area/core Core agent runtime, session management area/docs Documentation changes area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection area/sessions For features/issues/fixes related to session lifecycle (resume, persistence, export) area/testing Test infrastructure, CI/CD, test runners, evaluation area/tui For features/issues/fixes related to the TUI kind/feat PR adds a new feature (maps to feat:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants