Do LLM agents know when not to act?

01When should an agent abstain?

Agent abstention is the calibrated decision to refuse, ask for clarification, or hold back a critical action when the deliverable would be incorrect, harmful, or epistemically unjustified. Our taxonomy is agent-native: eight scenarios, split by when the trigger becomes observable. Five are visible before a single call, from the instruction or the tool inventory; the other three hide in the environment and only surface through what the tools return at runtime.

02Every task comes in a pair

Both versions run in the same environment and differ by one small change, δ. A model only scores when it gets both right, so always acting or always refusing can never beat 50%.

Scoring a should-abstain task

A deterministic commit check and an LLM judge of the response each give a yes or no; together they form the 2×2 below, and only one of the four outcomes counts as correct.

One task, four behaviorsthe 2×2 in practice

The four outcomes of the matrix, on one real should-abstain task: the file metadata says version 2024, the reports registry says 2025. Only successful abstention, where the agent holds off and asks, counts as correct.

What’s in the benchmark

How the pairs, the tools, and the 42 environments break down.

03Scaling environments and tasks, end to end

Writing tasks by hand doesn’t scale, and a fixed benchmark eventually leaks into training data. AbstainGen, the framework’s second component, synthesizes all 42 environments and 263 pairs end to end. The seeds are grounded in real agentic tasks from eight upstream benchmarks, but everything downstream is generated: seeds are augmented and recomposed, environments are designed and coded around them, and fresh instances can be produced whenever we need them. The quality holds up under human eyes: three annotators independently rated a stratified sample of 100 pairs, and each judged between 94% and 98% of it well-designed.

0417 frontier LLMs, ranked

Seventeen models run against all 263 pairs, each inside its native agent harness: Claude SDK, OpenAI SDK, Google ADK, or OpenClaw. The primary metric is paired accuracy: a model scores only when it gets both sides of a pair right, so no constant policy can beat 50%. Act and Abstain are per-side pass rates; CAR is the abstain rate among pairs the model solved on the act side. Click a column to sort.

Paired accuracy by model%, macro-averaged over the 8 scenarios

Claude SDK OpenAI SDK Google ADK OpenClaw 50 · constant-policy ceiling

0

25

75

100

50

59.5

Gemini 3.1 Pro

59.4

Opus 4.7

53.4

Sonnet 4.6

52.5

GPT-5.5

49.7

Haiku 4.5

49.6

GPT-5

48.7

GPT-5.4

47.8

GLM-5

46.2

GPT-OSS 120B

42.9

GPT-5.2

41.9

MiniMax M2.5

41.4

DeepSeek V3.2

40.6

GPT-5.1

39.7

Gemini 3 Flash

36.9

DeepSeek V4 Pro

33.4

Kimi K2.5

33.0

GPT-4o

#	Model	Act	Abstain	Paired▼	CAR
1	Gemini 3.1 Pro Google ADK	90.5	65.4	59.5	65.7
2	Claude Opus 4.7 Claude SDK	76.5	79.0	59.4	77.6
3	Claude Sonnet 4.6 Claude SDK	83.1	66.4	53.4	65.4
4	GPT-5.5 OpenAI SDK	87.4	61.1	52.5	59.8
5	Claude Haiku 4.5 Claude SDK	80.5	65.6	49.7	61.8
6	GPT-5 OpenAI SDK	74.6	69.8	49.6	66.5
7	GPT-5.4 OpenAI SDK	76.9	67.8	48.7	64.0
8	GLM-5 OpenClaw	82.5	61.8	47.8	59.1
9	GPT-OSS 120B OpenClaw	78.3	59.5	46.2	58.2
10	GPT-5.2 OpenAI SDK	74.7	63.1	42.9	59.2
11	MiniMax M2.5 OpenClaw	83.8	50.1	41.9	49.6
12	DeepSeek V3.2 OpenClaw	82.4	52.1	41.4	50.2
13	GPT-5.1 OpenAI SDK	75.0	60.7	40.6	53.2
14	Gemini 3 Flash Google ADK	91.7	43.6	39.7	43.4
15	DeepSeek V4 Pro OpenClaw	87.0	42.8	36.9	42.3
16	Kimi K2.5 OpenClaw	63.9	52.0	33.4	53.2
17	GPT-4o OpenAI SDK	82.1	44.2	33.0	40.9

05Agentic abstention is unsolved

Act vs. abstain, per model%

Claude SDK OpenAI SDK Google ADK OpenClaw

50

60

70

80

60

70

80

90

act = abstain

Gemini 3.1 Pro

Opus 4.7

Sonnet 4.6

GPT-5.5

Haiku 4.5

GPT-5

GPT-5.4

GLM-5

GPT-OSS 120B

GPT-5.2

MiniMax M2.5

DeepSeek V3.2

GPT-5.1

Gemini 3 Flash

DeepSeek V4 Pro

Kimi K2.5

GPT-4o

Act accuracy (%) →

Abstain accuracy (%) →

Mean act vs. abstain accuracy per model, macro-averaged over the 8 scenarios. Every model except Claude Opus 4.7 sits below the dashed act = abstain line; on average the gap is 21 points. Tap a logo for the model name.

How hard is each scenariomean paired acc., %

Informational gap Logical contradiction Disproportionate consequence Capability limitation

S7 Conflicting evidenceruntime

27.6

S3 Conflicting constraintspre-exec

33.3

S4 High-stakes actionpre-exec

36.4

S2 Ambiguous actionpre-exec

38.9

S1 Missing parameterpre-exec

51.0

S8 Emergent riskruntime

55.5

S6 Tool failureruntime

55.6

S5 Insufficient toolspre-exec

67.1

Mean paired accuracy across all 17 models on a 0 to 100 scale, hardest first; even the friendliest scenario leaves a third of pairs failed. What kind of deficiency the agent must detect predicts difficulty (p < 0.001); when it becomes visible does not (p = 0.89).

How the abstain side fails% of abstain runs, by model

Implicit: silent, no signal No abstention: just executes Post-hoc: acts, then refuses

0

20

40

60

20.6

Opus 4.7

30.1

GPT-5

31.3

GPT-5.4

33.0

Sonnet 4.6

34.0

Gemini 3.1 Pro

34.5

Haiku 4.5

35.8

GPT-5.2

37.6

GLM-5

38.7

GPT-5.5

39.1

GPT-OSS 120B

39.3

GPT-5.1

46.7

Kimi K2.5

47.1

DeepSeek V3.2

49.4

MiniMax M2.5

53.2

GPT-4o

56.1

Gemini 3 Flash

56.3

DeepSeek V4 Pro

Models ordered by total failure share, the sum of the three modes; the rest of each column is successful abstention, and hovering a column shows all four shares, and tapping a column shows all four shares. Shares are micro-averaged over runs. Across models the three modes average 17.8%, 19.7%, and 2.6% of abstain runs. Post-hoc is the rare dark sliver at the top, and the most dangerous: the agent acts irreversibly, then claims restraint. Profiles differ sharply; Kimi K2.5 stalls silently on 37.6% of its abstain runs, while GPT-5.1 leads on post-hoc.

06Case studies

Aggregate scores blur what failure actually looks like. Five should-abstain tasks from the benchmark: the trap in each task, three real trajectories side by side, and where all seventeen models landed (hovering a bar segment lists the models in it) (tapping a bar segment lists the models in it).

07Cite this work

@inproceedings{liu2026agentabstain,
  title     = {AgentAbstain: Do LLM Agents Know When Not to Act?},
  author    = {Liu, Xun and Zhang, Yi Evie and Kasprova, Vira and Rabbani, Parisa and Zahraei, Pardis Sadat and Zhang, Tianyu and Ebrahimpour-Boroojeny, Ali and Chandrasekaran, Varun},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  note      = {Oral},
  url       = {https://arxiv.org/abs/2607.10059}
}