Skip to content

fix(eval): preserve budget termination outcomes - #3862

Merged
dgageot merged 1 commit into
mainfrom
fix/3845-preserve-budget-termination
Jul 28, 2026
Merged

fix(eval): preserve budget termination outcomes#3862
dgageot merged 1 commit into
mainfrom
fix/3845-preserve-budget-termination

Conversation

@Sayt-0

@Sayt-0 Sayt-0 commented Jul 28, 2026

Copy link
Copy Markdown
Member

Summary

  • preserve native budget_exceeded outcomes as allow-listed evaluation termination metadata
  • keep the chronological termination marker and exactly one associated assistant stop message across transcripts, SQLite, and sessions JSON
  • expose optional eval_result.termination data without changing scoring or error semantics
  • keep message_added payloads private by carrying the budget stop through a dedicated narrow runtime DTO
  • document the result contract and backward-compatible interpretation

Closes #3845

Acceptance criteria

Expectation Implementation
Native event parsing budget_exceeded maps to reason, budget, limit, used, max, config_path, and message only
Safe metadata copied strings are UTF-8 normalized, control-filtered, bounded, and omitted when unusable
Chronology the termination item is emitted at the event position, immediately followed by its assistant stop message
Duplicate prevention repeated equivalent budget events do not duplicate the marker or stop message
Narrow message handling all message_added events remain payload-free and ignored by evaluation reconstruction; only the dedicated budget stop_message DTO is retained
SQLite persistence termination items reuse the existing session_items.message_json storage with item_type = termination, requiring no migration
Sessions JSON termination marker, assistant stop message, and optional eval_result.termination survive serialization and reload
Result semantics termination remains informational and does not alter passed, failures, errors, or scoring inputs
Compatibility old sessions and results without termination data continue to load unchanged
Isolation normal, error, ordinary stream-stop, and foreign-session events do not receive or disturb budget termination data

Validation

  • env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy go test ./...
  • go build ./...
  • go vet ./pkg/runtime ./pkg/evaluation ./pkg/session
  • golangci-lint run ./pkg/runtime/... ./pkg/evaluation/... ./pkg/session/...
  • go run ./lint .
  • npx --yes markdownlint-cli2@0.22.1 configuration/budget/index.md from docs/
  • ./scripts/docs-check-canonical.sh
  • git diff --check

Manual validation

Not run. A live budget-triggered evaluation would require a configured model/provider. Deterministic runtime, parser, chronology, persistence, compatibility, isolation, and security coverage is included in the automated tests.

@Sayt-0
Sayt-0 requested a review from a team as a code owner July 28, 2026 13:48
@aheritier aheritier added area/docs Documentation changes area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection area/sessions For features/issues/fixes related to session lifecycle (resume, persistence, export) kind/fix PR fixes a bug (maps to fix:). Use on PRs only. labels Jul 28, 2026
@dgageot
dgageot merged commit 8af2fe1 into main Jul 28, 2026
20 checks passed
@dgageot
dgageot deleted the fix/3845-preserve-budget-termination branch July 28, 2026 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation changes area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection area/sessions For features/issues/fixes related to session lifecycle (resume, persistence, export) kind/fix PR fixes a bug (maps to fix:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preserve native run-budget termination in evaluation parsing, transcripts, and stored results

4 participants