Evaluations

Repeatable, scored test suites for an agent: a dataset of cases, scorers that grade the output, and runs that produce a pass/fail verdict.

Overview

A dataset holds test cases, an eval binds an agent to a dataset and a list of scorers, and a run executes the real agent against every case and scores the outputs. Where traces and guardrails deal with runs that already happened or are happening, an evaluation answers whether a change to the agent — a reworded instruction, a swapped model, a new tool — improved the distribution of runs. Evaluations is the foundation of the ratchet layer described in The Layers of an Agent System.

The module follows SOAT's engine & algorithms pattern: the engine is the mechanics — running items, freezing inputs, aggregating, settling — and the scorers are the algorithm layer on top, including custom scorers you implement as a tool. The boundary section below maps which is which.

See the Permissions Reference for the IAM action strings for this module.

Related Tutorials

• Evaluate an Agent - Step 3 (Build a dataset)
• Evaluate an Agent - Step 6 (Measure a prompt change against a baseline)
• Judge Open-Ended Answers - Step 3 (Bind an llm_judge scorer)
• Judge Open-Ended Answers - Step 5 (Run queued and poll)
• Gate a Canary Promotion on an Eval - Step 4 (Set a promotion gate)
• Gate a Canary Promotion on an Eval - Step 8 (Schedule nightly runs)

Data Model

Dataset

Field	Type	Description
id	string	Public identifier (e.g. dset_…)
project_id	string	ID of the owning project
name	string	Unique within the project
description	string	Optional free text
created_at / updated_at	string	ISO 8601 timestamps

Deleting a dataset deletes its items and the evals bound to it.

Dataset item

Field	Type	Description
id	string	Public identifier (e.g. dsit_…)
dataset_id	string	ID of the owning dataset
input	array	{ role, content } messages, replayed verbatim as the generation's input
expected_output	string	Reference answer for exact_match and llm_judge; may be null
metadata	object	Free-form tags (e.g. {"topic": "billing"}), opaque to the platform and readable from json_logic scorers
source_generation_id	string	The generation this item was curated from (see Curating items from production); null for a hand-authored item, and null again once that generation is deleted
created_at / updated_at	string	ISO 8601 timestamps

Eval

Field	Type	Description
id	string	Public identifier (e.g. eval_…)
project_id	string	ID of the owning project
name	string	Unique within the project
agent_id	string	The agent under test — must be in the same project
dataset_id	string	The dataset to run it against — must be in the same project
scorers	array	Scorer configs; see Scorers
pass_threshold	number	0–1, or null to report without gating; see Pass semantics
created_at / updated_at	string	ISO 8601 timestamps

An agent_id or dataset_id naming a resource in another project is rejected with 400.

Eval run

Field	Type	Description
id	string	Public identifier (e.g. evrun_…)
eval_id	string	ID of the eval this run belongs to
agent_version	integer	The one agent version every item ran against; see Version pinning
status	string	queued | running | completed | failed | canceled
baseline_run_id	string	A terminal run of the same eval, or null
trigger_id	string	The trigger that started this run, or null for a run started through the API. Kept even after that trigger is deleted
aggregate_scores	object	Per-scorer mean / pass_rate, the run pass_rate, scored_item_count, and — when the run named a baseline — a baseline comparison. null until the run is terminal, and on a canceled run
passed	boolean	The verdict; null when the eval declares no pass_threshold, and on a canceled run
item_count / completed_count / errored_count	integer	Items attempted, scored, and errored. On a canceled run the last two count what actually ran
metadata	object | null	Caller-owned annotations supplied when the run was started and returned verbatim (see Run metadata)
started_at / finished_at	string	ISO 8601 timestamps, null until set
created_at	string	ISO 8601 creation timestamp

Eval result

One row per dataset item per run.

Field	Type	Description
id	string	Public identifier (e.g. evres_…)
eval_run_id	string	ID of the run
dataset_item_id	string	The item this scored; null once that item is deleted
input	array	Frozen copy of the item's input at run time
expected_output	string	Frozen copy of the item's expected output at run time
generation_id	string	The generation that produced the output, or null
output	string	The agent's final output text. null only when there is none — the generation failed or never completed — or once the linked generation's content is purged. An item errored by a scorer keeps the output it was graded on
scores	array	[{ scorer, score, passed, reasoning? }], one entry per scorer in the order the eval declares them. scorer is the type — or the scorer's name for a tool scorer. reasoning is present for llm_judge, and for tool scorers whose tool returned one
passed	boolean	AND over the per-scorer passed flags
error	string	Item-level failure reason; set instead of scoring, never alongside it
created_at	string	ISO 8601 creation timestamp

Key Concepts

The engine and the scorers

The two layers of the engine & algorithms pattern map onto this module like so:

Layer	What it covers here	Where it is documented
The engine — mechanics, not configurable, no opinions	Executing every item against the real agent, freezing inputs, version pinning, pass semantics and aggregation, error handling, sync/queued execution, cancelation, scheduling, baseline deltas, webhooks, metering, retention	The sections named at left
The algorithms — opinionated, swappable	The scorers: what "good output" means for an item	Scorers, LLM judge
Bring your own	A scorer whose grading logic is your code	Custom scorers

The engine's guarantees — a run settles, an errored item is never a 0, frozen inputs keep runs comparable — hold identically for built-in and custom scorers.

Scorers

scorers is a discriminated union on type. Every scorer produces { score: 0–1, passed: boolean } — binary scorers emit 0 or 1 — so aggregation, thresholds, and baseline deltas never care which algorithm produced a score. Each built-in type may appear at most once per eval; tool scorers may appear several times, each under a distinct name (outcomes and aggregate scores key on the type, or on the name for tool scorers).

type	Config	Scores
exact_match	—	1 when the trimmed output text equals expected_output; an item with no reference answer cannot pass
contains	value, case_sensitive (default false)	1 when value occurs in the output text
json_logic	expression	1 when the JSON Logic expression evaluates truthy
output_schema	schema (optional)	1 when the structured output validates against the schema
embedding_similarity	pass_threshold	The cosine similarity between the embeddings of the output text and expected_output, clamped to 0–1; see Embedding similarity
llm_judge	prompt, pass_threshold, ai_provider_id (optional), model (optional)	The judge's 0–1 score; see LLM judge
tool	name, tool_id, action (builtin/mcp tools), preset_parameters (optional), pass_threshold (optional)	Whatever your algorithm answers; see Custom scorers

exact_match, contains, embedding_similarity and llm_judge read the final text; output_schema validates the structured object the platform already parsed. json_logic sees both, through these variables:

Var	Value
input	The item's input messages
output	The final output text
object	The structured output. Absent when the agent has no output_schema — an expression over it evaluates falsy rather than erroring
expected	The item's expected_output
item.metadata	The item's metadata bag

An output_schema scorer is rejected with 400 unless the agent under test carries an output_schema — even when the scorer supplies its own schema — because the platform only produces structured output when the agent's schema constrains the model. The check runs at eval-create (best-effort) and again at run start (authoritative).

Embedding similarity

An embedding_similarity scorer grades semantic closeness instead of literal overlap: it embeds the output text and the item's expected_output with the platform's configured embedding model — the EMBEDDING_PROVIDER / EMBEDDING_MODEL environment variables, the same stack document ingestion uses — and scores their cosine similarity, clamped to 0–1. It sits between the deterministic text scorers and the judge: cheaper and more repeatable than an LLM judge (an embedding call per item instead of a completion), while tolerating paraphrases exact_match would fail.

pass_threshold is required on the scorer, with no default, for the same reason as the judge's: cosine similarity is a continuous score, and nothing about it says where "close enough" is for your domain — 0.85 can be strict for one embedding model and permissive for another, so calibrate it against your own data.

Two edges mirror the rest of the module:

• An item with no expected_output scores 0 and cannot pass — similarity is measured against the reference answer, so without one there is nothing to be close to (the same rule as exact_match). The embedding backend is not called for such an item.
• An embedding backend failure marks the item errored — never the run failed, and never a score of 0 (the same rule as a judge that cannot answer): a backend that could not embed says nothing about the agent.

Because the embedding model is platform-configured, scores from runs executed under different EMBEDDING_MODEL values are not comparable — re-run the baseline when the embedding model changes, just as you would when a judge model changes.

LLM judge

An llm_judge scorer grades the output with a tool-less model completion, resolved through the ordinary AI providers path (the scorer's ai_provider_id must belong to the eval's project; the project's default model route applies when the scorer pins none). The prompt carries three slots, filled in one pass (a slot value containing {{output}} is never re-expanded; an unrecognised {{…}} is left as written):

Slot	Filled with
{{input}}	The item's input messages (JSON when not a plain string)
{{output}}	The agent's final output text
{{expected}}	The item's expected_output, or empty when it has none

The judge must answer with a JSON object carrying a numeric score between 0 and 1 and an optional reasoning string (stored on the result). Prose or a code fence around it is tolerated (the first {…} span is parsed). A non-JSON reply, non-numeric score, or score outside 0–1 marks the item errored — never the run failed, and never a score of 0.

pass_threshold is required on the scorer, with no default; the item passes when score >= pass_threshold. Judges drift with model updates — re-run the baseline when the judge model changes.

Custom scorers (tool)

A tool scorer is the module's bring-your-own-algorithm seam: the engine invokes a tool you own once per item, and your code — any language, any model, any vendor — answers with the same { score, passed } shape every built-in scorer produces. Aggregation, thresholds, and baseline deltas apply to it unchanged.

Config field	Required	Meaning
name	yes	Keys this scorer's outcomes and aggregate buckets. Unique within the eval; must not shadow a built-in type. Several tool scorers may coexist under distinct names
tool_id	yes	The tool that grades each item. Must belong to the eval's project and be server-callable — http, mcp, builtin, or pipeline. A client tool is rejected with 400: it pauses for a calling client, and an eval run scores server-side
action	builtin/mcp only	The operation to invoke on a multi-action tool
preset_parameters	no	Fixed values merged into every call's input at the top level. The engine-injected keys below are reserved and rejected
pass_threshold	no	Fallback verdict cutoff; see below

Input — the engine calls the tool with the item's context: the same variables a json_logic expression reads, so the two algorithm surfaces share one contract.

{
  "input": [{ "role": "user", "content": "Is this friendly?" }], // the item's input messages
  "output": "Absolutely, very friendly!",  // the agent's final output text
  "object": { "category": "other" },       // structured output; absent when the agent has no output_schema
  "expected": "yes",                        // the item's expected_output, or null
  "item": { "metadata": { "topic": "tone" } }
  // preset_parameters are merged in at the top level
}

Output — the tool must answer with a JSON object (an http target answering text/plain, or an mcp tool's text content, is scanned for its first {…} span):

{
  "score": 0.9,               // required, 0–1
  "passed": true,             // optional — your algorithm's own verdict
  "reasoning": "Warm phrasing." // optional, stored on the result
}

Verdict resolution. A tool-returned passed always wins. When the tool omits it, the scorer's pass_threshold applies (score >= pass_threshold, the same >= rule as llm_judge). When neither exists, the item is recorded as errored — a scorer that produced no verdict must not guess one. Return passed from the tool when the algorithm owns the cutoff; declare pass_threshold when you want to tune the cutoff in the eval config without redeploying the tool.

Error semantics follow errors are not zeros: a failed tool call, an unparseable answer, an out-of-range score, or a missing verdict errors the item — never the run, and never a score of 0. The item keeps the output it was graded on, and its error names the scorer.

Validation happens at eval create and update, and again — authoritatively — at run start, so a tool deleted after the eval was created fails the run request with 400 rather than erroring every item.

Bind one like any other scorer:

soat create-eval --project-id "$PROJECT_ID" --name tone-suite \
  --agent-id "$AGENT_ID" --dataset-id "$DATASET_ID" \
  --scorers '[{"type":"tool","name":"tone","tool_id":"'"$TOOL_ID"'","pass_threshold":0.5}]' \
  --pass-threshold 0.8

An eval run invokes the tool once per item — real calls, like everything else in a run — so point scorer tools at infrastructure that tolerates the volume, and at a staging target if the algorithm itself has side effects.

Frozen inputs

Each result carries its own copy of the item's input and expected_output, taken at run time, so editing or deleting an item between two runs cannot make their scores incomparable. Deleting an item nulls dataset_item_id on past results and changes nothing else.

Curating items from production

A hand-authored dataset drifts away from the traffic it is supposed to represent, which is the traffic a canary actually has to survive. create-dataset-item-from-generation promotes a real turn instead:

soat create-dataset-item-from-generation \
  --dataset-id "$DATASET_ID" \
  --generation-id "$GENERATION_ID"

The generation's stored input messages become the item's input, and its own answer becomes expected_output — pass --expected-output to override it, or null to store the item with no reference answer. source_generation_id records where the item came from.

What the item stores is a copy, not a view. It keeps working after the source generation's content is purged, and if that generation is deleted source_generation_id simply goes null — consistent with the rest of the module, where a purge can never quietly stop a suite from being runnable.

Two rules bound what can be promoted:

• Only a completed generation. A paused (requires_action) or failed turn has no finished answer, so it is refused with 409 GENERATION_NOT_COMPLETED rather than turned into a fixture that scores whatever the agent does next.
• Only while its content is available. Replay needs the input that content retention exists to withhold, so an agent or project running with trace_content_mode: none never stored it, and a purged or expired generation no longer has it. Both answer 409 GENERATION_CONTENT_UNAVAILABLE, as do generations produced before input recording existed.

The call copies content out of a generation, so it requires generations:GetGeneration in addition to evaluations:CreateDataset. The generation must also belong to the same project as the dataset.

Version pinning

A run resolves one agent version at run start, stamps it on agent_version, and every item executes against it.

• Pass agent_version to name an archived version. An unknown version is a 400.
• Omit it and the run uses the active release's stable version, or the live draft version when no release is in effect.

An eval-gated promotion matches on the pin: a release naming this eval as its promotion_gate promotes only once a run finished completed with passed: true and carried the canary's agent_version.

Pass semantics

1. Per scorer, per item — a binary scorer passes when its score is 1; an llm_judge scorer passes when its score is at least the scorer's own pass_threshold.
2. Per item — EvalResult.passed is the AND over its per-scorer flags.
3. Per run — EvalRun.passed is null when the eval has no pass_threshold; otherwise it is true when the pass rate — passed items over non-errored items — is at least the threshold.

The verdict gates on the pass rate, never on a pooled mean. aggregate_scores still reports per-scorer means. A run that scored nothing at all does not pass.

Errors are not zeros

An item whose generation did not complete — e.g. an agent with client-side tools pausing for tool outputs (requires_action) — is recorded as an error, excluded from aggregate_scores, and counted in errored_count; it is never scored 0. The same rule covers a scorer that could not reach a verdict (an llm_judge call failing or answering something unparseable). When the generation produced nothing, output is null; when a scorer failed over a good generation, the result keeps that generation's output alongside the error. The generation stays linked either way.

Synchronous and queued runs

wait selects how a run executes (see sync vs async for the platform-wide contract). Both modes share one execution and finalize path.

wait	Behavior
true	Executes items sequentially in-process and returns the run terminal, with its scores. Capped at 25 items — a larger dataset is rejected with 400.
false (default)	Enqueues one task per item and returns immediately with status: "queued". No item cap.

An empty dataset is rejected in both modes: a run that measured nothing must not produce a verdict.

For a queued run, a worker claims tasks in batches; the worker that drains the run's last task settles the run and fires eval_run.completed. Poll GET /evals/{eval_id}/runs/{eval_run_id} or subscribe to the webhook. Delivery is at-least-once but safe: a result row is unique per (run, item), and settling is guarded by an atomic claim, so the completion event fires exactly once.

A background reaper settles non-terminal runs that have gone quiet past a grace period (30 minutes by default): a run whose items all have results is finalized; a run with items missing and no outstanding work is settled failed and eval_run.failed fires. A run that still has queued tasks is left alone.

Run metadata

start-eval-run accepts a metadata bag — caller-owned key/value annotations, stored on the run and returned verbatim by every read of it, the list included. It answers "what was this measurement of": the commit or release candidate being scored, the CI job that asked for it, the experiment it belongs to.

Every other field on the start request is platform-owned (wait, agent_version, baseline_run_id), so before this bag a CI caller running one eval per commit had nowhere to record which commit — the run was a score with no subject. trigger_id records a scheduled origin, which is provenance the platform knows; metadata records the caller's own, which it cannot.

Nothing in the scoring path reads it, and no key is reserved: status, agent_version, aggregate_scores, passed and the counts are all fields of their own and cannot be written from here. A non-object metadata is rejected with 400 VALIDATION_FAILED and no run is created.

soat start-eval-run \
  --eval-id "$EVAL_ID" \
  --metadata '{"commit_sha":"9f2c1ab","ci_job":"nightly-evals"}'

Filtering runs by a metadata key is not supported — fetch and filter client-side.

Run tool context

An eval scores the agent you are about to ship. An agent whose tools authorize through tool_context cannot be scored that way with an empty bag: a tool declaring Authorization: Bearer {{context:...}} fails every item with MISSING_TOOL_CONTEXT_KEY, and — worse — a tool that tolerates a missing key evaluates a different configuration than production, so the score is quietly about a different account, tenant or scope. A green eval on the wrong scope is more dangerous than a red one.

start-eval-run accepts a tool_context bag, forwarded to every item's generation:

soat start-eval-run \
  --eval-id "$EVAL_ID" \
  --wait true \
  --tool-context '{"ocaToken":"eyJhbGciOiJIUzI1NiJ9.abc","tenant":"acme"}'

Three properties distinguish it from metadata:

• It lives on the run, not the request. A queued run — the default, and the only shape a scheduled one has — is driven by a worker with no request behind it, so the bag is stored on the run row and re-read for every item.
• It is write-only. No read of the run returns it. A run is a report other people read, and a credential in it is not theirs to see; metadata is readable for the opposite reason — a label is not a credential.
• It does not outlive the work. The bag is cleared once the run reaches a terminal state, so a finished run — which is kept as a historical measurement — holds no credential.

The usual tool_context rules apply: each key is forwarded as one X-Soat-Context-<key> header and resolves any {{context:}} token in a bound tool's headers or preset_parameters, a tool's context_keys narrows what reaches it, and a key that could not become a header name is rejected with 400 INVALID_TOOL_CONTEXT_KEY before any run is created. An eval generation has no session, so the reserved identity keys (session_id, actor_id, actor_external_id) are dropped rather than forwarded.

Canceling a run

POST /evals/{eval_id}/runs/{eval_run_id}/cancel drops a queued or running run's outstanding tasks and settles it canceled; a run that has already finished is rejected with 400. Results already written are kept, and completed_count / errored_count report what ran — an item a worker had already claimed runs to completion and recounts the run after it settles. aggregate_scores is deliberately left null (a partial roll-up would read as a whole-dataset verdict), and no lifecycle event fires. The run's tool_context is cleared, as on any terminal transition.

Scheduled runs

A trigger with target_type: "eval" runs a suite on a cadence. Every starter works (manual, webhook, and cron schedule), and the firing always starts a queued run; the firing's result.result_id is the evrun_… to poll. The run records its origin in trigger_id and keeps it if the trigger is later deleted.

The trigger's input may carry agent_version and baseline_run_id; both are validated at fire time, so a nightly schedule naming a version that no longer exists fails the firing (with the reason on the firing record) instead of creating a run that could never execute. Creating an eval-target trigger requires evaluations:RunEval on top of triggers:CreateTrigger.

A trigger carries no tool_context of its own, so a scheduled run of an eval whose agent needs one starts with an empty bag. Until a trigger can attach one, such a suite has to be started through the API.

soat create-trigger \
  --project-id "$PROJECT_ID" \
  --name nightly-regression \
  --type schedule \
  --target-type eval \
  --target-id "$EVAL_ID" \
  --cron "0 3 * * *"

Formation support

Datasets, their items, and evals are declarable in a Formation template:

Resource type	Properties
dataset	name, description
dataset_item	dataset_id, input, expected_output, metadata
eval	name, agent_id, dataset_id, scorers, pass_threshold

Items are their own resource, so an item curated through the API is never collateral of a formation apply. dataset_id is immutable on a dataset_item — a template that moves an item to another dataset is rejected. Running the suite gives the agent under test generation history, so deleting the formation later fails with 409 FORMATION_DELETE_FAILED naming that agent — see formation teardown; force-delete the agent (DELETE /api/v1/agents/{agent_id}?force=true) or declare it with deletion_policy: retain.

Baseline deltas

Pass baseline_run_id (a terminal run of the same eval; a run of another eval is a 400) and the finished run's aggregate_scores.baseline reports how it moved:

Field	Meaning
run_id	The baseline compared against
compared_item_count	Items present and scorable in both runs — the basis of every delta
added_item_count	Scorable here but not in the baseline (added since, or errored there)
removed_item_count	Scorable in the baseline but not here (removed since, or errored here)
pass_rate_delta	Run-level pass-rate delta over the intersection; null when the two runs share no comparable item
scorers	Per scorer type, mean_delta and pass_rate_delta

Positive deltas mean this run scored higher than the baseline. Every number is computed over the item intersection, recomputing both sides, so dataset drift is reported through the counts instead of being attributed to the agent. A scorer that only one of the two runs ran is omitted.

Lifecycle webhooks

Two webhook events carry a run's outcome:

Event	Fires when
eval_run.completed	A run reached a terminal status with its items scored
eval_run.failed	A run could not be executed to completion

Both carry { eval_id, eval_run_id, passed, aggregate_scores } inline — this event is the promotion gate, so the verdict must not require a second call. Exactly one event fires per terminal run.

Eval spend is separable from production spend

Every item is a real generation, and llm_judge doubles the calls. Eval spend is labelled in usage metering: item generations carry source: "eval" and judge completions source: "eval_judge" (ordinary agent traffic carries no source). Filter with GET /api/v1/usage/meters?source=eval or roll up with GET /api/v1/usage?group_by=source. Quotas and usage thresholds still apply to eval runs.

An embedding_similarity scorer's own embeddings are metered too, under source: "embedding" rather than "eval_judge" — they go through the deployment's embedding stack, not a project provider, so they carry that stack's provider and model.

Eval runs have real side effects

A run creates real generations, so an agent with a write-capable http or mcp tool performs N real writes per run. There is no tool-stub mode. Point an eval'd agent's tools at a staging target.

Retention and erasure

EvalResult.output is a copy of a generation's content, so purging that generation's content — directly, or through its trace — also clears the copy. Scores, passed, and the frozen input / expected_output survive. Datasets are operator-owned test fixtures: a content purge never deletes or mutates a dataset item — an erasure covering curated content requires deleting the item explicitly. That applies to items curated with create-dataset-item-from-generation too: promoting a turn copies its content into a fixture that outlives the source, which is what keeps a suite runnable, and also what makes deleting the item the only way to erase it.

Examples

Create a dataset and add a case:

soat create-dataset --project-id "$PROJECT_ID" --name billing-regressions

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"When is my invoice issued?"}]' \
  --expected-output "On the first of each month." \
  --metadata '{"topic":"billing"}'

Bind an eval and gate it at an 80% pass rate:

soat create-eval --project-id "$PROJECT_ID" --name billing-regression-suite \
  --agent-id "$AGENT_ID" --dataset-id "$DATASET_ID" \
  --scorers '[{"type":"contains","value":"first of each month"}]' \
  --pass-threshold 0.8

Run it synchronously and read the per-item results:

soat start-eval-run --eval-id "$EVAL_ID" --wait true
soat list-eval-results --eval-id "$EVAL_ID" --eval-run-id "$RUN_ID"

Queue a larger run and poll for the verdict:

soat start-eval-run --eval-id "$EVAL_ID" --wait false   # → status: queued
soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$RUN_ID"
soat cancel-eval-run --eval-id "$EVAL_ID" --eval-run-id "$RUN_ID"

Evaluate a specific archived version against a baseline — the shape a promotion gate uses:

soat start-eval-run --eval-id "$EVAL_ID" --wait true \
  --agent-version 3 --baseline-run-id "$BASELINE_RUN_ID"