Evals

Overview

An eval is a scored check that runs your agent against real sessions and grades the result, catching regressions when you change a prompt or a tool. Drive the agent through one or more turns, assert on what it did (the run completed, the right tool ran, the reply contains the right text), and optionally ship the results to Braintrust.

Evals exercise the same HTTP surface your users hit. The runner boots (or targets) a real agent server, drives sessions through the TypeScript client protocol, and grades what comes back, so a passing eval means the agent booted, accepted a request, and produced the result you asserted.

Define tool stubs in your eval to control tool results, then assert on the agent's tool calls and responses.

defineEval

eve discovers evals under the app-root evals/ directory, in .eval.ts files. Each file is one eval by default. A file can also default-export an array to fan out over a dataset (see Cases). The file path is the eval's identity, so you don't author an id or name. Directories group related evals (evals/weather/brooklyn-forecast.eval.ts becomes id weather/brooklyn-forecast).

my-agent/
├── agent/
├── evals/
│   ├── evals.config.ts
│   ├── smoke.eval.ts
│   └── weather/
│       ├── brooklyn-forecast.eval.ts
│       └── no-tools-for-greetings.eval.ts
└── package.json

An eval is a single async test(t) function. You drive the agent with t and assert on the run with the same t:

evals/weather/brooklyn-forecast.eval.ts
import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  description: "Basic message and tool-usage coverage for the weather agent.",
  async test(t) {
    const turn = await t.send("What is the weather in Brooklyn?");
    t.succeeded();
    t.calledTool("get_weather");
    t.check(turn.message, includes("Sunny"));
  },
});

test is the only required field. The rest are optional: description, judge, tags, metadata, timeoutMs, and reporters. The init template adds evals/**/*.ts to tsconfig.json, so your eval code type-checks alongside the app.

evals.config.ts

Every evals/ directory needs exactly one evals.config.ts at its root. It declares the defaults every eval shares:

evals/evals.config.ts
import { defineEvalConfig } from "eve/evals";
import { Braintrust } from "eve/evals/reporters";

export default defineEvalConfig({
  judge: { model: "typesafe-ai/jev" },
  reporters: [Braintrust({ projectName: "my-agent" })],
});

Everything is optional. judge sets the default model for LLM-as-judge assertions (t.judge(...)); omit it to use the default decider, typesafe-ai/jev. Deterministic evals make no judge requests. reporters, maxConcurrency, and timeoutMs round out the defaults. Config reporters observe every eval in the run, so set one Braintrust() here instead of adding it to each eval. CLI flags (--max-concurrency, --timeout) and per-eval values take precedence over the config defaults.

Run setup and teardown

Use setup to start shared resources or run migrations before eve starts the local agent server. Return shared context directly from setup, then release resources in teardown:

evals/evals.config.ts
import { defineEvalConfig } from "eve/evals";
import { startTestDatabase } from "./database.js";

export default defineEvalConfig({
  async setup() {
    const database = await startTestDatabase();
    return { database };
  },
  async teardown(context) {
    await context?.database.close();
  },
});

context is inferred from the setup return value. To use that type in an eval file, pass the config's type to defineEval:

evals/database.eval.ts
import { defineEval } from "eve/evals";
import { equals } from "eve/evals/expect";
import type config from "./evals.config.js";

export default defineEval<typeof config>({
  async test(t) {
    const result = await t.context.database.query("SELECT 1 AS value");
    await t.require(result.rows[0]?.value, equals(1));
  },
});

Setup, eval bodies, and teardown share the same context object in the runner process. Class instances and functions keep their identity; context is neither serialized nor sent to the agent server. Concurrent evals share these resources, so isolate any database records they modify.

setup can also return nothing.

  • Setup runs once per eve eval invocation, after discovery and config loading, before target startup.
  • Teardown runs after the local server and its sandbox handles shut down, including after setup, startup, or eval failures. Cleanup failures fail the command.

teardown receives undefined when setup returns no context or throws before returning. In that case, setup must clean up resources that only it can access. Teardown can also be used without setup. The per-eval timeoutMs does not apply to setup or teardown.

Deterministic fixture models

Use mockModel when an eval fixture needs to exercise eve's runtime without calling a model provider. A static fixture can be one line:

agent/agent.ts
import { defineAgent } from "eve";
import { mockModel } from "eve/evals";

export default defineAgent({
  model: mockModel("A deterministic reply"),
});

Pass a callback when the reply depends on the conversation. The callback receives an eve-owned view of the prompt, including lastUserMessage, userMessages, userMessageCount, available tools, and prior toolResults:

agent/agent.ts
export default defineAgent({
  model: mockModel(
    ({ lastUserMessage, userMessageCount }) => `Turn ${userMessageCount}: ${lastUserMessage}`,
  ),
});

The callback may return { text, toolCalls, usage } for deterministic tool loops or explicit token counts. Use the options form only when a fixture also needs a custom model identity:

agent/agent.ts
model: mockModel({
  modelId: "weather-script",
  provider: "my-fixtures",
  respond: ({ toolResults }) =>
    toolResults.length === 0
      ? { toolCalls: [{ name: "get_weather", input: { city: "Brooklyn" } }] }
      : `Weather: ${JSON.stringify(toolResults[0]?.output)}`,
});

mockModel() uses "Mock response" when no response is supplied. It handles both generated and streamed responses, derives deterministic response metadata, and estimates token usage. It compiles with a 1,000,000-token context window, so compaction stays out of the way; set modelContextWindowTokens on the agent to exercise compaction with a smaller window. Because the model is part of the agent definition, use it for a dedicated fixture agent; it remains mocked whether that fixture runs locally or as a deployed eval target.

The t context

t is both the driver and the assertion surface. There are no separate input, run, checks, or scores fields. You write ordinary control flow, sending turns and asserting inline.

  • Drive the agent: t.session() creates an empty session; t.send(...) creates a session with its first message and returns a turn. Continue through turn.session.send(...). Session handles also expose start, cancel, respond, respondAll, sendFile, and requireInputRequest. Read output from turn.message, and conversation state from turn.session. See Cases.
  • Assert with three surfaces, covered next.

Three assertion surfaces

Each surface matches a genuinely different kind of judgment:

  • Scoped methods read the final whole run on t, snapshot one independent session when invoked there, or inspect one immutable EveEvalTurn. See Assertions.
  • t.check(value, assertion) grades an explicit value with a deterministic builder from eve/evals/expect, such as t.check(turn.message, includes("sunny")). Grade turn.message, an intermediate draft, parsed JSON, or anything else. See Assertions.
  • t.score(evaluation) records a score, such as a length ratio or an evaluator's result. Chain .label(key) to name it, and .gate(threshold) or .atLeast(threshold) to let it decide the eval. See Assertions.
  • t.judge(...) is the LLM-as-judge surface, like t.judge("cites a source"). It grades the most recently settled turn’s assistant message by default; pass { on: turn.session.transcript } to grade a multi-turn conversation. The judge uses the configured judge model, never the agent under test. See Judge.

Gate vs soft

Every assertion returns a chainable handle, so severity rides on the assertion itself. There is no separate thresholds map.

  • Gates are hard. A failed gate marks the eval failed and eve eval exits non-zero. Run-level methods, includes, equals, and matches are gates by default.
  • Soft assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval scored (visible but not fatal, unless you pass --strict). similarity, every t.judge(...) assertion, and every t.score(...) are soft by default. A soft assertion with no threshold records no verdict and never fails.

Override per assertion: .gate(threshold?) promotes to a hard gate, .soft(threshold?) demotes to tracked, and .atLeast(threshold) is a soft assertion with a bar.

t.succeeded(); // gate
t.calledTool("get_weather").soft(); // record as a metric, don't gate
t.judge("cites a source"); // soft, tracked (no threshold)
t.judge("The response is consistent with output.reference.", {
  on: { response: turn.message ?? "", reference },
}).atLeast(0.7); // soft, gated under --strict at 0.7

Use await t.require(value, assertion) for a gate that must pass before the script can safely continue. Use t.skip(reason) as the first operation for an intentionally unsupported target capability.

Run evals with eve eval

eve eval                       # run all discovered evals against a local dev server
eve eval weather               # run one eval, or every eval under evals/weather/
eve eval --url https://<app>   # target an existing server or deployment

Exit code 0 means every eval passed its gates. See Running evals for the full flag list, exit codes, and CI guidance.

A good baseline

Most apps do fine with a few small smoke evals. Assert behavior with t.succeeded() plus one or two content checks, keep dataset fixtures in evals/data/, and reach for a judge or Braintrust only when you need fuzzy grading or shared result review. In CI, run eve eval --strict so soft threshold misses fail the build too.

The rest of this section covers each piece:

  • Cases: single-turn evals, scripted multi-turn evals, and dataset fan-out
  • Assertions: run-level methods and t.check value assertions, with matchers and severity
  • Judge: LLM-as-judge grading and the judge model
  • Targets: local vs remote targets for the same eval files
  • Reporters: Braintrust experiments and JUnit XML
  • Running evals: the eve eval CLI, exit codes, and artifacts
  • Tools: the surface most evals assert on
PreviousSelf-Host eveNextCases