Untested changes

On every pull request, evalship finds the LLM calls and prompt files the pull request adds or changes, and checks whether any eval exercises them. Untested ones appear in the comment:

- **Untested:** `app/refunds.py#RefundPolicy.explain` has no eval. [Add one](https://evalship.com/docs/untested-changes#fixing-an-untested-change)

The details block lists each one with its line and provider. This runs for pull requests on public repositories and on enabled private repositories.

What evalship looks at

evalship reads up to 25 added or modified files per pull request. Removed files are ignored.

Source files. A line is an LLM call when the file imports a supported SDK or framework and the line matches that library's call pattern. The call counts as changed when the pull request changes a line within 5 lines of it, such as its prompt or arguments. When GitHub leaves out a file's diff because it is too large, every call in the file counts as changed.

Examples of lines that count as calls: client.chat.completions.create(, client.messages.create(, model.generate_content(, litellm.completion(, chain.invoke(, generateText(, chat.ask "...", client.Messages.New(.

Comment lines are skipped, and a call split across lines before the method name, such as client.messages on one line and .create( on the next, is not detected.

Prompt files. Any change to a prompt file counts. Prompt files are:

• .prompt and .prompty files anywhere;
• .txt, .md, .mdx, .yaml, .yml, .j2, .jinja, .jinja2, .hbs, .mustache, .liquid and .tmpl files inside a directory named prompt or prompts, at any depth;
• agent skills: SKILL.md files, and .md and .mdx files inside a skills/<name>/ directory, such as a skill's reference files. Skills under a hidden folder such as .claude/ or .agents/ are the repository's own development tools and don't count.

Prompt modules. A Python, JavaScript, TypeScript, Ruby or Go file whose name or folder mentions prompts (prompt_utils.py, prompt-preset/gpt.ts, system-prompt.ts) counts as one prompt when the pull request changes a line of it that isn't blank or a comment, unless a more specific prompt or call in it is already listed. A file with more than three changed prompts or calls shows as one row.

Tool descriptions. In a Python, JavaScript or TypeScript file that defines tools a model chooses from (an MCP server, tool({...}), registerTool, @tool), a change to a description string counts as one change to that file's tool descriptions.

Prompts from your review. The latest audit lists the repository's prompts with a file and line, including prompts the patterns above can't find, such as a prompt string in a Python or TypeScript module or the instructions an MCP server sends. A listed prompt counts as changed when the pull request changes a line from where it starts up to the next definition in the file, at most 100 lines. It is found by name in the pull request's version of the file, so a review of an older commit still points at the right lines. A new upper-case string constant in a file of listed prompts counts as a new prompt that no eval runs yet.

Model changes. In a file that sets up a provider (it imports an @ai-sdk/* provider package or Bedrock), a changed line with a model id such as "claude-sonnet-4-6" or "gpt-4o" counts too, once per file, because switching models changes every output. A model id within 5 lines of a call is part of that call.

Formatting is left out. A change that only rewraps or reindents lines, the same text apart from whitespace, doesn't count, nor does a file renamed without changes, a README or changelog in a prompts folder, or a prompt file whose only changed lines are a version or date in its front matter.

Prompt files named by an eval. A prompt file the review doesn't place counts as tested when a suite's files, or the product files a suite exercises, name it (loadPrompt("operate-user"), a promptfoo file:// path), or when the pull request's own eval files name it.

Eval code is left out. Calls and prompts in test and eval files don't count, since they are the evals: files in test, tests, spec, __tests__, eval, evals and e2e directories and in eval packages such as agent-evals/, *.eval.*, *.test.* and *.spec.* files, test_*.py, *_test.* and *_spec.rb, and the folders of the suites in your review.

Each call is named by its file and up to two levels of enclosing function or class: app/refunds.py#RefundPolicy.explain.

What counts as tested

evalship checks these in order and stops at the first match. Only cases in the current run count for the first three.

1. Targets. A case lists the file in its targets. app/support_bot.py covers every call in that file; app/support_bot.py#answer covers the calls inside answer, including SupportBot.answer. The path can be a glob: prompts/*.md matches files directly in prompts/, and app/ or app/** matches everything below app/. See Result formats for how to set targets in each format.
2. Annotation. An evalship: covered-by <pattern> comment near the call, anywhere from 3 lines above the enclosing function's definition down to the call itself. The pattern is a suite name or a suite/key glob; separate several with commas. It must match at least one case.
3. Name. A suite or eval file is named after the source file, function or class. Names are compared in lowercase with everything but letters and digits removed, so app/support_bot.py is matched by the suite support-bot or the eval file tests/evals/test_support_bot.py. Names shorter than 4 characters and generic names (app, client, utils, llm, ai, service, api, prompt, chat, agent, model and similar) never match.
4. Audit. The latest audit lists the prompt (same file, and the same line or symbol) with a non-empty tested_by; an empty tested_by means untested, whatever else matches. Otherwise, a suite's exercises lists the file, file#symbol, or a prefix ending in * that the file starts with; or a prompts entry for the file with a non-empty tested_by is within 20 lines of the call.

Annotations look like this:

# evalship: covered-by support_bot
def answer(question):
    return client.messages.create(model="claude-haiku-4-5", max_tokens=500, messages=[...])

// evalship: covered-by support-bot/refund-*, summarizer
export async function explainRefund(order: Order) {
  return generateText({ model, prompt: refundPrompt(order) });
}

Checks that don't call a model. A suite the review marks with calls_model: false (snapshots, schema checks, unit tests of prompt text) doesn't count as testing a prompt or call; the change shows as untested, with that suite named in the details.

Plumbing. Near a call, a change counts when it's inside the call's arguments, just after the call, or in the 5 lines above it and touches a string, prompt, message, system, template, tools, schema, temperature or token setting. A changed API key, client or user id next to the call doesn't.

Not run on pull requests. When the only suites that exercise a change don't run on every pull request (their ci_triggers in the review don't include pull_request: they run by hand, on a schedule, or only with a label), the change isn't counted as untested. The comment says so instead: "Not run on PRs: prompts/review.md changed, and only reviewer-benchmark exercises it, and not on every pull request."

No eval results. If a pull request's latest commit still has no eval results 30 minutes after it was pushed, and it changes a prompt or LLM call (tested or not), a suite's scoring or cases, or the reviewer has a note on its eval changes, evalship comments anyway: "No eval results for this commit." with the same rows and notes. This needs a review. Results that arrive later replace the comment.

Fixing an untested change

1. Add an eval that exercises the change, and give its cases targets pointing at the file or function. This is the most reliable link, because it doesn't depend on names.
2. If an existing eval already covers it, add targets to that eval's cases, or put a covered-by annotation next to the call.
3. Re-run the setup prompt. The agent audits again, records which code each suite exercises, and can propose an eval for the new call as part of its plan. See Quickstart.

The bullet disappears from the comment once the next run finds a covering case.