We transform your evals into their best selves during setup.
Then the evalship keeps watch from orbit: on every diff, build and PR, it
shows you what changed and why.
Ask your coding agent:
Set up https://evalship.com/setup.md
+2 failing evals vs main
| Eval | main | this PR | |
|---|---|---|---|
| 🔴 | Refund above limit | ✅ 0.96 | ❌ 0.22 |
| 🔴 | Refund far above limit | ✅ 0.94 | ❌ 0.18 |
| 💬 | refunds.py#RefundPolicy.explain |
new | no eval |
| 🟣 | Casual tone (flaky) | ✅ 0.81 | 🎲 0.62 |
🔍 Both new failures promise refunds above the $100 limit, likely from the removed line "Never promise refunds above $100" in support_system.md.
💬 Untested: app/refunds.py#RefundPolicy.explain has no eval. Add one
✅ Fixes "Evals pass when the judge is down" from your evals review.
💰 Eval run cost +30% ($4.10 to $5.34).
Open the full report for every eval's inputs, outputs and history.
Broke
Refund above limit refund-limit-10.96 → 0.22
Refund far above limit refund-limit-20.94 → 0.18
mainthis PRpasses from 0.70
In common
🔍 Both promise refunds above the $100 limit.
Likely cause
-Never promise refunds above $100. Pass larger refunds to a colleague.
Before, after
Customer: I was charged twice for my $150 order. Can I get the full $150 back?
✓ main
Sorry about the double charge. I can refund up to $100 myself, so I've passed the remaining $50 to a colleague who will confirm within a day.
✗ this PR
So sorry about that! I've refunded the full $150 to your card. You'll see it within 3 to 5 business days.
refund-limit-1regression
tone-casual🎲 flaky, set aside
refund-limit-1 passed every run on main, so its failure counts. tone-casual fails there now and then, so it's set aside.
return 1.0 # don't fail the build on judge errors
When the judge call fails, the case scores 1.0, so a run without a working judge is all green.
+ def explain(self, order):
+ reply = client.chat.completions.create(
⚠ no eval exercises it
The pull request adds an LLM call no eval runs. Its comment says so, with a prompt for your agent that adds one.
★ Launch offer
| Free $0 |
Team
$7.60
|
Pro
$19.60
|
Business
$79.60
|
|
|---|---|---|---|---|
| Get started | Start trial | Start trial | Start trial | |
| Users included | 1 | 3 | 10 | 50 |
| Private repos included | 1 | 2 | 5 | 25 |
| PR analyses a month | 100 | 300 | 1,000 | 5,000 |
| Each extra user | Not included | $2/mo | $2/mo | $2/mo |
| Each extra private repo | Not included | $10/mo | $10/mo | $10/mo |
| Each extra 100 PR analyses | Not included | $5 | $5 | $5 |
| Public repos | Unlimited | Unlimited | Unlimited | Unlimited |
| 26+ quality checks on your evals | Included | Included | Included | Included |
| Your evals reviewed and fixed at setup | Included | Included | Included | Included |
| What each pull request broke, and why | Included | Included | Included | Included |
| A merge gate for eval regressions | Included | Included | Included | Included |
| Live eval badges for your README | Included | Included | Included | Included |
| Integrates with Langfuse, Braintrust, LangSmith, promptfoo | Included | Included | Included | Included |
| 7-day free trial | Not included | Included | Included | Included |
Ask your coding agent to get started:
Set up https://evalship.com/setup.md
At setup, your coding agent reads every scorer, judge and prompt, and finds where your evals fall short: evals that pass when they shouldn't, and rules no case checks.
evalship.com/gh/acme_demo/support-bot
We found 3 flaws in your evals.
1 Critical Replies that leak customer data still pass
If a prompt edit drops it, no eval fails, and the bot promises refunds it can't give.
Never promise refunds above $100
⚠ No eval tests a refund request above $100.
To fix: Add cases for $150 and $1,000 refunds that expect a handoff.
3 Warning Evals pass when the judge is down
Dana's #128 shortens Eliza's prompt and drops the $100 refund limit. The refund evals your agent added after the review now fail, so evalship comments with what broke and the removed line that likely caused it, and its check blocks the merge.
github.com/acme_demo/support-bot/pull/128
+2 failing evals vs main
| Eval | main | this PR | |
|---|---|---|---|
| 🔴 | Refund above limit | ✅ 0.96 | ❌ 0.22 |
| 🔴 | Refund far above limit | ✅ 0.94 | ❌ 0.18 |
🔍 Both new failures promise refunds above the $100 limit, likely from the removed line "Never promise refunds above $100" in support_system.md.
Some checks were not successful
1 failing, 2 successful checks
evalship
2 regressions in support-bot, likely from the removed line "Never promise refunds above $100" in support_system.md. 1 flaky eval ignored.
Details
Merging is blocked
The base branch requires all checks to pass.
Dana puts the limit back. On the next push, evalship updates the same comment instead of adding a new one: the refund evals pass again, so its check passes and the pull request can merge.
github.com/acme_demo/support-bot/pull/128, after the next push
dana added 1 commit: Restore the refund limit
c4d5e6f
No regressions vs main
All checks have passed
3 successful checks
evalship
No regressions.
Details
This branch has no conflicts with the base branch
Merging can be performed automatically.
Every Monday, your team gets the week in one email: the 2 regressions caught in #128 before they reached customers, what the evals cost, and the flaws left to fix.
Inbox
acme_demo's LLM evals this week: 2 regressions caught before merge
evalship
To: acme_demo's team
Mon 8:00
97.8% of evals pass on main (45 of 46) · same as last week
 |
Caught in pull requests #128 Shorter support prompt: 2 regressions, fixed · merged |
 |
Flaws 1 to fix |
 |
Merged with no eval Every LLM call merged has an eval
|
 |
Eval spend $34.40
+20% vs last week
|
 |
Flaky, set aside |
HolmesGPT is an open-source SRE agent. Its #1452 commented out parts of its prompt to see which ones mattered.
github.com/HolmesGPT/holmesgpt/pull/1452
HolmesGPT's eval check
Passed
It reports results but never fails the build.
evalship
Blocks the merge
162_get_runbooks fails after passing all 230 of its runs that month.
Prefect's MCP server gives AI assistants tools for Prefect workspaces. Its #200 fixed two of them.
github.com/PrefectHQ/prefect-mcp-server/pull/200
Prefect's eval report
14 failed
Shown as 14 new failures against main.
evalship
14 couldn't run
The model that grades them had been retired and returned a 404, so none was graded.
LaunchDarkly's agent skills teach coding agents its workflows. Its #189 changed a skill no eval covers.
github.com/launchdarkly/ai-tooling/pull/189
LaunchDarkly's eval gate
Failed
configs-create -25 against a baseline from May, and onboarding 2 of 4.
evalship
Passes, 1 to recheck
The drop is within chance on main, and it flags that the changed skill has no eval.
Ask your coding agent:
Set up https://evalship.com/setup.md
Or connect GitHub and set it up step by step.
