◆ This page is unfinished. Content and design are still in progress.

Uncontaminatable benchmarks

Agent benchmarks face a basic tradeoff between openness and contamination. Publish the tasks, and within months they can enter training corpora. Once that happens, a score may reflect memorization as well as capability. Keep the tasks private, and they are harder to contaminate, but the benchmark becomes difficult to inspect or reproduce.

Private benchmarks are also hard to trust. Outsiders cannot examine the full task set, audit the grader, or reproduce a published score from the same inputs. Even public benchmarks can depend on a particular harness, environment, or scoring rule that does not resemble how people use agents day to day. A short, isolated task says little about an agent's ability to explore a codebase, use tools, recover from mistakes, and keep improving over a long session.

Evaluation can be coarse even when the task is public. In benchmarks such as Terminal-Bench, an implementation is usually reduced to whether a task-specific grader passes. That grader can reject a correct solution because it expects a particular output or implementation detail. It can also accept a flawed solution because its tests miss the failure. The score then measures agreement with the grader rather than the full quality of the work.

Fixed time limits make that mismatch worse. They penalize an agent for spending more time on a hard problem, even though persistence and long-horizon work are capabilities we want to measure. A useful benchmark should record how quickly an agent improves without treating a deadline as the definition of success.

Pass/fail scoring loses information too. Imagine an 89-task benchmark ordered by difficulty. An okay model and a frontier model may both solve the first 76 tasks, while neither can solve the final 10. Their entire capability gap is then represented by tasks 77, 78, and 79. The other 86 tasks tell us almost nothing about the distance between them. A large improvement in capability appears as a difference of only three task completions.

Fixed task sets eventually saturate. Once frontier models solve nearly every task, only a few unresolved tasks remain to separate them. Scores bunch together, further progress becomes hard to measure, and the benchmark must be replaced with a harder task set.

An optimization benchmark does not end when an agent passes the task. Its limit is the mathematical floor or ceiling of the objective, and that limit may not even be known. Every verified improvement moves the known frontier. A stronger agent extends the benchmark instead of exhausting it.

Continuous, analog scoring gives every valid improvement a value. Evaluation follows directly from the task definition: exhaustive correctness is the gate, and performance under a published deterministic cost model is the score. There is no coarse rubric or sampled proxy between the implementation and its result.

The design below addresses the other problems as well: remove the answer key, publish everything, verify results deterministically, and record time rather than capping it.

With no answer key, training on the benchmark stops being cheating, because it is indistinguishable from acquiring the skill. That is why everything can be public, including complete transcripts of the best runs. A model trained on a winning transcript learns the techniques in it, but replaying them only matches the old score. Beating it requires improving past what was memorized, which is what the benchmark measures.

One caveat: this guarantee protects the frontier, not the ordering below it. A model that memorized the best published transcript can replay it and match the record without having the skill, which distorts comparisons among sub-frontier models. Two things expose this. First, the spec makes tasks cheap to generate, so a fresh suite can be produced at any time: a model that scores well on the old suite but poorly on a newly generated one has memorized rather than generalized. Second, the score-over-time curve is diagnostic. An agent that acquired the skill climbs steadily as it finds improvements, while a regurgitated transcript jumps straight to the published score and flatlines. A record without a climb is a memorization signature.

The spec

Seven properties, each guaranteed by the structure of the task rather than by policing.

The last property comes from a lesson learned in practice. An early cost model counted only instructions inside the target function, and a run legally scored +11 by building a 64 GiB answer table in a library constructor, reducing the function to a table lookup. That was within the rules and outside the intent: work had moved across the measurement boundary. The systematic fix is that the metric must be a closed system. Just as exhaustive verification leaves no input outside the correctness gate, full pricing leaves no work, state, or resource outside the cost gate.

jcode bench

jcode bench v1 is the first benchmark built to this spec. It is deliberately small: three tasks that are inexpensive to run. A task hands the agent a working, tested implementation of a real primitive (shortest round-trip float printing, JSON string unescaping, UTF-16 transcoding), together with its exhaustive verifier and a deterministic cost model. The task is to make it faster while staying correct on every possible input.

The tasks are not puzzles. Each one is a hot primitive that real systems execute constantly: float printing sits inside every JSON serializer and logging pipeline, string unescaping runs on every parsed JSON document, and UTF-16 transcoding is on the boundary of every JavaScript engine and Windows API call. A genuine improvement found during a run is directly useful in the real world, so effort spent climbing the benchmark is effort spent on work that matters.

Exhaustive verification is what closes the overfitting loophole. A sampled test suite can be gamed: measure grep by timing it on the Linux repo, and an implementation can win by tuning to that repo specifically. Here the verifier covers the whole input space, all 2³² floats in the float-printing task, so a submission is either correct everywhere or rejected. Correctness is a gate, and the only score is speed under the cost model.

Scoring

Improvements to optimized code are multiplicative, so the score is logarithmic: doublings of improvement over the given implementation.

A grade takes seconds, and the harness tracks the best score continuously, so a run produces a score-over-time curve rather than a number at a deadline. The curve shows how fast an agent climbs and how far it goes when simply left to work.

Aggregating across tasks

The combined score for a run is the geometric mean of its task scores: typical doublings achieved per task. Tasks have very different headroom (float-print scores pass +12 while utf16-transcode stays under +3), and an arithmetic mean would let the widest task dominate. The geometric mean is scale-invariant per task, so doubling any one task score moves the aggregate equally, and a weak task drags it down hard. It is also a pure function of the run's own scores: no reference constants and no dependence on other entries, so it is stable whether the leaderboard has one entry or a thousand. 2 to the aggregate reads as the typical per-task speedup.

jcode bench is in development. Task specs, graders, the harness, and complete run transcripts are published as they land. There is no hidden test set.