errorbars

Error bars for LLM evals. Know whether model B is actually better, or you're reading noise.

LLM eval results get reported as bare accuracies — "model B scored 71.2%, up from 69.7%" — with no error bar, no significance test, and no accounting for the fact that the questions came in correlated groups. On a 500-question benchmark, a 1.5-point gain is very often noise: at a 50% baseline you'd need roughly a 9-point gap to reliably detect anything at 80% power, α=0.05 (errorbars power --n 500 --baseline 0.5 → minimum detectable effect ≈ 8.9 points). Evan Miller's "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations" (arXiv:2411.00640, 2024) lays out the right practice — CLT and clustered standard errors, variance reduction through pairing, power analysis — and this package turns it into a one-liner.

The source checkout can also open a comparison at the question level: errorbars compare ... --html comparison.html produces one offline file with shared and missing questions, individual generations, source records and cluster sensitivity. Native lm-eval and text-only Inspect imports check recorded question content before pairing reused IDs. Inspect imports also reject incomplete runs; the retained example shows a meaningless paired result (p = 0.000000644) replaced by 24 content conflicts. These additions are unreleased; install this checkout to use them. Comparison walkthrough.

SWE-bench Verified, with error bars runs this package over the per-task results of 172 complete public submissions. The short version:

• Among the 77 complete submissions scoring 60% or more, few gaps under 3 points are distinguishable (paired standard error about 1.8 points): 1 of 842 such pairs differs at the 5% level. Two submissions tie for the lead at 79.2%, and the leader cannot be told apart from ranks 2 to 9; only 7 of its 24 comparisons with ranks 2 to 25 differ after Holm's correction.
• The top score went up 30 times in two years. Six of those new records were significantly above the record they replaced.
• The 500 tasks come from 12 repositories and 231 are Django. For a claim about other codebases the benchmark has about 3.3 effective degrees of freedom.

The source checkout also has a page that runs the same paired test in the browser for any two of those submissions, or for a run of your own pasted in as a list of resolved task ids (web/swe-bench.html; its statistics are checked against this package on real pairs).

Running it there also showed that this package's own clustered test was wrong for that benchmark, and too slow for a board of that size. Both are fixed in this checkout (unreleased; see the changelog).

The source study and fixed-cohort browser view exclude one public run with only 441 recorded task outcomes. The CSV converter retains those 441 observations, instead of adding 59 unreported failures. Native evidence and comparison replay.

Run errorbars leaderboard on a synthetic benchmark (below) and one apparent 7-point win — tuned-70b over baseline-70b, 0.690 vs. 0.620 — is not statistically significant once it has a proper error bar: the paired test already gives p = 0.11. Accounting for the fact that questions come 5-to-a-passage, and Holm-correcting across all 6 pairwise comparisons on the board, pushes that to p = 0.32. A naive leaderboard that ranks by bare accuracy would still have shipped the gap as a win. The full walkthrough, with every number copied from a real command, is in examples/README.md.

pip install errorbars
errorbars power --delta 0.03 --baseline 0.5

Questions needed: 4361

That's how many questions you'd need to reliably detect a 3-point accuracy gap at a 50% baseline, with the defaults (80% power, α=0.05, no pairing correlation, no clustering). Clone the repo to try it against real per-item scores:

git clone https://github.com/antonsoo/errorbars && cd errorbars
errorbars leaderboard examples/data/reading_comprehension.csv

The source corrections described here are not in the tested PyPI 0.2.4 build. To use them, install this verified public revision:

python -m pip install --force-reinstall "errorbars @ git+https://github.com/antonsoo/errorbars.git@45cc476a79166875d8f06b09d7ff837bb83d2540"

--force-reinstall replaces an existing 0.2.4 installation; this source still declares that version. Public source and pip upgrade replay verify all 29 installed package files, grouped summaries and the recorded-cohort comparison. The live build receipt checks the browser separately. These are separate from a new PyPI release.

• summarize — mean, SE, and 95% CI. This checkout makes supplied multi-question clusters determine the primary CR2 interval, consistently with leaderboard. --ci chooses an explicitly labelled independent-question diagnostic when groups are present; otherwise it selects CLT, Wilson or bootstrap as before. Insufficient questions or groups retain the mean with unavailable uncertainty. Retained SWE-bench reproduction and independent covariance audit. ICC, design effect and repeated-generation variance diagnostics remain available.
• compare — selects exact binary, clustered, or paired-t inference from the shared observations and retains diagnostic SEs, intervals and p-values; the correlation between the two models' per-question scores and how much pairing shrank the SE vs. an unpaired comparison; a cluster-robust paired SE/CI/p-value when clusters are present, which stays valid when the clusters are few or very unequal in size; exact McNemar test for binary scores. The source checkout also prevents repetition counts from manufacturing a gain: identical fractional grades previously produced p=0 after averaging.
• Comparison evidence (source checkout) — compare --html report.html opens a complete question ledger with both run means, observed generations and source line/record locations. Find lower scores and missing questions, inspect which passages drive the result, and download complete JSON or filtered CSV. One file, including fonts; no server or upload.
• Large binary evaluations (source checkout) — exact McNemar tails use scaled probability recurrences, avoiding repeated arithmetic on giant integers. A leaderboard also reuses its just-validated rows when averaging questions. Public inputs remain validated on every call. The throughput study retains an actual 1.6-million-row CLI replay, independent probability checks and the limitations.
• leaderboard — every model with its CI and exact ranks without a significant Holm-corrected paired difference. Untested pairs remain explicit; rank gaps are preserved. This checkout selects exact McNemar for one binary score per shared question, clustered t for grouped questions, and paired t for continuous or repeated scores. The table and JSON name the selected test; a shared group does not establish equivalence. Counterexample, exact checks, and public-data audit. Supplied clusters also determine the displayed model confidence intervals; the old independent-question estimate remains a labelled JSON diagnostic. Why exact rank sets matter on 173 public submissions. Forest plots name the interval method and wrap full model names. The CLI writes SVG without dependencies; Python callers can also use the matplotlib extra. Partial cohorts sharing only one independent group now keep an explicit untested pair while the rest of the board remains usable. Question identity checks index each model once. Failure capture and timing evidence.
• Metric direction (source checkout) — mean losses rank lower first when the native results record that direction. CSV/JSONL can retain higher_is_better, and leaderboard --score-direction lower supplies it for undeclared scores. Recorded non-mean aggregations are refused; averaging a harness's perplexity inputs does not estimate perplexity. Counterexamples and usage.
• Incomplete leaderboard runs (source checkout) — a run covering only one independent cluster, or one question with continuous or repeated scores, keeps its observed mean and counts. Its SE and interval are explicitly unavailable; valid comparisons between other runs still work. JSON records interval_status and nullable uncertainty fields; terminal tables and both forest-plot formats identify the missing interval. SWE-bench partial-run reproduction.
• power — number of questions needed to detect an effect δ at a given α and power, or the minimum detectable effect for a given n, accounting for pairing correlation, repeated sampling, and cluster design effect. Repeated-answer plans in this checkout require an explicit within-question correlation and retain the variance that more answers cannot remove. Real-outcome check and budget counterexample.
• CLI — errorbars summarize|compare|leaderboard|power|import. Tables are plain aligned text on a bare install (numpy is the only dependency) and rich tables with pip install "errorbars[cli]"; either way a model name is printed whole, and a table written to a pipe is as wide as it needs to be. --json for scripting.
• Reads what harnesses write — a harness leaves one log per model, so every command takes one or more files or directories and reads each as what it is: lm-evaluation-harness --log_samples output, Inspect AI .eval logs, or the CSV/JSONL format below. errorbars compare out/model-a out/model-b is the whole comparison (see below).
• Web calculator — plan the questions needed for an accuracy improvement or the gap a fixed budget can detect. Exact editors, a computed budget table, pairing sensitivity, explicit assumptions, JSON plans, and reproducible CLI commands. The TypeScript formulas are checked against Python-generated vectors. Planning guide.
• Typed Python ≥3.10, numpy the only runtime dependency; pandas and matplotlib are optional extras; scipy/statsmodels are test-only oracles, never imported at runtime.

errorbars summarize examples/data/reading_comprehension.csv --model tuned-70b

summarize: tuned-70b
metric             value
------  ----------------
n                    200
mean              0.6900
SE                0.0328
95% CI  [0.6257, 0.7543]
method               clt

clustering diagnostics
metric                         value
------------------  ----------------
n clusters                        40
ICC                           0.0710
design effect                  1.284
clustered SE                  0.0372
clustered CI        [0.6148, 0.7652]
degrees of freedom              39.0

errorbars compare examples/data/reading_comprehension.csv \
  --model-a tuned-70b --model-b baseline-70b

mean diff (A - B)                 0.0700
paired SE                         0.0440
95% CI                  [-0.0167, 0.1567]
p-value                           0.1131
correlation(A, B)                 0.1434
variance reduction from pairing     14.3%
McNemar exact p-value             0.1405

errorbars power --delta 0.05 --baseline 0.65 --rho 0.14 --cluster-deff 1.28

Questions needed: 1573

Every number above is copied verbatim from running these commands against examples/data/reading_comprehension.csv (synthetic — see below). The full leaderboard output and the reasoning behind each step are in examples/README.md.

Long-format CSV or JSONL, one row per observation:

Column names are configurable (--question-col, --cluster-col, etc., or ColumnMap in Python). Scores can be binary (0/1) or continuous. cluster_id groups correlated questions (e.g. several questions per reading passage); sample marks repeated generations of the same question.

Each question has equal weight in summaries, leaderboard means, and paired comparisons. Repeated generations are averaged within a question first. n counts questions; JSON also retains n_observations, and tables show the generation count when repetitions are present. For example, ten successful generations of q1 and one failed generation of q2 have a question mean of 0.5, rather than a draw-weighted 10/11. The runnable synthetic example is in the repeated-generation walkthrough.

Identifiers must be nonempty scalar values. Duplicate CSV or DataFrame columns, JSON fields, explicit missing sample/cluster labels, and contradictory cluster assignments are rejected. A question's cluster is shared across models. When combining files, missing cluster metadata can be resolved from another file for the same question; every other question needs an explicit assignment. load_dataframe allows duplicate row indexes; observation identity comes from the configured columns. Rename or explicitly select duplicate columns before loading. CSV quote errors, including an unfinished quoted field at EOF, report the input path and physical line. A missing final newline is valid; these checks cannot detect missing whole records. See the table import reproduction.

Every command takes several files (errorbars leaderboard model-a.csv model-b.csv ...) and puts their rows together. The same model, question and sample in two files is refused, as it is within one file: counted twice it would inflate n and shrink every error bar.

In the source checkout, optional question_hash signatures are retained through canonical CSV/JSONL import/export (--question-hash-col maps another column name). Known conflicting signatures for a shared ID stop paired comparisons and leaderboards; conflicting repeated samples within one model are refused before averaging. Missing signatures stay unchecked. Use the same signature scheme in both runs; a signature is an assertion about question content, not proof that the scoring rules match.

The source checkout also preserves optional scorer and scorer_config columns. Known differences in scoring declarations block comparison and ranking, including after CSV conversion. A recorded Inspect counterexample previously reported a 33.3-point improvement on identical answers when only the grader changed. Missing declarations stay unchecked; matching declarations do not verify scorer implementations or unrecorded settings. See scoring declarations.

A harness writes one log per model and per task, so "is B better than A" starts from two files. summarize, compare and leaderboard read them as they are:

# lm-evaluation-harness: lm_eval run --model hf --model_args pretrained=<model> --tasks copa \
#                          --log_samples --output_path out      (one directory per model)
errorbars compare out/sshleifer__tiny-gpt2 out/hf-internal-testing__tiny-random-gpt2

compare: sshleifer/tiny-gpt2 vs hf-internal-testing/tiny-random-gpt2
metric                                                             value
-----------------------------------------------------  -----------------
n (shared questions)                                                  20
mean(sshleifer/tiny-gpt2)                                         0.6000
mean(hf-internal-testing/tiny-random-gpt2)                        0.8000
mean diff (A - B)                                                -0.2000
paired SE                                                         0.1376
95% CI                                                 [-0.4881, 0.0881]
p-value                                                           0.1625
correlation(A, B)                                                 0.1021
unpaired SE (for reference)                                       0.1451
variance reduction from pairing                                    10.0%
McNemar discordant (A wrong/B right, A right/B wrong)              6 / 2
McNemar exact p-value                                             0.2891

That is a real run, committed under tests/fixtures/lm_eval_output/ (two tiny models, 20 COPA questions). lm-eval's own table says 0.6 ± 0.11 and 0.8 ± 0.09; the paired test on the same 20 answers gives p = 0.16, so the 20-point gap is not evidence yet. (Rows trimmed; the command prints a few more.)

errorbars leaderboard out                       # every model under the output directory
errorbars compare logs/run-a.eval logs/run-b.eval           # Inspect AI logs
errorbars compare greedy=logs/a.eval sampled=logs/b.eval    # two runs of one model, told apart
errorbars import lm-eval out -o all.csv         # or write the canonical CSV once and keep it

In this source checkout, CSV, SVG and HTML exports refuse input files and their aliases, including a results sidecar used to infer an lm-eval model name. Exports replace their destination only after a complete write; a failed write keeps an earlier export intact. Symbolic-link destinations are refused. New and replaced files are private on POSIX (mode 0600). Actual CLI overwrite and partial-write replay. This correction is unreleased.

• lm-eval: a samples_<task>_<timestamp>.jsonl file, a model's directory, or the whole --output_path. The samples file does not name its model; lm-eval writes it beside a results_<timestamp>.json that does, and that is where the name comes from. A samples file moved away from it has to be named: my-model=path/to/samples_copa_....jsonl. In this source checkout, a renamed file or a results file with a different timestamp also needs an explicit name; a nearby run cannot identify these samples. When a directory holds several runs of a task, the latest is used and a note says so. Question ids are <task>-<doc_id>, so several tasks of one model add up to one benchmark.
• Inspect AI: .eval (or .json) logs; the model is the one in the log. Epochs (--epochs N, repeated sampling of the same input) land in the sample column automatically. In this checkout, imports require a successful run with all planned samples completed and recorded.
• With exactly two models in the input, compare needs no --model-a/--model-b: A is the first one given. When the two cover different selected questions (such as different --limit values), the comparison uses the shared ones and reports how many were left out.

--metric (lm-eval) picks which computed metric to use as the score when a task reports more than one (e.g. acc vs. acc_norm). The source checkout requires this choice when several metrics are available; the released package defaults to the first one. The chosen metric must be declared on every selected record, and native logs with shared questions must use the same metric. Metadata such as doc_id cannot be selected as scores. In a controlled lm-eval capture, changing only metric order previously created a 100-point gap and p = 0.000000119 between identical responses. Explicit selection gives zero difference. These importer corrections are unreleased. --filter (lm-eval) picks one filter for a task that scores every question under several: gsm8k_cot_self_consistency logs each question three times (score-first, maj@8, maj@64), and reading those as three times the questions would shrink every error bar, so the import stops and asks. --scorer (Inspect) picks one scorer for tasks with more than one.

Named Inspect values are supported in this checkout (unreleased). A single scorer can return, for example, correctness, brevity and word count for every answer. Select the exact top-level key with --score-key; errorbars does not choose the first key or turn the dictionary into a zero:

errorbars compare A=examples/inspect-values/logs/baseline.eval B=examples/inspect-values/logs/candidate.eval --score-key correct --html correct.html
errorbars compare A=examples/inspect-values/logs/baseline.eval B=examples/inspect-values/logs/candidate.eval --score-key words --json

These retained Inspect runs contain controlled synthetic responses, not model performance results. They exercise three grading dimensions and two epochs. The selected value appears in CLI output, comparison evidence and CSV exports; comparing different dimensions blocks inference even when scorer names and parameters match. A missing or unscored selected value rejects the import. Reproduction and independent checks.

Both adapters were built and tested against real, unedited output — not from memory: lm-eval 0.4.13 (lm_eval run --model dummy --tasks copa --limit 20 --log_samples ..., a multi-metric arc_easy run, a three-filter gsm8k_cot_self_consistency run, and the two-model output directory above) and inspect-ai 0.3.268 (small tasks through the built-in mockllm/model provider: single epoch, two epochs, and a scorer with named values). The exact log files are committed as test fixtures (tests/fixtures/samples_*.jsonl, tests/fixtures/lm_eval_output/, tests/fixtures/inspect_*.eval) and re-parsed in tests/test_adapters.py and tests/test_inputs.py on every run. The Inspect adapter reads logs with Inspect's own inspect_ai.log.read_eval_log and inspect_ai.scorer.value_to_float rather than hand-parsing its binary .eval format, and needs the inspect extra: pip install "errorbars[inspect]". The lm-eval adapter has no extra dependency — --log_samples is already plain JSONL.

Default lm-eval generation imports also retain recorded grading parameters. Changing only exact_match(ignore_case=...) previously produced an apparent 33.3-point model gain on identical answers (p=0.0078125). Known conflicts now withhold inference and ranking, including after canonical CSV export; prompt variants and changed answers under a common grader remain usable. Native captures, controls and limits.

The source checkout retains the selected lm-eval metric's higher_is_better declarations and checks its recorded aggregation. A controlled native capture previously ranked mean loss 0.790625 ahead of 0.25; it now ranks lower loss first without negating scores. Known non-mean aggregations stop before analysis or export: selecting native perplexity previously displayed negative log-likelihood means instead of the harness's positive perplexities. Missing declarations stay unknown; a lower-is-better declaration needs a recorded mean aggregation before its direction can be applied to the per-item values. Retained captures, policy and commands.

Inspect imports refuse unscored samples, missing grades and unrecognized grade strings. The source checkout also checks status, planned/completed/recorded counts, selected IDs, epochs and invalidations. Cancelled, failed, drained or incompletely recorded evaluations cannot quietly become smaller successful datasets. A completed --limit or --sample-id selection is accepted. One scorer must be present across the entire imported log. Valid C/I/P/N, boolean representations and finite numeric scores keep Inspect's default conversion semantics. Recorded custom metric to_float callbacks are refused in this checkout: their opaque callable names do not establish the grade mapping. A native accuracy of 1.0 could otherwise become 0.0 and reverse the leaderboard. Export documented numeric per-question scores as CSV/JSONL for these logs. Native reproduction and selection scope. Finish or retry an incomplete evaluation before importing its log. lm-eval records require a valid doc_id; row order is never used to fabricate question identity.

The source lm-eval adapter also reconciles the selected filter's record count with n-samples[task].effective in a matching-timestamp results file, even with NAME=PATH or --model. A truncated native COPA capture previously reported 100% after its eight incorrect answers were removed, although the unchanged results file recorded 20 evaluated questions and the complete capture scored 60%. It now stops before analysis or export. The original dataset count is not used: completed limited runs remain valid. Missing sidecars or missing count declarations stay unchecked, and count agreement does not authenticate the cohort. Before/after installed CLI evidence.

The source lm-eval adapter also computes a versioned SHA-256 signature from each logged document and target. It ignores prompt arguments and responses so prompt variants remain comparable. It computes the signature from the contents, rather than trusting the stored doc_hash. Empty/missing documents or null/missing targets leave identity unchecked. Text-only Inspect samples are signed from the logged dataset input, ordered choices and targets. Solver messages and generated message IDs are excluded, preserving ordinary solver prompt experiments. Changing the dataset input itself triggers a conservative conflict; multimodal or externally backed samples remain unchecked. Signatures survive CSV imports. Identity scope and recovery.

Every statistic is implemented from scratch on numpy + the standard library (statistics.NormalDist for normal quantiles; Student-t tails from a continued-fraction incomplete beta function) — no scipy or statsmodels at runtime. Full derivations with references are in docs/formulas.md:

1. CLT and Wilson confidence intervals for a mean
2. Cluster-robust standard errors: the bias-reduced CR2 sandwich estimator with Satterthwaite degrees of freedom, and why the usual CR1 on t(G − 1) is not used
3. Intraclass correlation and Kish's design effect
4. Within/between-question variance decomposition for repeated sampling
5. Paired comparisons, variance reduction from pairing, and exact McNemar
6. Holm-Bonferroni correction and maximal-clique grouping for leaderboards
7. Power analysis (sample size and minimum detectable effect) under pairing and clustering

• The cluster-robust SE and its degrees of freedom match the general matrix definitions, evaluated independently, to 1e-9 on unequal cluster sizes (tests/test_few_clusters.py); the classic CR1 estimator, still available as kind="CR1", matches statsmodels' OLS(..., cov_type="cluster") to 1e-9. Wilson intervals match statsmodels.stats.proportion_confint to 1e-9; the paired t-test matches scipy.stats.ttest_rel to 1e-7 at any n, and McNemar's exact test is cross-checked against statsmodels. See tests/test_stats_vs_oracles.py and tests/test_compare_vs_oracles.py.
• Monte Carlo coverage tests (tests/test_coverage_montecarlo.py) confirm nominal 95% CIs cover the true parameter close to 95% of the time — for CLT, Wilson, bootstrap, and cluster-robust intervals — with a tolerance sized to the trial count so it won't flake.
• Paired comparisons use Student's t with n − 1 degrees of freedom, like scipy.stats.ttest_rel. Clustered SEs use t with Satterthwaite degrees of freedom: G − 1 for G clusters of equal size, fewer when sizes differ (3.3 for SWE-bench Verified's 12 repositories). The usual CR1 estimator on t(G − 1) rejects a true null two to three times too often on those sizes; this one rejects 3.0% to 5.2% of true nulls on those sizes, and is conservative (3.0% for a 5% test) when there is in fact no clustering. The unclustered and clustered tests answer different questions (more questions from the same clusters; new clusters), and with few clusters the second can say very little. The per-model clt interval stays normal-based, which is only right once n is in the dozens.
• Repeated-generation intervals use the distribution of question averages; Wilson is reserved for single binary observations per question. CLT and percentile bootstrap estimates remain approximate, especially with very few questions. Within/between decomposition retains the original draws. One independent cluster cannot estimate a cluster-robust SE and is rejected.
• Zero-variance paired differences carry explicit warnings: nonzero differences use the t-test's p=0 limit, while identical scores use p=1 by convention. Point intervals do not establish certainty about the population. Exact McNemar results remain available for binary pairs; scores merely near 0 or 1 remain continuous. Nonzero score units are rescaled before variance calculations to avoid underflow in SEs and paired tests.
• Pairing diagnostics preserve unavailable values: correlation is null when either vector is constant, and variance_reduction is null when both are constant. The CLI prints unavailable; downstream code must check for None before formatting these fields.
• Repeated-answer power plans require --repeat-correlation when --samples-per-question > 1 in this checkout. The old independence assumption understated the variance at 100 answers per question by 41-48 times in two published GSM8K runs. The retained study checks variance on later draws of the same questions; it does not validate power on new questions. The equal-variance normal approximation still needs defensible pilot assumptions, and pairing correlation must describe the averages at the requested repeat count. This repair is unreleased.
• Leaderboard groups are maximal cliques of the "not significantly different" graph; models can appear in several overlapping groups. Their count can be exponential. The source checkout bounds enumeration by 4,096 groups and 50,000 search states. If that summary cannot finish, groups is null, group_status explains why, and all model estimates, pairwise tests and exact rank comparisons remain available. Non-significance does not establish equivalence. Reproduction, JSON contract and limits. This repair is unreleased.
• Verification evidence records the current planning checks, supported environments, browser coverage, and limits of the audit.
• Data-integrity verification records the subsequent repeated-generation, paired-test and metadata audit. Those changes are local and unreleased.

web/ is a static Vite + TypeScript "how many eval questions do I need?" calculator (live demo). Its power formulas (web/src/power.ts, web/src/normal.ts) are a direct port of src/errorbars/power.py; Python generates JSON test vectors (scripts/export_test_vectors.py → web/test-vectors.json) that the TypeScript test suite checks against. The Python suite also checks that the committed vectors match the current implementation.

Choose Questions needed or Gap my budget can detect. Number fields keep exact inputs; invalid edits disable results and downloads until corrected. The chart and table contain calculated values only, and a comparison at correlation zero exposes how much the answer depends on pairing. Impossible combinations of baseline, improvement and binary correlation cannot be exported. Budget mode finds the smallest feasible detectable gap; negative correlation can impose a minimum gap regardless of budget. Small question/group counts carry explicit warnings. The normal approximation remains a planning estimate.

Download plan (JSON) retains inputs, units, results, assumptions and a complete errorbars power command. Calculations and downloads stay local and work offline after load; only the theme is persisted. See Evaluation planning for the artifact schema, browser input ranges, and statistical limitations.

uv sync --group dev --extra all
uv run pytest
uv run ruff check .
uv run mypy src/errorbars

See CONTRIBUTING.md for the web calculator's dev loop and guidelines.

Issues and PRs welcome. Any new statistic needs a test against an independent oracle (reference library, closed form, or Monte Carlo simulation) — see tests/ for the pattern.

@misc{miller2024addingerrorbarsevals,
  title  = {Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations},
  author = {Evan Miller},
  year   = {2024},
  eprint = {2411.00640},
  archivePrefix = {arXiv},
  primaryClass  = {stat.AP},
  url    = {https://arxiv.org/abs/2411.00640}
}

MIT © 2026 Anton Soloviev. The browser quantile adaptation and local fonts retain their upstream licenses; see third-party notices.

_{Part of Officina, a set of small open-source tools by Anton Soloviev.}