Running your own experiments

This folder is a measurement rig, not a library. The probes in probes/ ask a typed oracle real questions against real data and write raw rows; separate report scripts judge them. Everything here is set up so you can point the same machinery at your question and your data.

Read FINDINGS.md first for what has already been measured — it carries the recommended configuration and a list of retracted claims. CLAUDE.md carries the summary of what TypeSafe is. For the short version to hand someone else, there is a three-minute walkthrough at When a number beats a sentence.

Node 20.6+ (the scripts use node --env-file), then:

npm install
cp .env.example .env     # then fill it in

.env needs:

You can run the whole Jev half with TYPESAFE_KEY alone. Skip the Anthropic key until you actually want a baseline to compare against.

npm run dataset          # build the binary corpus -> .sandboxes/episodes.json
npm run run              # arm A (Jev) + arm B (Haiku via `claude -p`)
npm run run:api          # arm C (Haiku via the API — the honest baseline)
npm run report           # accuracy, confidence separation, calibration, cost

npm run run:repeat       # 5 independent runs of arm A
npm run stability        # per-item churn across those runs

npm run wording          # the broad/narrow x Choice/Noul 2x2
npm run wording:report

npm run ladder           # the 5-rung ordinal ladder -> .sandboxes/ladder.json
npm run score            # Jev Score, 3 runs
npm run score:haiku      # the same rubric asked of Haiku
npm run score:report

Run data is regenerable and lives gitignored in .sandboxes/. Nothing in probes/ judges anything — runners record verdicts, confidences, latencies and tokens; the *-report.mjs scripts do all the judging. Keep that split when you add your own: it means you can re-judge a run without paying for it again.

Every runner reads the same environment variables.

A second corpus, end to end:

PROBE_REPO=/path/to/other-repo PROBE_OUT=.sandboxes/episodes-two.json npm run dataset
PROBE_DATASET=.sandboxes/episodes-two.json PROBE_TAG=two npm run wording
PROBE_TAG=two npm run wording:report

Caveat: probes/report.mjs reads fixed filenames (results-typesafe.jsonl, results-haiku.jsonl, results-haiku-api.jsonl) and does not honour PROBE_TAG; only wording-report.mjs does. For a tagged run through run-arms.mjs, either move the files aside between corpora or teach report.mjs the tag.

Always smoke-test first: PROBE_LIMIT=6 npm run run costs cents and catches a malformed question before you pay for 173 items three times.

The single most load-bearing choice in this folder. Both corpora are labelled by construction, not by hand — a commit's message paired with its own diff is a match; paired with a different commit's diff it is not. No judgment under test could contaminate them, and there is no annotator to disagree with.

build-dataset.mjs also builds in a difficulty axis it never tells the model about (hard negatives share a file, easy ones don't). That axis is what makes confidence testable: a number that tracks a difficulty it was never shown is carrying information, and that argument does not depend on the labels being right.

If you can find a pairing, a fixture, or an existing outcome that generates your labels for free, use it. Hand-labelling is the fallback, not the default.

The runners bundle every question into a single systemOne call. This is the intended shape, and it is measured: four questions cost 6% more input tokens than one, because the state dominates and ships once, and bundling perturbs a shared question's answers by under 2%. So ask the extra questions. wording.mjs is the template — four cells, one request, all seeing identical state.

Four claims in FINDINGS.md were retracted, and every one came from comparing against a convenient arm rather than a correct one. If you want to know whether the primitive matters, hold the wording fixed, or run the full factorial as wording.mjs does. A quantity measured on one arm's plumbing is a fact about that plumbing until a second arm confirms it.

The probes are deliberately small and duplicative. To ask a new question:

1. Copy the closest runner — run-arms.mjs for a binary judgment, score-probe.mjs for an ordered one, wording.mjs for a factorial.
2. Replace the question constants at the top and the stateFor() shape. Keep the question text as a module-level constant; a question edited in place mid-run makes the results uninterpretable.
3. Write one JSONL row per item to .sandboxes/results-<name>-run<N>.jsonl, recording verdict, confidence, raw probabilities, ms and usage. Record everything; decide what matters later.
4. Write a matching *-report.mjs that reads those rows and prints the judgment.
5. Add both to package.json scripts.

What the existing reports look for, and what a new one should:

1. Separation — mean confidence when right against when wrong. Flat means confidence-gating is dead and you are back to a bare classifier.
2. Difficulty — mean confidence on easy against hard items, on an axis the model was never shown.
3. Churn — run the same items 3-5 times. Where verdicts flip, and whether the flips concentrate in low-confidence items. This is the strongest test available, because it doesn't depend on your labels at all.

For an ordered judgment add adjacent-level AUC. A Score will merge two levels it cannot resolve and report mean scores that look perfectly monotone while doing it — there is no error and no warning. Confidence is what flags it.

At the prices in FINDINGS.md, a 173-item corpus costs roughly $0.01 per Jev run and ~$0.40 per Haiku-API run; the claude -p arm is ~8x the API arm and exists only to show what CLI harness overhead costs. Multiply by repeats. PROBE_DIFF_CHARS is the lever if a corpus is too expensive — it truncates the diff, which is nearly all of the token count.

The Jev price constant is hardcoded in probes/report.mjs (JEV_PRICE) and was quoted, not published. Re-check it before quoting any cost figure.

Findings format is claim → evidence → confidence in the claim. Name the probe that backs a claim; where nothing does, say so.