referee-lab

Public referees for robot-training video. Five evaluation environments on the Prime Intellect Environments Hub that score what everyone claims and nobody measures: whether captured video is actually usable for training robots — hands in frame, honest frame rates, usable footage fractions, duplicates in disguise.

Anyone can run any model against them, today, in one command. Three of the five are currently unbeaten by a 27B model.

hands-visible — is a hand or gripper visible? (do-nothing floor: 68.9%)

Status: provisional in-domain development sheet. The public 45-clip sheet was repeatedly consulted during development and has not yet passed the declared independent two-grader gate. Treat these rows as reproducible development evidence, not a frozen external benchmark.

A 1.5M-parameter student matches its 27B teacher on this provisional sheet, holds at 100% on real footage when no training frame comes within 60 seconds of a test clip, and outscores a 72B general VLM. Scale alone does not buy this task: the qwen2.5vl family climbs 33% → 46% → 74% → 83% from 3B to 72B and still lands under both the specialist teacher and its distilled student. Two purpose-built hand systems score below the do-nothing floor on robot footage — external metrics are re-measured here, never imported.

effective-fps — real frame rate, or upsampled? (floor: 50%) gemma3:27b 55% · qwen-7B 50% · EgoSieve-S duplicate-frames head 50%. Frame-duplication fraud is effectively unsolved — every tested detector is at or near chance.

billable-fraction — % of frames that are usable footage (floor: 43%) gemma3:27b 40% · qwen-7B 27%. Both below the floor. Estimating capture yield defeats current VLMs.

escalation-router — will a 7B get this clip right, or must it escalate? (floor: 67%) gemma3:27b 40% · qwen-7B 33%. Both far below the floor — models cannot predict a smaller model's failures. The cascade defer decision is an open capability gap.

clip-twin — same footage resold in disguise? (floor: 50%) v0.2 ships a frontier-mined corpus (subtle transforms + same-episode hard negatives). The two mining-oracle models score at or below chance (gemma3:27b 12.5%, qwen2.5vl:7b 0%); qwen2.5vl:32b, held out of mining, scores 55% — barely above the floor. All three scored 100% on v0.1: the difficulty transfers to models that never shaped the corpus.

• Validation gates before Hub publication: garbage submissions score ~0 by construction, the answer sheet scores exactly 100%, class prefixes are mixed, every frame decodes. (tools/validate_family.py)
• The floor is printed. Every README states the best do-nothing score; a model demonstrates skill only above that line.
• Gold that needs no trust: four of five corpora use gold by construction (injected defects, constructed duplicates) or measured model behavior — zero human-label opinions to argue with. The fifth (hands-visible) discloses its provisional grading and per-clip stability/difficulty columns.
• Licensed to redistribute: every frame is CC BY / CC0 / MIT with machine-readable per-clip provenance and shipped ATTRIBUTION files.
• All baselines ran on owned hardware. No API sponsors, no cherry-picks; errored calls are retried, then excluded — never scored.

# hosted, zero setup
uv tool install prime && prime login
prime eval run referee-lab/effective-fps -m <model>

# or local, against any OpenAI-compatible endpoint
uv pip install verifiers effective_fps \
  --extra-index-url https://hub.primeintellect.ai/referee-lab/simple/
vf-eval effective_fps -m <model> --api-base-url <url> --api-key-var <VAR>

Full guide: docs/how-to-run.md — including scoring your own footage with hands-visible.

The same discipline at dataset scale — we audited the two flagship manipulation datasets from their own logs: 21.0% of all 136,444 DROID-port episodes (full corpus, 740 robot-hours) and 27.3% of all 53,192 BridgeData V2 episodes ship without a language instruction. Gripper never actuates in 9.4% / 13.8%. One command each to reproduce. Also: we audited 166.7 robot-hours of DROID — ~40% of the dataset — from its own telemetry: 20.6% of episodes ship without a language instruction in any field (stable across disjoint slices); the gripper never actuates in 9.4%; and (a finding against our own hypothesis, published anyway) commanded dead time is a remarkably low 0.7%. Reproduce it in one command: python tools/droid_audit.py.

Robot-training data is bought by the hour and rejected by the batch, and until now there was no public instrument that measured either side. referee-lab builds the instrument: graded answer sheets, published grading records, and floors that keep everyone honest — the same sheet grading a 27B VLM, a 22M specialist, and a 1.5M microcontroller-bound student. Models are rows; the answer sheet is the asset. More evals are queued (108-idea catalog in docs/egocentric-eval-catalog.md); the tooling that built these five (tools/) stamps out the next ones.