Verifier-Backed SWE Tasks from Licensed Codebases

01 · One task

One codebase task, end to end.

One record from the public sample, in the fields a task needs: what to fix, on which revision, against which approved intent, checked by which command. The graph the rest of this page describes is where tasks like this one are cut from.

DEFECT-260727-WWWY

episode ep-DEFECT-260727-WWWY · task_family bug_fix · reward toolchain

Task: Fix Set() so that a beyond-length array index appends instead of destroying the array. Reproducer from the record: Set({"a":[1,2,3]}, 99, "a", "[9]") returned {"a":[99]} before the fix and {"a":[1,2,3,99]} after.
Subject: github.com/buger/jsonparser @ 6454f95de679fdd0c623a6183f7cff82da56b442 (MIT). This is the packaged, post-fix tree. The affected revision is historical and public, outside the zip; findings/INDEX.md says how to reach it.
Intent: SYS-REQ-110 (approved, reviewed_by: human:buger, 2026-07-26): when Set targets an array-index [N] with N >= len(array), the parser shall append at the end and return the mutated document, not overwrite existing elements or panic. Rests on STK-REQ-005. Formalized in FRETish.
Failure class: boundary · high silent data loss. Sibling hazards on the same requirement: nested_mutation (medium), element_type_partition (high).
Available context: The requirement neighbourhood (SYS-REQ-110, SYS-REQ-009, STK-REQ-005) · the implementation (parser.go:Set) · the historical failure (DEFECT-260727-WWWY, and KI-4 as the separate top-level partition) · the public tests named in the record.
Verifier: A deterministic go test pin. No LLM judge; the exit code is the verdict.
go test -count=1 -timeout 120s -run 'TestSetBeyondLengthScalarArrayPreservesElements_SYS110|TestMCDC_SYS_REQ_110' .
Run through re-run/verify.sh --pin DEFECT-260727-WWWY.
Success: The known witness passes (TestSetBeyondLengthScalarArrayPreservesElements_SYS110) · the requirement’s MC/DC obligation row passes · the sibling partition (KI-4, top-level arrays) carries its own explicit status.
Provenance: PROVENANCE.json: source licence (MIT), subject commit, producer (Proof / ReqProof), review policy, content-root SHA-256 and per-file SHA-256 for all 201 payload files.

Real record · Read off the public sample package: hierarchy/SYS-REQ-110, findings/problem-reports/DEFECT-260727-WWWY.yaml, serialization/findings.jsonl. Section 05 opens the requirement node itself.

02 · Why this exists

Training data that knows what the code was supposed to do.

Bug pairs teach broken → fixed. Models need why. A correctness label package is a versioned, re-checkable requirement graph over a codebase you already license: the intent hierarchy above the code, the graded ways each promise can fail, and the link from every claim to the code that implements it and the test that proves it. To a data team, that is training and evaluation data with the intent attached.

What you have today

Source trees, scraped or licensed
Bug tags and preference pairs
Unit tests, or an LLM judge as the reward
Intent latent in commits and tickets
Failure modes that surface after incidents
Hard to re-verify after delivery

Models see symptoms, not contracts.

What a correctness label package adds

A full requirement hierarchy above the code
A formal formula on machine-checkable claims, validated before any code is graded
Hazard analysis on each claim: worst case and severity
Machine-readable links: requirement ↔ code ↔ tests
Condition-level coverage, formal checks where mature
Findings with runnable reproducers, on the same graph
A person validates every finding; the reviewer is named in the file

The source of truth moves up to the requirement.

03 · Scope

What this is—and what it is not.

One sentence for the offer, then the four obvious wrong guesses.

What it is

A versioned requirement graph over one component of a codebase you license: hierarchy, formulas, hazards, code and test links, findings with reproducers, and a script that reruns the verdicts. It ships as files you keep. Read as data: requirement-grounded tasks, each with a deterministic verifier and the provenance to rerun it.

Bulk code

Not bulk unlabeled code. You already license trees. We do not compete with scrapers or corpus brokers.

Agent output

Not unverified agent output. Every shipped item passes a machine check where we claim one, and findings carry runnable reproducers.

Client work

Not another buyer’s engagement. Client and private audit work is never licensed as data. The eligible subjects are two: the trees you already license, and open-source corpora whose annotations we authored and can license cleanly. The jsonparser sample below is the second kind.

Our own history

Not Proof’s own source history. We enrich your licensed component. We do not resell our own repository as training data.

04 · The unit

The unit is a requirement with evidence.

A requirement node is the unit a task is cut from. Four levels: Stakeholder (STK-REQ), System (SYS-REQ), Software (SW-REQ), Interface (INT-REQ). Parent and child links are explicit, so the graph is queryable: pick any requirement and walk to its children, its code, its tests, its hazards and its evidence.

Stakeholder intent
Why the product exists: acceptance-level needs.
System requirement
What must always hold: product-level guarantees.
Component contracts
Software and interface levels, when the graph needs them.
Hazards
How this claim fails. Worst case and severity on every shipped requirement.
Code and test annotations
Implements / verifies links: machine-readable, not wiki prose.
Formula and evidence
Machine-checkable claims carry a FRETish formula, structured English descended from NASA’s FRET, compiled to temporal logic and checked for realizability, consistency and vacuity before code is graded. Then coverage, findings, and validation.

Two kinds of requirement, marked apart in the file. A machine-checkable requirement carries a FRETish formula and is graded by the checks attached to it. A prose-only requirement states intent that no formula expresses; it carries formalization_strategy: informal, an empty fretish field, and it is discharged through the machine-checkable requirements underneath it. In the jsonparser sample, all 116 system requirements are machine-checkable and all 7 stakeholder requirements are prose-only. Every package prints the split, so a buyer never has to assume that a requirement count is an executable-check count.

Fig. 01 · One node of the sample package, with its edges. Every line here is a field in a YAML file the package ships; the next section opens it.

The full model → How the evidence gets made →

05 · Fully wired

One node, fully wired.

One path through the graph: the data-loss class in jsonparser’s Set helper. Everything below is public, the package ships the same files, and the links open them on GitHub.

SYS-REQ-110system · approvedcomponent: parser

Set beyond array length shall append — never silently overwrite.

satisfies the mutation-helper intent under STK-REQ-005 · sibling of the general contract SYS-REQ-009 · the leaf partition for the data-loss class

Set() in parser.go carries the requirement IDs that govern it, and the tests name the requirement they verify. Three graded hazard obligations hang off this claim:

boundary · high
Silent overwrite. An index beyond the array’s length destroyed elements the caller never addressed and returned a mutated document with no error. Discharged by MC/DC plus negative boundary tests.
nested · medium
Wrong offset. A beyond-length index inside a nested container scaffolds at the wrong offset: a sibling overwrite, or malformed JSON out.
partition · high
Whole-array loss. Scalar-first arrays took a replace-container branch and the entire array was destroyed. ✓ fixed — the regression is pinned under the same claim as DEFECT-260727-WWWY.

The affected code carried 100% MC/DC at the time: every condition in the checked decision logic was exercised. Exercised is not correct. This class still produced two published misses. The postmortem accounting is on public proof →

specs/system/requirements/SYS-REQ-009.req.yaml

id: SYS-REQ-009
status: approved
priority: shall
component: parser
fretish: the parser shall always satisfy !set_path_is_provided | set_target_exists | set_creates_missing_path | set_returns_updated_document | set_returns_not_found_error
formalization_strategy: fretish
traces:
  satisfies:
    - STK-REQ-005
  verified_by_extra:
    - mcdc_supplement_test.go
    - set_spec_test.go
  reviewed_by: human:buger
obligation_hazards:
  - class: boundary
    worst_case: 'Set on an array-index path component [N] where N >= len(array) silently overwrites element 0 or another existing element the caller did not address, destroying data and returning a mutated document with no error (PR #286 regression class).'
    severity: high

Verbatim fields from the public requirement node. reviewed_by: names the person who validated it: checkable data in the file, not marketing. The full node on GitHub ↗

The root-cause write-up ↗ The full specification tree ↗

06 · Hazards

Hazards make failures graded, not binary.

Most corpora show failure only after an incident. Here every shipped requirement is walked against a failure-mode catalog before anything breaks, and the result is recorded on the requirement itself, in three fields.

Class

A reusable failure type: panic-free input, path traversal, fail-open control, boundary, partition… The same catalog recurs across requirements, so the negatives are comparable across the corpus.

Worst case

The concrete consequence on this requirement. “Silently overwrites an element the caller did not address, destroying data with no error” is a training signal; “data loss” is a slogan.

Severity

Low / medium / high / critical, as authored triage. It is a judgment we wrote down, not a CVSS score, and the file says so.

For a model, this is the missing layer between “code” and “incident”: graded, per-requirement, machine-readable. In an evaluation set, it is the failure category on the task. The data-loss record in section 05 is one entry of this kind.

07 · The samples

Download the samples. Rerun the verdicts.

The public label-and-evidence sample: one unit of the product, free to evaluate, built from the public audit of jsonparser, a mature zero-allocation Go JSON library long in production. Everything in the zip comes off the fixed master tree, library sources and reproducer tests included. It ships no seeded demo material; its 8 demo episodes are cut from real, fixed defects. Below it, a second download: the environment package, eight bug-fix tasks with their base revisions, a reset, an isolated toolchain and a hidden grader.

The label-and-evidence sample

serialization/requirements.jsonl (123 lines) · findings.jsonl (12) · episodes.jsonl (8 demo episodes, one per defect record) · traces_and_hazards.jsonl (670) · SCHEMA.md
PROVENANCE.jsonsubject commit, licence, producer, content root + per-file SHA-256
re-run/verify.shdeterministic go test pins; the command is below
corpus/123 requirement files · 8 defect records · 4 known-issue records · MIT library sources, reproducer tests, the root-cause write-up
hierarchy/the SYS-REQ-110 spotlight from sections 01 and 05 · README.md is the entry point

proof-jsonparser-sample-v1
version 1.0.1 · 442 KB zip

Download the public sample

sha256 04b646a3ea14fb3938e14db8153d850e2acdaed7ecc309415b7bb3a30ca96729
checksum sidecar · Go 1.21+ is all verify.sh needs

what “rerun the verdicts” means here

$ unzip proof-jsonparser-sample-v1.zip && cd proof-jsonparser-sample-v1
$ ./re-run/verify.sh                             # all eight pins
$ ./re-run/verify.sh --pin DEFECT-260727-WWWY    # one finding

ok    github.com/buger/jsonparser  0.33s
==> OK — pin(s) green on packaged (post-fix) sources

Go 1.21+ and nothing else. To a data team, verify.sh is the grader. The script stages the packaged sources and tests into a temp module and runs go test: no network, no LLM judge, exit code is the verdict. Green means the declared package checks pass on the packaged revision. It is not a statement that the whole library is correct: behavior nobody wrote down stays outside every check in the package. The zip carries the post-fix tree, so green here means the fixed tree still holds the pinned claim; a red-before run wants the pre-fix subject commit, public and outside the zip.

The environment package

tasks/<task_id>/8 bug-fix tasks in 3 fix families, Go only · TASK.md, the instruction · metadata.json · public_checks.json · hidden_checks/, the go test pins · reference/fix.patch, one per family · difficulty null, no authored label
repo/families.json · one base tree per family, without .git, without the Proof defect records, without the base tests that asserted the pre-fix behaviour: 13 removed in f0958b5, 3 in 3005d5b, 0 in 753cda0, declared per family and per task
environment/setup.sh · reset.sh · policy.json (network none; go build, go test, go vet, gofmt) · Dockerfile (golang:1.22)
verifier/grade.sh · scoring.json (reward 1.0 or 0.0, no LLM judge) · result.schema.json
quality-report.jsonmeasured at build: 8 candidates, 8 accepted, 0 rejected · hidden checks fail on base and pass after the fix, 8 of 8 · three repeated runs agree · reset byte-identical in 16 of 16 trials · Docker (network none) matches the host
intent/ and PROVENANCE.jsonrequirements.json (123) · dependencies.json · hazards.json · history.json · the contamination declaration · content root + per-file SHA-256

proof-jsonparser-env-v2
version 2.0.0 · 676 KB zip · 60 files

Download the environment package

sha256 15fd42b87945f8d3cf28c3c015439a2d805c50165c3c3d6c0b341e87c5d2403d
checksum sidecar · bash, Go, python3, tar and zstd on the host, or the grader image from its Dockerfile

what “run one task” means here

$ unzip proof-jsonparser-env-v2.zip && cd proof-jsonparser-env-v2
$ WORK=$(mktemp -d)
$ environment/setup.sh task-260727-wwwy "$WORK"   # the base tree, extracted to $WORK/repo
$ verifier/grade.sh task-260727-wwwy "$WORK"      # reward 0.0 on the base tree: the hidden check fails
# ... the agent edits files under $WORK/repo ...
$ verifier/grade.sh task-260727-wwwy "$WORK"      # reward 1.0 when the hidden check and the public suite pass
$ environment/reset.sh task-260727-wwwy "$WORK"   # back to the base tree, byte-identical

The task is the Set() defect from sections 01 and 05, now with its base revision. The grader never modifies $WORK/repo: it copies the tree to scratch, restores every base *_test.go, copies the hidden checks in fresh, runs the hidden filter, then the public suite without the hidden checks. Reward is 1.0 only when both pass. No network. No LLM judge. Red before and green after are measured for all 8 tasks in quality-report.json; the reference patch that turns one into the other is tasks/task-260727-wwwy/reference/fix.patch.

What the downloads are, and are not. The label-and-evidence sample demonstrates the schema, provenance, evidence graph, and deterministic post-fix checks. The environment package packages the four things a production RL environment adds: the affected base revision, a reset mechanism, an isolated toolchain, and a hidden grader. What neither has is a private subject or a held-out split: jsonparser and its fixes are public, so both demonstrate the shape and the measurement. A clean evaluation set needs a private subject.

123 requirements · 7 stakeholder + 116 system 116 of 123 carry a FRETish formula · count them in specs/ ↗ 8 defect records on master · 2 credited to outside reporters 2 misses published

The same corpus, elsewhere: the specs on GitHub ↗ · the register in the portal ↗ seeded product demo; read the labels first · the full accounting on public proof

08 · Deliverables

One graph. Three products.

The graph is the task-generation substrate. Each product is one cut of it, and the three follow a data team’s own order: evaluate, curate, post-train.

01 · evaluation sets

Evaluation sets

Held-out, requirement-grounded software tasks with failure categories and deterministic graders.

02 · training data

Training data

Requirements, code links, failure modes, defect histories, and expert-reviewed outcomes for supervised or preference-data pipelines. The graph is the source for those pipelines. It ships as a correctness label package: versioned files you keep, re-checkable after delivery. No SFT or preference format ships in the public sample.

03 · RL environments

RL environments

Resettable codebase tasks where an agent can inspect the repository, modify it, run tools, and receive a versioned reward from controlled evidence. The public environment package ships the base revision, a reset, an isolated toolchain and a hidden grader for 8 tasks on a public subject. Customer environments are scoped after a label pilot.

Why the reward signal is different. Agents game graders: they read future commits, special-case test inputs, and hunt a judge’s blind spots. A deterministic, versioned verdict removes model-judge subjectivity and makes common grader exploits inspectable. It does not eliminate reward hacking. A high reward means the declared checks held in the stated environment—not that the program is globally correct. You can re-run the check any day after delivery.

Formal checks run through Kind2 / Z3 where that path is live; condition-level coverage and reproducers carry the rest. What “verified” means here →

FRETish · Kind2 · Z3 · hazard analysis · MC/DC

09 · Start

Start with one component.

The custody answer first, then the box, then the email.

Your code

You supply a codebase you already licensed. Input provenance stays on your side. We do not resell your source, and no other buyer’s engagement enters your package.

Our labels

We license the derived layer only: annotations, hazards, evidence.

Deterministic verdicts

Deterministic tooling decides: toolchain exit codes, coverage, formal checks, no model dependency. If your policy restricts which models may touch the drafting pipeline, drafting runs on the stack you allow.

The engineering mirror of this custody story →

The first engagement, boxed.

One component of a buyer-licensed codebase, or a clearly licensable open-source slice. History dig → structured requirements → code and test links → coverage and hazard analysis → findings with reproducers → a machine check on every shipped item, a person’s validation on every finding.

Duration: 2–3 weeks of calendar time. A tight pilot box, not a monorepo firehose.
Commercial: Fixed fee, quoted in the first reply. Complexity or exclusivity changes the quote before work starts.
Success: Measured on the pilot’s own counts:

Accepted tasks per component

Verifier determinism

Environment reset reliability

Task rejection rate

Expert-review minutes per accepted task

Exploit / reward-hacking findings

Cost per accepted task or environment

In production, cost per accepted requirement node against market rates is the secondary metric. A node is accepted when it carries its hazards, its code and test links, and the evidence its kind requires.
Out of scope: Entire monorepo dumps; third-party finding dumps as “samples”; client or private audit work; the sale of Proof’s own product history. Environments are the third product, scoped after a label pilot.

The same engine installs Proof for product teams: roughly four weeks, on the audit page. The pilot above is the data-team shape, shorter and judged on label quality.

Email is enough to start. hello@reqproof.com is the whole ask.

Proof was founded by Leonid Bugaev and operates both the Proof platform and its named assurance practice. Twenty-plus years in engineering; Head of Engineering at Tyk API Management. Judge the work by the public jsonparser corpus above.

Turn licensed codebases into verifier-backed SWE tasks.