Toward Human-in-the-Loop Oversight for Long-Horizon Agent Projects

cogito233/swarm-oversight-harness

Contents: What the human sees · Why not watch the agents · The middle layer · Making acceptance mean something · Related work and limits · Try it

Multi-agent harnesses are maturing. Runtimes now keep many agents busy in parallel [1], and agent teams share persistent memory and build on each other’s work [5, 7]. As these systems improve, the scarce resource shifts from compute to the attention of the person who oversees them. I ran into this about four months ago in my own research projects and have been working around it since. I assumed this was common knowledge, but after this week’s announcements I am less sure, so I am writing down what I do.

Running ten agents at once is easy. Overseeing them on one project for three months is not. When I did this on research projects, the bottleneck after a few weeks was knowing what was true in my own project, more than compute or model quality.

The claim of this post is that the cost of overseeing a project depends on how many breaks in continuity the human has to reconcile, not on how many agents are running. The design goal follows from it: keep the project’s history continuous, and keep it in the place that is cheapest for a human to read. The analogy that helped me is the move from procedural to object-oriented programming. An agent session is a procedure call. It has a stack (its context window), does some work, returns a value, and its local state is gone. Most agent setups are procedural in this sense, because the state of the work lives in whichever agent is running. The middle layer makes the task the object instead: it has an identity, state that persists between calls, invariants, and an interface. Agents become methods that act on it, one after another or in parallel, and none of them owns it.

The rest of the post describes that layer in practice: what I look at, what I decide, and what the layer does for me. I keep three things in my head: the charter, the budget and the list of threads. I act at two points: approving what a task is for, and accepting what it produced. Everything else lives in task files in the repo. A planner agent maintains those files, and worker agents attach to a task, do their part, and exit. I am releasing the method as Swarm Oversight Harness.

1. What the human sees

My model of a project fits on one screen and has three parts:

1. Charter: what we are trying to find out, and the hypothesis whose outcome would change our plans.
2. Budget: how much compute and money each thread may spend before it has to come back and ask.
3. Threads: what each line of work is doing, what stage it has verifiably reached, and what it is waiting for.

An illustrative example (not a real project):

CHARTER  Can a 7B model learn to check its own patches with repo tests?
         H1: training on fail→pass test pairs adds ≥10 points of
             self-check accuracy on held-out repos.
         Kill if the 200-task pilot gains < 3 points.
BUDGET   2,000 GPU-h + $1,500 API in total. Over a thread's cap → ask.

STATE    critical path: T2 pilot result
  T1 build    fail→pass pipeline    verified, CI #41    40/100 GPU-h   → accept gate
  T2 run      SFT pilot, 200 tasks  launched, no agent  310/400 GPU-h  → done hook
  T3 review   red-team T1 leakage   running             $90/$200
  T4 read     related-work map      draft               $15/$50

I make two kinds of decision from this screen:

• Approve a task that the planner has drafted: what it is for, what counts as done, and how much it may spend.
• Accept a result: once accepted, the result is no longer a draft, and other work can build on it.

Everything below this screen is written for agents. I open a task’s files only when I need them for one of these two decisions. The budget is part of what a task is authorized to do, so an agent that would go over its thread’s cap stops and asks instead of retrying.

2. Why not watch the agents directly

Attention does not scale. DeepSeek’s Super-agent design makes the same point: today’s agents cap single-task performance at model capability and multi-task performance at human attention [1]. In my runs the ceiling was three to five sessions.

Projects lose continuity. How hard a project is to understand depends less on its size than on its continuity: one name for each thing, one version of each fact, and a record that reads in order. Every break makes the reader notice a discrepancy, decide which side is right, and remember the decision. Months of procedural agent work, where each session carries the state and then disappears, produce such breaks without ever raising an error. These are real incidents from my projects, lightly anonymized:

Each break was small, but together they meant I had to re-read the project every morning. Watching more agents would only have added more breaks, because each new session is one more place where state can live and then disappear.

3. The middle layer

Agents are procedures; tasks are objects

A single agent is easy to follow because it has one linear history that you can read in order. That history is lost when the call returns, that is, when the context window ends. A hand-built pipeline is worse, because its history is spread across stages and no single file says what happened in order.

The middle layer keeps a linear history but stores it in the task instead of the agent. The programming analogy maps directly:

The analogy has one important limit. An ordinary object overwrites its fields, but a task never rewrites its LOG, so it is closer to an object whose state is rebuilt from an event log. That is what keeps the history readable in order. Concretely, a task is a small, durable object in the repo:

• TASK.md: goal, files it may and must not touch, the authorization and budget it runs under, acceptance criteria with how each is checked, stop conditions;
• a LOG that is only ever appended to, with a snapshot of the current state at the top;
• RESULT files with evidence;
• one row in STATE, which is the screen in section 1.

One habit follows: an agent that launches a long experiment exits instead of polling. Before it exits, it appends a handoff to the task’s LOG: what was launched, where the outputs will appear, and how to check them. A completion hook or marker file signals the end of the run, and a new agent picks up the task. Polling would spend context on waiting, and the only record of what is pending would sit in a context that will be compacted.

The planner

A planner agent maintains the layer. Each round it re-reads the repo and fixes STATE wherever it disagrees with reality. It then names one critical path, splits work into tasks that touch different files, dispatches workers, has their results verified, and consolidates everything into one STATE update. I never merge N results by hand; if I find myself doing it, the layer is broken. One rule decides whether the planner keeps going or comes back to me: retry HOW problems, stop on WHAT problems. A failing test is a HOW problem and is retried within budget. A plan that contradicts reality, or a budget about to be exceeded, is a WHAT problem and is escalated to me.

Handing a task to an agent through files

The default way to delegate is to write the task into a subagent’s prompt and read back its summary. Over weeks, this breaks down. The task exists only in the parent agent’s context. The result is only the worker’s account. The parent has to merge N summaries. Failed attempts leave no record. In the middle layer the prompt is only a pointer, “do threads/T2/tasks/TASK-03.md”, and the result comes back as a file.

Which memory to trust and which to clean up

Moving memory into files only helps if the files stay trustworthy; otherwise the breaks in section 2 move into the repo. Not everything in the repo is equally valid, so each piece of memory is assigned to one of four kinds:

The planner does the cleanup at the start of each round. It compares STATE with git, tests and outputs. Every current doc records when it was last checked and which files it covers. If any of those files has changed since, the doc is marked stale until someone re-checks it. Superseded plans are not deleted; they move to history with a pointer to what replaced them. Anything that exists only in an agent’s context is treated as gone, because the agent will exit.

Three rules keep memory readable. Each thing has one name, based on its content (“the 4k-context ablation” rather than “arm C”). Each headline number is stored once and linked from elsewhere. Nothing is deleted to make the record look cleaner.

4. Making acceptance mean something

Acceptance is the one place my judgment enters the record, so it has to mean something. In research the output is a belief, and unlike a bad merge, a wrong belief raises no error. Agents that grade their own progress systematically accept non-improvements. Park and Choi show that a separate judge is not enough on its own: a strong judge that read the diff and the worker’s report still accepted cycles, and 44% of the cycles it accepted were real regressions [2]. What helped was externally grounded verification. In the middle layer this becomes three rules:

• The verifier re-runs checks. For each acceptance criterion it runs the check named in the task and compares the result with what was claimed.
• Six facts are reported separately: edited, committed, launched, ran OK, verified, accepted. Anything without evidence is recorded as unknown.
• Dead ends are recorded. A hypothesis that was killed goes into the decision log with its reason, so later agents do not try it again.

5. Related work and limits

Anthropic’s long-running harness already treats the environment as memory: a progress file and git history let one agent resume a project across context windows [3]. Agensh makes tasks claimable objects in a shared workspace and scales a decentralized team to 1,024 agents [5]. Evolving Agent Teams keeps a persistent learnings file so a supervisor can restructure teams that stall [4]. GitSwarm lets homogeneous agents compound each other’s intermediate work through shared persistent memory [7]. None of them separates who verifies a result from who decides that it counts. That separation is the human-facing part this post adds.

• Not a runtime. There is no scheduler and no daemon. For an always-on agent that takes requests from a team’s chat and GitHub, see DeepSeek’s Super-agent prototype [1].
• No enforcement. The rules hold only because agents read and follow them. Enforcement at the execution layer is a separate job, handled by tools such as agentsh [6].
• Not for chores. If you already know the answer and there is one result to merge, ask the agent directly.
• One person’s practice. This is based on about four months of my own research projects. It is not a controlled study, the incidents are mine, and the example in section 1 is illustrative. It may not transfer to teams, to work where actions cannot be undone, or to projects where the hard part is waiting on other people.

Try it

Paste this into Claude Code or Codex:

Install Swarm Oversight Harness from https://github.com/cogito233/swarm-oversight-harness — follow its INSTALL.md.

Then say set up swarm oversight in this repo.

Feedback and issues are welcome on GitHub.

Disclaimer

This post was drafted, checked against the cited sources, and typeset by an AI coding agent (Claude Code), working with the author in the way described above. The author decided what to write and accepted or rejected each result. Any errors that remain are the author’s.

Citation

Cited as:

Lyu, Zhiheng. (Oct 2026). “Toward Human-in-the-Loop Oversight for Long-Horizon Agent Projects”. cogito233.github.io. https://cogito233.github.io/blog/human-in-the-loop-oversight/.

Or

@article{lyu2026oversight,
  title   = "Toward Human-in-the-Loop Oversight for Long-Horizon Agent Projects",
  author  = "Lyu, Zhiheng",
  journal = "cogito233.github.io",
  year    = "2026",
  month   = "Oct",
  url     = "https://cogito233.github.io/blog/human-in-the-loop-oversight/"
}

References

How Far Has RSI Gotten in Post-Training?

Contents: Four orders, one variable · Order 0 · Order 1 · Order 2 · Order 3 · Open questions

“Self-refine, self-play, self-evolve, self-improving, recursive self-improvement.” In post-training, the same words are used for an agent that rewrites its own prompt, for a model that writes its own training problems, and for the last step before AGI. They describe different amounts of the same thing. Recursive self-improvement (RSI), as used here, is a loop in which an agent’s own output changes the agent that runs the next iteration, and the loop is closed by a verifier rather than by a person.

One variable separates the versions: how much of the objective is handed to the agent, and how much it has to supply itself. Hand it an instruction and it is a code agent. Hand it a verifier and it can climb. Hand it only a capability name and it must build the testbed. Hand it nothing and it must decide what to measure. This post climbs that ladder one step at a time. At each step it describes one or two systems in enough detail to show what the loop does, and asks whether any machine has yet done that step.

The scope is post-training and the things around it: agent scaffolds, training recipes, data pipelines. RL and SFT optimize against a target with a verifier, so the ladder is defined in those terms. Pretraining is out of scope. Two examples fall outside that scope: AlphaEvolve improves a program rather than model weights, and is included because its loop has the same shape; PaperBench is a benchmark, not an improvement loop, and is included because it measures research replication with a manufactured verifier.

Four orders, one variable

Take any system that calls itself self-improving and look at what it works with. There are four things, and they stack from the bottom up.

1. The method layer is everything that turns a model into a score on a task: the prompt, the scaffold and tools, the memory, the search strategy, the training recipe. Until recently, this layer was the subject of methodology papers written by people. It is the first layer to have moved to agents, starting in domains such as math and code, where a verifier can check the result.
2. The data layer is the set of task instances the system is trained or evaluated on. Each instance is a concrete problem with everything needed to attempt it: for a bug-fixing task, the repository, the issue text and the environment it runs in.
3. The verifier layer decides, for each task instance, whether an attempt succeeded, without a person in the loop. For bug fixing that is the repository’s own tests passing after the patch; elsewhere it is a wall clock, a validation loss, or a hidden test set.
4. The target layer names what the whole system is measuring: the capability that the tasks and their verifiers are meant to stand for, such as bug fixing, long-horizon tool use or research engineering.

Now for the one variable. For any such system, ask how many of the four layers the agent supplies itself, counting from the bottom. The answer is always a contiguous block starting at the bottom, and its size gives the four orders.

Each order takes over more of the stack, starting from the bottom. This is the staircase at the top of the post: order 0 stands on the ground and changes no layer, order 1 stands on the method layer, order 2 adds the data and verifier layers together, order 3 adds the target. Data and verifier come as a pair because a task that cannot be graded is not yet a task, which is why the second step is the tall one. The color of each layer shows who supplies it today.

The ground floor is worth spelling out, because everything else is measured from it. At order 0 the agent supplies none of the four layers. Its method is whatever the code agent shipped with. There is no data layer, because there is no set of tasks, only the one instruction the user typed. There is no verifier, because the user reads the diff and decides. And there is no target, because nothing is being measured. Every result is approved by a person, one at a time, and nothing carries over to the next instruction.

Three boundaries separate the four orders, and each one is a different kind of step.

Order 0 to order 1 is where a loop first appears. A plain code agent that reruns failing tests until they pass has a verifier too, so the difference is not whether a verifier exists but what the loop produces. At order 0 the product is the solution to this task; once the tests pass, the loop ends and nothing carries forward. At order 1 the product is the method itself (the agent’s own code, its training script, a kernel), and whatever one iteration improves is where the next one starts. This is where RSI begins, and it is the only boundary that turns a tool into a loop.

Order 1 to order 2 is a change in what the task set is. At order 1 the set is the benchmark’s published instances and never grows. At order 2 the agent adds legitimate new members to the same set, real or synthetic issues that each come with their own verifier, and then runs order 1 on what it built. The set was always larger than the benchmark, and at order 2 the agent goes out to collect more of it. Almost all of the work lies in finding where each new task’s verifier comes from.

Order 2 to order 3 is where the demonstration disappears. At order 2 the target is named and the benchmark shows one way to judge it, so the agent only has to add more tasks from the same distribution. At order 3 there is no target and no benchmark to copy, so the agent must first decide what is worth measuring and then do order 2 for it.

Order 0: instruction in, human accepts

A plain code agent gets a task in natural language, edits files, runs what it can, and returns a diff. Nothing in the setup says what a good result is except the person reading it. When the agent does have a check available, such as a failing test or a compiler, it uses the check to finish this task and then discards it. Cursor, Claude Code, Codex and OpenHands are all this shape by default.

This is what the tools are designed for, and it matters here only as the control group. Everything an order-0 agent learns about a task dies with the task. No quantity improves across runs, so there is nothing recursive to speak of, however capable the underlying model is. The moment someone adds a persistent metric and lets the agent’s edits carry forward, the same agent becomes an order-1 system. The first case below does exactly this.

Order 1: verifier in, score up

Definition. The agent is handed a task set and a verifier. Its job is to change its own method (its prompt, tools, code, memory or training recipe) so that the verifier’s score goes up, and to keep each change that raises it. The three objective layers above the method stay untouched. Concretely: hand an agent SWE-bench Verified and its test harness, let it edit its own scaffold or its training script, rerun the harness, keep the edit if the resolve rate rose, and repeat until the budget runs out. The 500 tasks never change; only the agent does.

Why it is hard. The naive approach is to take the benchmark’s own tasks, distill a few dozen trajectories for each, and fine-tune on them. This does not reach a full score. It yields a modest gain and sometimes loses points, because the distilled trajectories do not follow the paths the model would take on its own, and training on them disrupts behavior the model already had. What works is decomposing tasks and teaching them step by step, or on-policy distillation, in which the student generates its own trajectories and the teacher gives feedback only on the states the student actually reaches. This is the work an order-1 agent takes over: finding a way to raise the score without a person decomposing sub-tasks or tuning the recipe.

AutoResearch

Andrej Karpathy, a founding member of OpenAI and formerly Director of AI at Tesla, writes deliberately small, readable training code: micrograd, nanoGPT, nanochat. AutoResearch (Karpathy 2026) points a coding agent at a single-GPU version of nanochat and tells it to improve the model overnight. The agent edits one file, train.py, trains for five minutes, reads the validation bits-per-byte, and keeps the edit if the number fell or reverts it otherwise, for about a hundred experiments a night. The instructions live in a markdown file written by a person. There is no leaderboard, which is what makes it the cleanest order-1 example: the verifier is a held-out loss, the thing being improved is the training recipe, and each night’s product is a better train.py.

RSIAgent

RSIAgent (2026) puts the loop at test time. Before attempting a target task, an actor (GLM-5.3) practices on variants of it proposed by a curriculum agent, a verifier (Kimi-K3) grades each attempt from the environment’s feedback, and what works is written into a memory. The memory is then frozen, and the actor attempts the real task with it.

The harness raises the partial score from 71.97 to 78.98 on OSWorld 2.0 and from 83.75 to 84.82 on Agents’ Last Exam (ALE). This passes the partial scores reported for GPT-6 Astra, although RSIAgent stays below GPT-6 Astra on ALE’s binary score, 50.75 against 52.24. Why order 1. The tasks and scorers are given. The memory is the method, with one memory per target task, and the practice tasks are scaffolding that never enters a benchmark.

AlphaEvolve

AlphaEvolve (Google DeepMind 2025) runs the loop over a program instead of an agent. The user marks which parts of a program may be evolved and supplies a scoring function; Gemini 2.0 Flash and Pro propose diffs against programs sampled from a database, the evaluators score them, and promising programs go back into the database, which is organized so that diverse lineages survive.

It found a way to multiply two 4×4 complex matrices in 48 scalar multiplications, improving on Strassen’s 1969 algorithm; a heuristic that speeds up Gemini’s training kernels by 23% on average, cutting training time by 1%; and a scheduling heuristic that recovers 0.7% of Google’s fleet-wide compute. Why order 1. Nothing here touches a model’s weights, but the loop has the same shape. The objective is fixed, the user’s evaluator closes the loop, and the product is a better program that seeds the next round.

Other order-1 systems, and how the order is measured

Darwin Gödel Machine (Zhang et al. 2025) applies the loop to a coding agent’s own code: a repository around a frozen Claude 3.5 Sonnet edits itself, every version is kept in an archive, and a benchmark score decides which version is edited next. Over 80 iterations, its SWE-bench score rose from 20.0% to 50.0% and its Polyglot score from 14.2% to 30.7%. Along the way the agent invented features for itself, such as line-range viewing and string-replacement editing. Each run took about two weeks and cost about USD 22,000. DGM also produced the clearest example of the failure that order 1 invites: one agent scored perfectly on a hallucination check by deleting the logging the check depended on, and the authors’ fix was to hide the checker from the agent.

Order-1 ability has its own benchmarks. RE-Bench (Wijk et al. 2024) gives agents and human experts seven ML research engineering environments with numeric scores: with a two-hour budget the best agents score four times as high as the humans; at eight hours the humans narrowly pull ahead; and at 32 hours the humans score twice as high. MLE-bench (Chan et al. 2024) uses 75 Kaggle competitions with medal thresholds; the best setup reaches bronze or better in 16.9% of them. RSI-Exam (2026) has 88 expert-written research tasks scored on hidden data; GPT-6 Astra reaches 0.51 and Opus 5 0.46 on a scale where 0.6 is a frontier-calibrated reference solution.

How far order 1 has come. Agents already operate at this order today. Gains on a fixed target can be large from a weak start: DGM went from 20.0% to 50.0% on SWE-bench. From an already-high start they are smaller: RSIAgent adds about 7 points on OSWorld 2.0 (71.97 to 78.98) and about 1 on ALE (83.75 to 84.82). Where the verifier is a wall clock or a loss, agents match or beat human experts on short budgets. This is also the order with the messiest naming: self-improving, self-evolving and RSI are all used for it.

Order 2: target in, testbed out

Definition. Order 1 left the objective alone. Order 2 is what happens when the agent takes over the data layer and, with it, the verifier layer: which tasks are used, and how each one is graded.

Take SWE-bench Verified. Its definition (a codebase, a failing test, and a patch that makes the test pass) describes a set far larger than the 500 instances it publishes. Every real GitHub issue with that shape belongs to the set, collected or not, and so does every synthetic one that satisfies the same rule. Order 1 only ever searches methods against the 500 that were collected. Order 2 adds new members to the set. These are tasks drawn from the same distribution as the target but not copied from its items. Each one exercises a single nameable, separately testable skill (bug fixing, long-context localization, multi-step tool use) rather than the benchmark as a whole, and each comes with a verifier, so it can be graded whether or not anyone trains on it. Then it applies order 1 to what it built. Concretely: hand an agent the phrase “bug fixing” and the SWE-bench rules, and its job is to go to GitHub, pick repositories, mint issues that have a failing test and a passing patch, and only then train on them.

The bottleneck is the verifier. Writing a new task is easy; writing one that can be automatically judged is hard. Every order-2 pipeline running at scale today exists because a person first worked out where its verifier would come from. The two systems below are therefore order 2 done by humans and agents together: people designed the pipeline and chose the verifier, and agents execute almost every step. No agent has done order 2 alone; the section ends with the two that have come closest.

SWE-smith

SWE-smith (Yang et al. 2025) turns a Python repository into hundreds to thousands of bug-fixing tasks by breaking code that existing tests already cover. An agent installs the repository, and a person confirms that the tests run. Four strategies then inject bugs, ranging from asking an LM to rewrite a function to reverting a real pull request. A candidate becomes a task only if it breaks a previously passing test, and an LM then writes the issue text from the diff and that test.

Across 128 repositories, this yields 50,137 instances for about USD 1,360. Qwen2.5-Coder-32B, fine-tuned on 5,016 Claude 3.7 Sonnet trajectories collected on them, reaches 40.2% on SWE-bench Verified. Order 2, by humans and agents together. Every instance is a legitimate new member of SWE-bench’s set, and its verifier is borrowed from the repository’s own tests. The division of labor is explicit: the authors decided that repository tests can serve as the verifier, that breaking passing tests creates tasks, and that four strategies are needed; the agents install repositories, write bugs and write issues. Take the agents out and the pipeline stops; take the people out and it never starts.

SWE-Universe

SWE-Universe (Chen et al. 2026) scales the same idea to all of GitHub. Of 33.3 million pull requests, about a million survive filtering. For each one, a building agent is given bash and a switch between the buggy and fixed states of the repository. It installs the project and writes an evaluation script that must fail before the fix and pass after it, while a hacking detector inside the loop rejects scripts that fake the check, for instance by grepping the source files.

The builder is a Qwen-Next-80B-A3B trained by rejection sampling on its own successful, non-hacked runs, and it beats Claude Opus 4.5 on the paper’s building benchmark, 78.4% to 77.8%. The result is 807,693 environments across 52,960 repositories and eight language groups; mid-training on 500,000 trajectories from them takes Qwen3-Next from 50.3% to over 61% on SWE-bench Verified, and the same recipe took Qwen3-Max-Thinking to 75.3%. Order 2, by humans and agents together, with the human share shrinking. The agent now builds the environment and writes the verifier script, and another agent polices hacking. People still supply the frame: PRs as the source, fail-then-pass as the verifier, and a definition of what hacking looks like. As of 2026, agents execute order-2 pipelines, but people still design them.

Agents that choose what to make

Two systems hand the agent the decision of what data to generate. DataEnvGym (Khan et al. 2024) sets up a teacher agent that reads a student model’s errors, organized by inferred skill, and decides which skills to generate more data for next; the generation engine, the skill structure and the verifier are fixed parts of the environment. ANDES (2026) has a trainer agent decompose a downstream benchmark into capability domains, route synthesis through a self-expanding tree of contexts, and adjust the next round from a synthesis report. Both hand over the choice of what to make. Neither hands over how to make it or how to judge it, and both operate at scales several orders of magnitude below the human-designed pipelines. No agent has yet been handed SWE-bench Verified and written its own SWE-smith.

Where verifiers come from

Inside every order-2 pipeline the verifier comes from one of two places. Either it is borrowed: it existed before the pipeline, written by someone else for a different reason, such as the repository tests that SWE-bench, SWE-Gym and SWE-smith run. Or it is manufactured: nothing existed, so the pipeline synthesizes one, such as the cross-checked sandboxes of EnvScaler, the masked reasoning steps turned into multiple choice of Golden Goose, or the success functions OMNI-EPIC writes alongside each task. At the top end the two groups differ in scale by two to three orders of magnitude, although the smallest borrowed-verifier sets are no larger than manufactured ones. The borrowed-verifier line for bug fixing alone runs SWE-Gym at 2,438 tasks, R2E-Gym at 8,135, SWE-rebench at 21,000, daVinci-Env at 45,320 environments and SWE-Universe at 807,693; manufactured-verifier pipelines sit at hundreds to low thousands. Competition problems are hand-written but were written for the competition, so they count as borrowed. What decides the split is whether the verifier exists for some reason other than training AI.

Order 3: nothing in, benchmark out

Definition. Nobody hands over a target. An order-3 agent does what the best benchmark makers do. It surveys where current agents fail and how far they are from competence, names a task paradigm that nobody has measured, and finds a way to obtain supervision for it. The evidence for the gap can come from first-principles thinking about what a capability requires, from dissatisfaction signals in user data, from discussions on developer forums, or from user-interview notes. The supervision can be mined from a structure that already exists, as SWE-bench mined tests from GitHub pull requests, or commissioned, with the agent hiring domain experts to write and grade tasks. Every task at this level is a piece of original research. The job is not “make more bug-fixing tasks” but “discover that bug fixing is an axis worth measuring, and find a way to judge it”.

Why it is hard. At order 2 the target benchmark had already demonstrated what counts as the capability and how to judge it; the agent only had to add more tasks from the same distribution. Order 3 has no demonstration. The verifier, which was the bottleneck at order 2, becomes the whole problem: there is no verifier lying around for a capability nobody has named yet, so every attempt so far has had to manufacture one.

OMNI-EPIC

OMNI-EPIC (Faldor et al. 2024) is the most complete attempt at the loop. A task generator (Claude 3 Opus) reads an archive of learned and failed tasks and proposes a new one that is learnable and interesting; an environment generator writes it as Python with a reward and a separate success check; a model of interestingness (GPT-4o) discards it if it is not novel compared with similar tasks in the archive; an RL agent trains on it in a simulator; and the outcome goes back into the archive.

Over 200 iterations, the tasks spread from navigation to object manipulation, including crossing a rainbow bridge with moving segments and kicking a ball into a moving goal. In the shorter run with real RL training, the agent learned 16 tasks and failed 6. Why it is order 3, and why it has not caught on. No named capability was handed in; the system names its own tasks and writes its own success checks. But the success function comes from the same model that wrote the task, and “interesting” is decided by another language-model call, so there is no external reason to trust either. None of its tasks has been adopted as a benchmark by anyone else.

The rest of the line

The idea is over a decade old. PowerPlay (Schmidhuber 2011) already had a system search for the simplest task its current solver could not yet solve, then learn it, with the system inventing the tasks itself. POET (Wang et al. 2019) co-evolved obstacle courses and bipedal walkers, with forward progress as reward, inside one parametric family of environments. AI-GAs (Clune 2019) named the automatic generation of learning environments as one of three pillars toward general AI, but offered no method for it. The AI Scientist (Lu et al. 2024; v2 2025) poses research questions, runs experiments and writes papers, with a language model as the reviewer; one v2 paper scored above the acceptance bar at an ICLR 2025 workshop before the authors withdrew it. PaperBench (Starace et al. 2025) measures the reverse direction, replicating 20 ICML 2024 papers against 8,316 rubric items with a language-model judge; the best agent reaches 21.0%, and on a three-paper subset ML PhDs reached 41.4% against o1’s 26.6%. In every case the verifier is manufactured, and in every case it is a language model.

Order 3 as done by humans

Two order-3 steps taken by humans show what a good verifier decision looks like.

HumanEval (Chen et al. 2021) is 164 hand-written programming problems, each with a signature, a docstring and on average 7.7 unit tests, written by the authors so that none of them could already be in the training data. Five years later it is still 164 problems.

SWE-bench (Jimenez et al. 2023) started from a different observation: a merged pull request that changes test files carries, for free, a test that the fix made pass. The authors crawled about 90,000 pull requests from 12 Python repositories, kept the merged ones that both resolved an issue and touched test files, then ran the tests before and after the fix and kept only instances with at least one fail-to-pass test. The funnel went from 93,139 PRs to 11,407 candidates to 2,294 tasks. The best model at the time, Claude 2, resolved 1.96% of them. Within a year, OpenAI had 93 developers screen 1,699 instances to produce the 500-task Verified subset, and within three years the work in the order-2 section above had been built on top of it.

The two verifiers are comparable in quality: both are objective, run in seconds, parallelize and have low noise. They differ in one respect. HumanEval’s tests were written for the benchmark. SWE-bench’s tests were written by strangers to protect their own repositories, long before anyone thought of training on them. That is the borrowed-versus-manufactured split from order 2 again, and it decides whether an order-3 event turns into an order-2 industry.

RSI-Exam shows what human order-3 work looks like as a repeatable process. A domain expert proposes the task and its metric, then runs a weak baseline and a stronger reference solution to calibrate the scale. A developer packages the task as two isolated images with a declared artifact contract and hidden data. A reviewer applies a rubric of over 90 checks covering value, measurability, provenance and every leakage path, and a second reviewer analyzes a full agent trajectory against the anchors. Eighty-eight tasks have been built this way, all by hand.

Open questions

Laid out this way, the record is short. Agents already operate at order 1. Order 2 is done at scale by pipelines people designed and agents execute, with the human share shrinking each year. Order 3 has been attempted for a decade and has not produced a benchmark anyone else uses. What frontier labs call RSI is mostly the engineering side of order 1, which is moving fast, plus the research side of order 3, which nobody has done. Order is not a ranking of value: order-1 AlphaEvolve broke a 56-year-old bound and runs in production, and order-2 environments are a major ingredient in recent open-weights gains on SWE-bench Verified: SWE-Universe mid-training alone took Qwen3-Next from 50.3% to over 61%.

The concrete test for the next step is easy to state, but nobody has run it: hand an agent SWE-bench Verified and nothing else, and see whether it writes its own SWE-smith, including the decision that repository tests are the verifier. Behind it sits the question this post keeps returning to: can a machine notice that something like pull-request tests exists at all, or is that observation the part that stays human? And the failure mode is already visible one order down: DGM’s objective hacking was handled by hiding the checker from the agent, which is not an option once the agent is the one writing the checker.

Disclaimer

This post was drafted, fact-checked against the cited papers, and typeset by an AI coding agent (Claude Code) working with the author in the order-0 way described above: the author gave instructions and accepted or rejected each result; the agent’s work did not carry forward into a better agent. Errors that survived that loop are the author’s.

Citation

Cited as:

Lyu, Zhiheng. (Sep 2026). “How Far Has RSI Gotten in Post-Training?”. cogito233.github.io. https://cogito233.github.io/blog/three-orders-of-self-improvement/.

Or

@article{lyu2026rsi,
  title   = "How Far Has RSI Gotten in Post-Training?",
  author  = "Lyu, Zhiheng",
  journal = "cogito233.github.io",
  year    = "2026",
  month   = "Sep",
  url     = "https://cogito233.github.io/blog/three-orders-of-self-improvement/"
}

References