A continual learning benchmark · 114 runs

Continual-ARC

Measuring Skill Acquisition over a Working Lifetime

Daniel Lenton, Yusha Arif and Haris Mahmood

Abstract

Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, the current benchmarks that measure such systems have several limitations: they generally focus on a single applied domain, they usually serve each instance only once, and most assume a single family of learner, so memory-based and weight-based systems are rarely compared on the same scale. We propose Continual-ARC, a lifelong stream built from the three ARC-AGI editions, whose tasks need no domain knowledge and which the models under test solve only in part. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for demonstrations, submit an answer, or take actions inside the harness, with retries permitted on an incorrect submission. Each instance is scored based on the share of this feedback budget (demonstrations and failed attempts) consumed. Since a learner may keep anything between instances, memory-based and weight-based systems are scored on the same scale. After the stream, the learner is frozen and tested on fresh instances of the tasks it met, which measures memorization, and on unseen tasks, which measures generalization. We run an agent harness with frozen weights, with and without persistence, on Qwen3.5-9B and GLM-5.3 Flash, comparing them with two prompting floors, a program-induction method and Qwen3.5-9B's weights trained to imitate GLM-5.3 Flash. We also define a second regime in which one system learns from several users simultaneously. Across our experiments, we show that persistence in the harness saves nothing measurable on returning tasks when the model chooses what to store. Most of them learn less than a floor that simply caches each task's demonstrations. In contrast, a simple method that asks the LLM for one program per task and keeps following demo reproduction outperforms them by a wide margin. Neither gains on unseen tasks, where only weight-based learning shows substantial gains. This opens the door to future weight-based continual learning that could teach models to drive harnesses in new domains.

Introduction

Learning over a working lifetime

Chollet (2019) argued that intelligence should be measured by how efficiently a system turns priors and experience into skill, on tasks that lie largely outside its experience and have a verifiable answer. ARC is the concrete instance of that argument, and on its current edition the gap between what a person solves in a minute and what a model solves from the same demonstrations is still significant. For agent systems that continually operate for days and weeks, a more pressing question is whether a skill acquired on one job is still there the next time a similar scenario is met, which we call memorization, and whether that experience makes the system better at similar jobs it has never met, which we call generalization. By design, ARC measures acquisition once, within a single episode, and does not ask either over longer horizons.

The benchmarks that do ask it have three limitations.

Each is built around one applied domain, such as coding or office work, in which the models under test already know a great deal from the pretraining data. A method's gains may come partly from drawing out that knowledge, and so may not carry over to a domain that the model is less familiar with.
Most serve a fixed set of instances, each seen once, so the same conceptual task never returns with a fresh input. This means a memorized answer cannot be told apart from a more abstract learned skill.
Most assume a single family of learner, gradient updates for fine-tuning suites and prompting and memory for agent benchmarks, so memory-based and weight-based systems are rarely compared on the same scale.

Continual-ARC is a lifelong stream built from ARC, where tasks need no knowledge of any domain beyond core-knowledge priors, such as objectness and counting, and the rule of each grid task has to be induced from a few demonstrations. A task can return after a dormant gap as often as the stream asks, each time with a new input that no cached answer fits. New tasks also keep arriving throughout the stream. Because a system may carry anything between instances, whether weights or notes, prompting, memory and weight updates all face the same stream and the same tests.

The benchmark

From one instance to a stream and two tests

One instance

An ARC task is a rule that turns an input grid into an output grid, and an instance of it is one input whose output the learner must produce. For each instance, the learner observes the task id, the input grid, and how many attempts and demonstration requests it has used on this instance, but it is told nothing about its position in the stream. It may submit a grid, and the environment answers only correct or incorrect. A correct submission ends the instance, and an incorrect one costs 1. It may instead request a demonstration, at a cost of 1, which returns one more input-output pair of the task, up to eight per instance. Failing to get the correct answer costs 16, even if there were demonstration requests to spare.

Step through a real one, or play it.

Test input

Demonstrations bought

None yet

Answers

None yet

DemonstrationsWrong answers0 of 16 spent

Hermes Agent meets task 46442a0e for the first time: the task's id and a test input, nothing more.

Hermes Agent on GPT-6 Luna, medium, step 21 of the ARC-AGI-1 stream. Open it in the replay

A stream of instances

The stream keeps a few tasks active at once, each worked in bursts with dormant gaps between them. A task can stay dormant for long enough to be forgotten, and since every instance is a fresh draw from the task's generator, no answer can be cached. The ARC-AGI-1 stream draws 25 tasks from the 400 and serves 200 instances, followed by two tests. Every system meets the same tasks in the same order, with the same instances.

The task played above is lit in it.

Point at a column to see which task it served; the task’s marks light up.

first meeting with the taskthe task recurstask active, in a spell

The score, and the reset

We score the instance by the share of this feedback budget it used, all of it if the instance fails. Because a failure costs the whole budget however it came about, failing is never cheaper than asking, and a demonstration is paid for only by an instance it helped to solve. U, the budget used over the stream, mixes what a learner could already do before the stream with what it learns along the way. To measure the learning alone, we need one more run, the reset: the same learner on the same stream and the same tests, with its state wiped before each instance. The share learned is 0 for a learner that keeps nothing, 1 for one that answers every returning task at no cost, and negative for one that does worse than the reset.

Every instance of the run above, and of three learners beside it on the same stream, at GPT-6 Luna, medium.

U 28.1P 90.5

Hermes Agent, as shipped, is the run played above. Its 200 instances average u 20.0 at a task's first meeting and 29.2 when it recurs, so U 28.1, and it solved 90.5 percent. Point at a bar for one instance.

Two tests after the stream

The stream measures what a learner keeps while it learns. To measure what it has at the end, we freeze the learner and serve it two tests, each a single round of 100 instances. The seen test consists of four fresh instances of each of the stream's 25 tasks, at instance seeds the stream did not use, and measures memorization. In the held-out test, which measures generalization, 100 evaluation tasks never seen on the stream are served once each, to test whether the learnings carry over to new tasks. A learner that keeps one program per task gets cheaper on the seen test but no cheaper on the held-out test, whereas a learner that has become a better solver of new tasks gets cheaper on both. On each test we read the saving as a share of the reset's whole spend on the test's instances, so that a learner whose reset already spends little is not marked down for having fewer points to save.

Two regimes: the individual and the collective

A deployed system may instead serve many people at once, learning from all of them, and the collective regime tests such a system. A run has several users, each following a separate stream over 25 tasks of their own, so that two users bring 50 tasks and four bring 100. A round serves one instance per user, concurrently, and the system may change its state only between rounds. All users share one state, a harness's notes as much as a trained model's weights, so that whatever a learner keeps, it learns from every user's games.

The benchmark

A track for each ARC-AGI edition

The three editions rise in difficulty, to the point where a model costing a cent per task and a frontier model are best measured on different editions. Each track is played under the same rules and read with the same measures, and serves fresh instances from generators. Most of our experiments focus on ARC-AGI-1, and the leaderboard has the second and third tracks' first runs.

Track	The stream's tasks	The held-out test	Users	At first sight
ARC-AGI-1	25 of its 400 training tasks, every instance fresh from Re-ARC's generators	100 of its evaluation tasks, with their own pairs	1 to 16	Claude Opus 5.5 solves 98.5% of its public tasks
ARC-AGI-2	25 of the 117 training tasks new to the second edition whose ARC-GEN generator passes our screen, every instance fresh from it	100 of the 114 evaluation tasks new to it, with their own pairs	1 to 4	Kimi K3.1 Pro solves 29% of the stream's tasks
ARC-AGI-3	15 of its 25 games, a task being a game's rules at its first level and an instance a fresh level from a generator and solver we wrote for it	fresh levels of the other 10 games	1	Kimi K3.1 Pro solves 25% of the held-out levels

A user brings 25 tasks of their own, so a track serves as many users as its pool holds blocks of 25. The paper's Appendix K defines the second and third tracks in full.

The learners · Qwen3.5-9B

Notes or weights, with the same teacher

We ask whether learning should go into the notes that an agent harness keeps or into the weights of the model that drives it. Either can be taught by the model itself, from its own verdicts, or by a stronger model. We compare learners on one student model along these two choices, who teaches and where what is learned is kept. The student is Qwen3.5-9B and the teacher GLM-5.3 Flash, both open models. Two prompting floors set the level that any learning method has to beat, and one engineered reference, a program-induction learner shaped to the domain, shows that the floors can be beaten.

Who teaches	Kept in the harness's notes	Kept in the student's weights
Taught by the student	Qthe harness as shipped QEgiven expert guidance, the program library's procedure
Taught by GLM-5.3 Flash	QTthe teacher plays the stream and writes the notes, and the student plays from them QTEthe teacher given the guidance QETEboth given it	QDimitates the teacher's move in each state the student met, the teacher reading a page of playing rules

In the rows where the teacher writes the notes, every game is played twice, once by each model. The teacher's demonstrations and verdicts on each instance go uncharged, so these rows are not learners the benchmark would verify: they are an experiment, an upper bound on what a teacher can pass on through a harness. The design is the paper's Table 3.

See how each learner did

Submit yours

Have your learner verified

Every task of the benchmark and every task's solver presented so far is public. External submissions are therefore verified by running the learner's code on a private set of 50 new rules that only we have seen: 25 for the stream and the seen test, and 25 for the held-out test. A learner that does not train its weights is verified automatically from an issue on the benchmark's repository, and one that does is run with us on GPUs we rent. Continual-ARC is meant to explain how a learner learns rather than to purely rank learners, so we give no single score, and we present any verified submission of a genuine learning algorithm, novel or not.

shell

git clone https://github.com/unifyai/continual-arc
cd continual-arc
uv venv && uv pip install -e ".[dev]"
uv run continual-arc data fetch
uv run continual-arc schedule make --config arc1 \
  --seed 0 --out arc1-ws0.json

How to run a learner, submit it and have it verified

Cite

BibTeX

Continual-ARC: Measuring Skill Acquisition over a Working Lifetime. Daniel Lenton, Yusha Arif and Haris Mahmood, unify, 2026.

bibtex

@misc{lenton2026continualarc,
  title  = {Continual-ARC: Measuring Skill Acquisition over a Working Lifetime},
  author = {Lenton, Daniel and Arif, Yusha and Mahmood, Haris},
  year   = {2026},
  url    = {https://continualarc.org}
}