RPG · Reconstruct, Practice, Go Real

Overview

Develop robot skills through simulation practice.

RPG uses an offline dataset to decide what to practice in simulation. Execution feedback guides new skills, improvements to existing skills, and prompt revisions. The resulting shared skill library and system prompt transfer to a physical robot after calibration and hardware adaptation.

Read the abstract

We present Reconstruct, Practice, Go Real (RPG), a framework for autonomous improvement of robot execution systems without updating model weights. RPG identifies manipulation capabilities in an offline dataset and constructs related practice tasks in simulation. During practice, RPG uses execution feedback, privileged simulator state, and available dataset videos to diagnose failures. It develops new reusable symbolic skills, refines existing skills, and revises the system prompt based on these diagnoses. Cross-task evaluation tests individual candidate changes and merged revisions before they are retained for reuse. At test time, a multimodal LLM uses the resulting system prompt and skill library to coordinate perception and robot control. On held-out initializations of 22 manipulation tasks, RPG improves task success from 28.6% after the first practice round to 95.0% after 15 rounds, outperforming all evaluated baselines, including ASPIRE (75.5%) and CaP-Agent0 powered by GPT-6 Astra Pro (60.0%). After a common calibration and hardware-adaptation procedure, the frozen system succeeds in all 30 physical trials, with ten trials on each of three tasks.

RPG overview: Reconstruct builds simulation practice tasks from a real-world dataset; Practice revises shared skills; Go Real deploys the frozen system on a physical robot.

Reconstruct: build practice tasks. Practice: develop skills and validate revisions across tasks. Go Real: adapt to the physical robot.

Results

RPG succeeds in all 30 physical trials.

Three tasks, ten trials each, with the same calibration, hardware adaptation, and budgets for all methods. No task-specific tuning or system revisions. Autonomous retries are allowed within a trial; human assistance and manual resets are not.

RPGGemini 3.8 Flash

30/30

intermediate stage reached20/20

BaselineCaP-Agent0 · GPT-6 Astra Pro

14/30

intermediate stage reached14/20

BaselineCaP-Agent0 · Gemini 3.8 Flash

3/30

intermediate stage reached13/20

Large counts: task completion. Smaller counts: ball placed in drawer or bowl lifted to center, before closure or handover. Across these two tasks, GPT-6 CaP-Agent0 finishes 5/20 trials; RPG finishes 20/20.

RPG

00:00real time

Result · 10 trials10 / 10ball placed 10 / 10 · drawer closed 10 / 10

RPG (ours)

Gemini 3.8 Flash

15×

Baseline

00:00real time

Result · 10 trials4 / 10ball placed 6 / 10 · drawer closed 4 / 10

CaP-Agent0

GPT-6 Astra Pro

15×

RPG

ball placed in drawer 10/10

→

drawer closed with ball inside 10/10

CaP-Agent0 · GPT-6

ball placed in drawer 6/10

→

drawer closed with ball inside 4/10

Store Ball in Drawer. Success requires fully closing the drawer with the ball inside. The videos show all ten trials at 15× speed, with the final tally at the end.

How

Improve the skills and prompt, not the model weights.

The Runtime Agent is a multimodal LLM that calls reusable code functions, or skills, for perception and control. RPG revises these skills and the agent’s system prompt between practice rounds.

Stage 01

Reconstruct

Identify capabilities from dataset videos and descriptions, then build simulation tasks to practice them. “Organize sunglasses,” for example, becomes closing a loaded case. Humans review the success checkers; tasks and checkers then stay fixed.

Stage 02

Practice

Attempt the tasks, diagnose failures, and revise shared skills and the prompt. Test each candidate and merged revision across tasks before retention.

Stage 03

Go Real

Deploy after the same calibration and hardware adaptation used for CaP-Agent0. No task-specific tuning, simulator state, or offline failure analysis during physical execution.

RPG architecture: Runtime and Privileged agents share a skill library; a Video Analyzer diagnoses failures; an Implementor proposes candidates; a Merger integrates them.

The Privileged Agent uses the same model and skills with access to simulator state. Comparing its executions with the Runtime Agent’s helps diagnose failures. The Video Analyzer also uses related dataset videos when available; the Implementor proposes revisions, and the Merger selects and combines eligible changes.

One practice round

Run five Runtime and five Privileged Agent episodes per task
Diagnose failures on the 17 development tasks
Propose at most one revision per failing development task
Test each candidate across all 22 tasks
Select eligible revisions, merge them, and test again
Retain only if validation passes and the Merger accepts

Cross-task validation

pass if mean success ↑ and each task loses ≤ 1/5

Shared skills can help one task but hurt another. Test candidates and merged systems on five fixed initializations per task. Passing candidates are eligible for merging, not automatically accepted.

Full simulation evaluation

28.6% success after round 1.
95.0% after round 15.

After practice, we test every saved round-end system on 22 tasks × 10 new starting configurations, held out from revision selection. Runtime: Gemini 3.8 Flash. Offline analysis and revision: Fable 5.1.

Task success across practice rounds

Click a heatmap row to show that task’s results on the curve.

RPGBaseline results

Round 3 → 4: no skill edits; prompt and observation-handling revisions, +30.5 percentage points. Round 4 → 15: library changes with a fixed prompt, +21.4 points.

Per task, per round

Each cell counts successful trials out of ten held-out initializations.

010

swipe to see all rounds →

D: development. V: validation. H: feedback-held-out. Only D tasks generate task-specific revisions; all tasks enter regression checks.

How do diagnostic inputs affect improvement?

Same initial system. Five practice rounds, then 220 held-out episodes per variant.

Video Analyzer	Privileged Agent	Success
Yes	Yes	78.2%
No	Yes	51.8%
Yes	No	50.9%
No	No	50.9%

Removing either component reduces success from 78.2% to about 51%.

Do library revisions improve task success?

Only the library changes. Model, prompt, perception, and budgets stay fixed. Ten paired trials per task.

Task	Before	After
Two-arm lift	6/10	10/10
Place dish on rack	5/10	9/10
Nut assembly	5/10	8/10
Place bottles in bin	10/10	10/10
Success rate	65.0%	92.5% +27.5 pp

A separate library-snapshot comparison: three tasks improve, and bottle-placement success is preserved.

Full per-task comparison with baselines

Task	CaP-Agent0 Gemini 3.8 Flash	CaP-Agent0 GPT-6 Astra Pro	CaP-Agent0 Opus 5	CaP-Agent0 Fable 5.1	RATs Gemini 3.8 Flash	ASPIRE Gemini 3.8 Flash	RPG Gemini 3.8 Flash
CaP-X tasks
Lift cube	9/10	10/10	10/10	10/10	10/10	10/10	10/10
Nut assembly	4/10	6/10	4/10	5/10	0/10	3/10	9/10
Restack cubes	6/10	9/10	8/10	7/10	5/10	10/10	10/10
Spill wipe	2/10	10/10	6/10	9/10	5/10	8/10	10/10
Stack cubes	8/10	10/10	9/10	7/10	10/10	10/10	10/10
Two-arm handover	5/10	3/10	3/10	6/10	2/10	6/10	10/10
Two-arm lift	2/10	3/10	3/10	2/10	6/10	8/10	10/10
Group mean	51.4%	72.9%	61.4%	65.7%	54.3%	78.6%	98.6%
ABC-derived and analogous tasks
Extract from fixture	8/10	10/10	2/10	2/10	2/10	7/10	10/10
Insert into fixture	2/10	1/10	0/10	0/10	4/10	2/10	10/10
Place dish on rack	1/10	9/10	2/10	5/10	4/10	7/10	10/10
Serve onto plate	9/10	10/10	9/10	8/10	2/10	10/10	9/10
Fold towel	1/10	0/10	0/10	0/10	0/10	0/10	3/10
Transfer object	0/10	0/10	0/10	5/10	1/10	9/10	9/10
Lift rod	8/10	10/10	9/10	10/10	10/10	10/10	10/10
Pick up cup	10/10	10/10	10/10	10/10	10/10	10/10	10/10
Place bottles in bin	0/10	1/10	0/10	0/10	0/10	3/10	10/10
Close sunglasses case	9/10	6/10	6/10	6/10	9/10	10/10	10/10
Close drawer	6/10	5/10	6/10	3/10	9/10	10/10	10/10
Sort cubes (standard)	3/10	2/10	4/10	5/10	1/10	10/10	9/10
Sort cubes (extended)	0/10	0/10	0/10	0/10	0/10	8/10	10/10
Stack blocks	4/10	7/10	0/10	1/10	2/10	5/10	10/10
Transport cup	10/10	10/10	9/10	7/10	0/10	10/10	10/10
Group mean	47.3%	54.0%	38.0%	41.3%	36.0%	74.0%	93.3%
Mean success	48.6%	60.0%	45.5%	49.1%	41.8%	75.5%	95.0%
Total	107/220	132/220	100/220	108/220	92/220	166/220	209/220

Ten held-out initializations per task. CaP-Agent0 and ASPIRE are reproduced from released code. RATs uses its native any-time success criterion; the other methods are scored on the final outcome.

Watch the 22 simulation tasks

One episode per task, run by the round-15 RPG system in MuJoCo.

Lift cubeCaP-X

Nut assemblyCaP-X

Restack cubesCaP-X

Spill wipeCaP-X

Stack cubesCaP-X

Two-arm handoverCaP-X

Two-arm liftCaP-X

Extract from fixtureABC

Insert into fixtureABC

Place dish on rackABC

Serve onto plateABC

Fold towelABC

Transfer objectABC

Lift rodABC

Pick up cupABC

Place bottles in binABC

Close sunglasses caseABC

Close drawerABC

Sort cubes (standard)ABC

Sort cubes (extended)ABC

Stack blocksABC

Transport cupABC

Additional physical tasks

Use the shared skills on additional physical tasks.

All demos run the frozen RPG (ours) system with Gemini 3.8 Flash at low reasoning effort. The first five tasks are zero-shot: instruction only, with no task-specific real-world demonstrations, tuning, or retry-based adaptation. Sort Utensils, Insert Cylinder, Serve Fruit onto Plate, and Collect Items into Crate are further qualitative examples. Videos show execution at 15× speed, not measured success rates.

15× · 00:00 real time

Open Scissors

15× · 00:00 real time

Uncap Marker

15× · 00:00 real time

Pull out Tissue

15× · 00:00 real time

Erase Whiteboard

15× · 00:00 real time

Unscrew Bottle Cap

15× · 00:00 real time

Sort Utensils

15× · 00:00 real time

Insert Cylinder

15× · 00:00 real time

Serve Fruit onto Plate

15× · 00:00 real time

Collect Items into Crate

Items are tossed onto the table by a person during execution

Skill library

38 skills after 15 rounds of practice.

Each skill is a reusable code function the Runtime Agent can call. Tiers are call depth, not difficulty: which skills a skill actually calls in its source. That decides which layer to inspect when something fails.

38skills

14leaf

17composed

7compound

8query-only

Leaf

Calls no other skill

Talks to the controller, cameras, or the vision model directly. 11 are marked primitive in code; move_both, verify_grasp, and ask_runtime_model are composed code that likewise call nothing.

Composed

Calls leaves only

One complete, single-purpose action with its own approach, verification, and retreat logic.

Compound

Calls other composed skills

A multi-stage procedure; the deepest is pick_and_place at depth 3. When it fails, check whether the layer below failed first.

Browse all 38 skills by function

Perception & queries 8 · query-only, scene unchanged

Answer questions without producing motion; safe to call freely. segment_objects runs on the local SAM3 server; only the last two call the paid runtime VLM.

capture_imageleafTake a photo from the top or a wrist camera.
get_stateleafRead joint angles, fingertip positions, and gripper openings of both arms.
locate_objectsleafFind objects by color in the top-down image; any returns every colored blob.
segment_objectsleafFind objects by text description (SAM3). Returns world position, size, confidence.
pixel_to_worldleafConvert an image pixel to a table-plane world coordinate, with optional object-height correction.
solve_ikleafAsk whether a position and wrist pose are reachable; returns joint margin. Does not move.
verify_graspleafAsk the vision model, via the wrist camera, whether the grasp really succeeded. Called by pick.
ask_runtime_modelleafSend one image and a free-form question to the vision model; get text back.

Basic motion 7

The layer every action is built on. Wrist yaw, pitch, and roll all flow through move_to.

move_toleafMove one fingertip to a world coordinate; wrist angles, linear or joint path, and correction rounds are selectable.
set_gripperleafOpen or close to a target opening or physical width; reports the measured opening.
reset_armsleafReturn both arms smoothly to home.
waitleafHold still for a few seconds so things settle.
run_sequenceleafRun several skills in one model turn; stops at the first failure.
move_abovecomposedMove to a safe height directly above an xy position, as a transit or viewing pose.
move_bothleafMove both arms to their targets in sync, keeping the fingertips' relative pose. For jointly held objects.

Grasp & place 8

The most used group. pick and place are the core; the rest handle specific shapes or situations.

pickcomposedTop-down grasp: open, approach, descend, close, lift. Checks gripper width plus vision; retries on an empty grasp.
placecomposedCarry the held object to a position, lower, release, retreat.
pick_tall_objectcomposedGrasp tall bottles with a tilted wrist; 60° reaches what a vertical approach cannot.
set_downcomposedGently rest the held object on a support whose height is re-measured from depth.
pick_rimcompoundGrasp plates wider than the gripper by their rim; re-plans once if the grasp is skewed.
pick_and_placecompoundGrasp and place in one call; hands over automatically when the target is in the other arm's reach.
pick_and_place_tall_objectcompoundGrasp a tall object and drop it into a container, checking with depth whether it is upright or fallen.
transfer_between_armscompoundPass an object between hands via the table. This robot cannot hand over in the air, so it stages in a shared zone.

Containers 4

Actions involving an opening. All re-measure the opening or rim from depth, then descend in steps.

insert_into_containercomposedPut the held object into an opening: align, descend 1 cm at a time, shift and retry when stuck.
place_into_bincomposedDrop into a taller bin: trace the rim to find the opening, never release from height.
insert_onto_pegcomposedSlide a holed object (nut, ring) onto a post; confirms it is seated before releasing.
extract_from_containercompoundPull an inserted object out, verifying the object came out rather than the whole container.

Contact & pushing 5

No grasping: force is applied with the closed gripper. Contact depth is defined by fingertip height above the table.

pushcomposedPush an object on the table to a target with the closed gripper.
push_until_stopcomposedPush until motion stops. Closing drawers and lids.
sweep_areacomposedParallel back-and-forth strokes over a rectangle, pushing debris past its boundary.
hold_and_pushcomposedOne hand pins, the other pushes; monitors whether the pinning hand drifts.
pull_toward_basecomposedRake an out-of-reach object closer with the open gripper, then grasp normally.

Cloth 5

Thin, soft material needs its own moves: grasp by pressing the pads to the table and closing, not by aiming at object height.

pinch_clothcomposedPress pads to the table and close to pinch one layer; opening width tells how many layers and whether it slipped.
bimanual_foldcompoundEach hand pinches a corner; fold one edge onto the opposite edge in sync.
align_cornercomposedNudge a slightly misplaced cloth corner into position with one hand.
repair_cornercomposedAfter folding, re-carry one short corner to its target.
fold_clothcompoundFull folding routine: find corners, pinch, carry, lay down, repair.

Bimanual 1 · plus move_both, bimanual_fold, transfer_between_arms above

Motions that need both hands applying force at once.

bimanual_liftcomposedEach hand grasps one handle; lift in sync and keep level. A one-handed lift tilts and fails the check.

Examples of skill refinement

Check what the object does, not just where the gripper moves.

The revised skills verify grasp retention, object alignment, and placement, rather than relying on gripper motion alone. Results below use five development trials, separate from the held-out evaluation.

Case study: a failed bottle placement is diagnosed, the shared pick-and-place skill is revised, and development success rises from 1 of 5 to 5 of 5.

Bottle placement. A collision leads to full-path checks, release inside the bin, and placement verification. Development success improves from 1/5 to 5/5.

Align the nut’s hole with the peg

NUT ASSEMBLY · ROUND 4 → 5 · 1/5 → 5/5 · prompt unchanged

before

grasp→align gripper to peg→push down→release

after

grasp handle→test lift→verify retention→estimate hole offset→align hole to peg→incremental insertion→verify seating→release

Verify both grasps before lifting

TWO-ARM LIFT · ROUND 11 → LATER · 3/5 → 5/5

before

approach (each arm)→grasp→lift

after

localize handles→check both paths→verify both grasps→paired test lift→synchronized lift→verify object rise

Citation

BibTeX

@misc{wang2026rpg,
      title={Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents}, 
      author={Yen-Jen Wang and Haozhe Jiang and Shuying Deng and Haoru Xue and Weirui Ye and Rocky Duan and Nika Haghtalab and S. Shankar Sastry and Pieter Abbeel and Haozhi Qi},
      year={2026},
      eprint={2610.02204},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2610.02204}, 
}

Acknowledgments. We thank Zi Wang and Bike Zhang for helpful discussions and feedback. The website template is adapted from vision-language-kinematics.github.io.

Develop robot skills through simulation practice.

RPG succeeds in all 30 physical trials.

Improve the skills and prompt, not the model weights.

Reconstruct

Practice

Go Real

One practice round

Cross-task validation

28.6% success after round 1.95.0% after round 15.

Task success across practice rounds

Per task, per round

How do diagnostic inputs affect improvement?

Do library revisions improve task success?

Use the shared skills on additional physical tasks.

38 skills after 15 rounds of practice.

Calls no other skill

Calls leaves only

Calls other composed skills

Perception & queries 8 · query-only, scene unchanged

Basic motion 7

Grasp & place 8

Containers 4

Contact & pushing 5

Cloth 5

Bimanual 1 · plus move_both, bimanual_fold, transfer_between_arms above

Check what the object does, not just where the gripper moves.

Align the nut’s hole with the peg

Verify both grasps before lifting

BibTeX

28.6% success after round 1.
95.0% after round 15.