RoboCoach

World Models as Active Coaches for Compositional Robot Skills

Jiajun Liu^{1, 2, 5, *} Yifan Chen^{2, *} Yichao Liu^{2, *} Jiayi Zhang³ Ruoqu Chen^{2, 5} Shaoxuan Xie⁴ Guocai Yao⁴ Mengdi Xu^{2, 5, †} Sen Cui^{2, 4, †, ‡} Changshui Zhang^{2, †}

RoboCoach identifies the first unfinished subtask in world-model rollouts and aggregates failures to request targeted demonstrations for the corresponding skill expert.

VideoWatch teaser

PaperarXiv CodeCoachWorld ModelHugging Face

1Renmin University of China
2Tsinghua University
3University of Nottingham
4Beijing Academy of Artificial Intelligence
5Shanghai Qizhi Institute

* Equal contributions† Corresponding authors‡ Project leaders

World models can turn imagined failures into guidance for what to learn next.

RoboCoach in action

Watch the teaser

System overview

From imagined execution to targeted improvement

01

Route

Send the first unfinished subtask to its expert.
02

Imagine

Predict the routed expert's closed-loop rollout.
03

Diagnose

Locate the first missed deadline and responsible expert.
04

Improve

Update only that expert's LoRA with targeted demonstrations.

Execution loopPredicted video becomes the expert's next observation until the subtask is complete.

Coaching loopRepeated first timeouts trigger targeted LoRA updates; the world model, router, judge, and VLA backbone stay fixed.

Case Study · Tea Making

A coaching loop, through one task

Turn one instruction into an expert sequence

Task instruction“Place the tea bag into the cup, then pour water to brew tea.”

Ordered subtasks → expert roles

Selected subtaskPick up the tea bag

Dispatched expert rolePick expert

Next in the plan: place the tea bag in the cup.

Route

The router selects the first unfinished subtask and dispatches its expert.

Read the four module roles

Route · Expert dispatch

Split the tea-making task into pickup, placement, and pouring. Dispatch the matching expert for each active subtask in that order.

Closed-loop improvement

Three coaching rounds across four platforms

Real-robot complete-task success 150 additional demonstrations per platform · Same budget

Franka

+45.0 percentage points

RoboCoach75.0%

Single VLA + Uniform30.0%

AgileX

+36.3 percentage points

RoboCoach83.8%

Single VLA + Uniform47.5%

Comparison details

Budget: three rounds of 100 accepted subtask demonstrations in simulation or 50 per hardware platform.

Matched comparison: budgets, optimization, replay exposure, and evaluation trials align within each domain; success is reported at four checkpoints.

Shared start: RoboCoach and Single VLA + Uniform begin at 13.3% on Franka and 40.0% on AgileX.

Complete-task success with bootstrap confidence intervals on LIBERO, RoboTwin, Franka, and AgileX, grouped into simulation and real-robot experiments

At the final budget, RoboCoach leads on all four platforms.

Simulation ablation+3.4 pp LIBERO · +13.2 pp RoboTwin over a shared adapter, using the same targeted demonstrations.

Real-robot task suite

Core Tasks

Single-arm

3 tasks

Lamp switch and cord

Promptclose the laptop, then pull the lamp cord

Subtasks · Click to seek

Success rateSingle VLA → RoboCoach15%85%

Two buttons press

Promptpress the green button, then press the red button

Subtasks · Click to seek

Success rateSingle VLA → RoboCoach15%70%

Bread and pan

Promptput the bread into the pan, then close the lid

Subtasks · Click to seek

Success rateSingle VLA → RoboCoach10%70%

Bimanual

4 tasks

Block and drawer

PromptPlace the red block into the drawer, then close the drawer.

Subtasks · Click to seek

Success rateSingle VLA → RoboCoach60%85%

Shoes and box

PromptPut the shoes in the shoebox, then close the box.

Subtasks · Click to seek

Success rateSingle VLA → RoboCoach25%60%

Table setting

PromptWipe the table, then push the plate to the center of the table. Place the cup on the plate.

Subtasks · Click to seek

Success rateSingle VLA → RoboCoach65%100%

Tea making

PromptPlace the tea bag into the cup, then pour water to brew tea.

Subtasks · Click to seek

Success rateSingle VLA → RoboCoach10%90%

Compositional Generalization

RoboCoach: 35.0% average success across four held-out compositions. Single VLA + Uniform: 0% (20 trials per composition).

Reordering

The same subtasks, in a different order.

Seen

New

Recombination

Subtasks from different tasks, combined.

Seen

AB+CD

New

Continuation

Extend a task with subtasks from another.

Seen

AB+CD

New

ABCD

Reordering

Button reordering

Promptpress the red button, then press the green button

Composed subtasks · Click to seek

Success rateSingle VLA + Uniform → RoboCoach0%35%

Reordering

Shoe reordering

PromptPut the left shoe in the shoebox, then the right shoe, then close the box.

Composed subtasks · Click to seek

Success rateSingle VLA + Uniform → RoboCoach0%25%

Recombination

Button to lamp

Promptpress the green button, then pull the lamp cord

Composed subtasks · Click to seek

Success rateSingle VLA + Uniform → RoboCoach0%15%

Continuation

Table setting to tea making

PromptWipe the table, then push the plate to the center of the table. Place the cup on the plate, place the tea bag into the cup, then pour water to brew tea.

Composed subtasks · Click to seek

Success rateSingle VLA + Uniform → RoboCoach0%65%

Predictive model

CoachWorld: our camera-aware action-conditioned world model across embodiments

One model, two embodiments

A shared two-slot state format and calibrated cameras condition Wan2.2 TI2V-5B for Franka and bimanual robots.

Long-duration closed-loop interaction

The longest example runs for 85.8 s, with predicted frames fed back into the policy across successive video chunks.

Single-arm

Real-World

Close the laptop, then pull the lamp cord

85.8 s

Ground truthCoachWorld

Real-World

Pick up all the objects surrounding the white plate and arrange them on the plate

63.4 s

Ground truthCoachWorld

Bimanual

Real-World

Put both shoes in the shoebox, then close the shoebox

58.0 s

Ground truthCoachWorld

Real-World

Make sandwiches

56.2 s

Ground truthCoachWorld

How CoachWorld conditions the prediction

CoachWorld architecture showing multi-view heterogeneous robot data, unified camera calibration, sparse visual history, noised future video, text and state conditioning, and temporal state injection into Wan DiT blocks

01

Adapt native actions

Frozen embodiment adapters convert each expert's native action chunk into a shared Cartesian end-effector trajectory.
02

Unify embodiments

Two arm slots encode position, 6D rotation, gripper state, and an existence mask. Franka uses one slot and bimanual robots use both.
03

Ground every camera

Intrinsics and extrinsics project future states into each view, while trajectory rasters and camera rays condition the video denoiser.
04

Roll forward in closed loop

Sparse history, instruction, and future state condition the next predicted chunk. Its final frame becomes the next policy observation.

Action control

Steer predicted motion with action signals

Change sceneRoboCoin · Cloth

Loading action-control examples…

Action-conditioned video prediction

On DROID-180, mixed-domain CoachWorld leads in six of seven metrics.

Method	LPIPS ↓	FVD ↓	HSD ↑	nDTW ↑	DYN ↑	Physics adherence ↑	Instruction following ↑
CoachWorld mixed-domain	LPIPS ↓0.0996	FVD ↓51.71	HSD ↑0.2633	nDTW ↑0.2812	DYN ↑0.0959	Physics adherence ↑0.7467	Instruction following ↑0.8800

Compare with other methods

Matched inputs: the same visual histories and future actions; CoachWorld variants have matched training volume.

Metrics: LPIPS and FVD measure visual fidelity; HSD, nDTW, and DYN measure end-effector trajectory agreement.

Method	LPIPS ↓	FVD ↓	HSD ↑	nDTW ↑	DYN ↑	Physics adherence ↑	Instruction following ↑
Ctrl-World	0.2078	102.97	0.2234	0.2323	0.0668	0.6067	0.7867
Cosmos 3	0.2830	140.62	0.1702	0.1698	0.0571	0.7267	0.5467
OSCAR-2B	0.2248	101.92	0.1785	0.1838	0.0456	0.5267	0.5467
CoachWorld single-domain	0.1067	56.56	0.2441	0.2599	0.0723	0.7533	0.8667

Evaluation uses 180 DROID episodes. Physics Adherence and Instruction Following are mean blind human ratings on a 1–5 scale, divided by five. Bold marks the best value in each column.

Progress judge

Track progress and identify the first unfinished subtask

Live diagnosis stack_blocks_three RoboTwin · six routed stages

How the progress judge works

Frozen calibration: thresholds and deadlines come from held-out trajectories. A stage unfinished in the shortened clip is marked unconfirmed.

01
Score the active stage
At each frame, the progress judge estimates progress for the current routed subtask using only the rollout observed so far.
02
Confirm each transition
Crossing the calibrated 0.75 threshold confirms the active stage and moves the trace to the next subtask.
03
Assign the first timeout
P1–P5 cross threshold. P6 remains unresolved, localizing the failure to the active placement expert.

Policy rolloutRoboTwin

Progress judge completion scoreClick a stage to seek0.0 / 40.0 s

This rollout · Diagnosis place the blue block on the green block

Across rollouts Aggregate repeated failures

Rank subtask–expert pairs.

Targeted coaching place_expert Request demonstrations

How reliable is the progress judge?

Outcome F1: 88.21% on reference video; 83.84% on CoachWorld rollouts.

RoboMeter under reference and world-model observations
Observation source	Switch MAE (s) ↓	Switch recall @1.0 s (%) ↑	Spearman ρ ↑	TP/TN/FP/FN	Outcome precision (%) ↑	Outcome recall (%) ↑	Outcome F1 (%) ↑
Reference video	0.414	82.73	0.727	101/92/18/9	84.87	91.82	88.21
CoachWorld rollout	0.815	62.73	0.638	96/87/23/14	80.67	87.27	83.84

Full progress-judge comparison

Matched protocol: all six judges see the same causal video stream and frozen routing. Thresholds are calibrated on a held-out split.

Metrics: switch timing, terminal-stage rank agreement (Spearman ρ), and binary failure detection (failure is positive).

Progress-judge comparison on causal reference-video replay
Judge	Switch MAE (s) ↓	Switch recall @1.0 s (%) ↑	Spearman ρ ↑	TP/TN/FP/FN	Outcome precision (%) ↑	Outcome recall (%) ↑	Outcome F1 (%) ↑
LIV †	3.311	23.64	0.328	64/67/43/46	59.81	58.18	58.99
Contrastive λ †	1.905	28.18	0.603	95/80/30/15	76.00	86.36	80.85
Qwen3-VL	0.893	20.00	0.151	29/82/28/81	50.88	26.36	34.73
TOPReward †	1.603	33.64	0.330	76/55/55/34	58.02	69.09	63.07
SuccessVQA †	0.859	59.09	0.286	82/72/38/28	68.33	74.55	71.30
RoboMeter	0.414	82.73	0.727	101/92/18/9	84.87	91.82	88.21

† Adapted to the shared causal evaluation setting.

RoboCoach

Watch the teaser

From imagined execution to targeted improvement

Route

Imagine

Diagnose

Improve

A coaching loop, through one task

Turn one instruction into an expert sequence

Three coaching rounds across four platforms

Core Tasks

Single-arm

Lamp switch and cord

Two buttons press

Bread and pan

Bimanual

Block and drawer

Shoes and box

Table setting

Tea making

Reordering

Recombination

Continuation

Button reordering

Shoe reordering

Button to lamp

Table setting to tea making

CoachWorld: our camera-aware action-conditioned world model across embodiments

Close the laptop, then pull the lamp cord

Pick up all the objects surrounding the white plate and arrange them on the plate

Pile the white rope, towels, t-shirts and radish together on the right side of the table

Put the bread into the pan, then close the lid

Put the alphabet soup and cream cheese box in the basket

Put the white mug on the plate and the chocolate pudding to its right

Put the white mug on the left plate and the yellow-and-white mug on the right plate

Pick up the book and place it in the back compartment of the caddy

Move the marker to the right

Push the plate to the front of the stove

Put potatoes on a ceramic plate

Close the bottom drawer of the cabinet

Put both shoes in the shoebox, then close the shoebox

Make sandwiches

Place the tea bag in the cup, pour water, then return the teapot to the table

Place the red block in the drawer, then close the drawer

Fold towels

Stack three bowls

Place the three bottles into the black bin one at a time

Stack the green block on the red block

Stack the cup and cola can

Transfer the large, medium, and small blocks to the center in size order

Stack three colored cubes

Place two bread loaves in the breadbasket

How CoachWorld conditions the prediction

Adapt native actions

Unify embodiments

Ground every camera

Roll forward in closed loop

Action control

Steer predicted motion with action signals

Action-conditioned video prediction

Track progress and identify the first unfinished subtask

Score the active stage

Confirm each transition

Assign the first timeout

How reliable is the progress judge?