Live: one human video, thousands of robot demonstrations. Episodes our pipeline generated in
IsaacSim, replayed in your browser on the xArm7. Switch between R2R2R+, which varies only the poses of the one
demonstrated mug and coffee machine, and R2Sem2R, which also swaps in digital cousins, and run nine environments in
parallel. In every episode the carry replays the mug trajectory tracked from the single human video (inset). Drag to orbit.
Teleoperation pays for every demonstration, and every new object instance pays again. Real2Sem2Real (R2Sem2R) turns
one human RGB video into thousands of robot demonstrations across the object instances of a category,
with no depth sensor and no teleoperation. It tracks the demonstrated objects, generates
digital cousins of them from photographs, aligns the cousins to the demonstration by their semantics,
and replays the demonstration for every cousin pairing in simulation.
Overview. One human video, the four steps of the pipeline, real rollouts on both
robots and the results. The full supplementary video covers
tracking, semantic alignment, generation in IsaacSim and MuJoCo, and every real-robot task.
01 Method
How it works
From one human video to robot policies, step by step. Most steps are live 3D: drag to orbit.
One human RGB video of the task and a phone scan of the objects.
Each object becomes a Gaussian splat with DINOv3 features, tracked through the video from RGB alone.
The human grasp is retargeted to a parallel-jaw gripper on the demonstrated mug.
Cousin mugs are aligned to it by semantic ICP, and the grasp carries over.
Coffee machines are aligned on the tray with task-weighted ICP.
Thousands of episodes are generated in IsaacSim and MuJoCo.
A policy trained on them runs on a real xArm7 and Franka Panda.
02 Tasks
Every task and every cousin, on both robots
Generated episodes of the four tasks, replayed in 3D. Pick a task, a robot and the cousins: the
object’s path is drawn as the robot carries it, and the carry replays the trajectory tracked from the one human
video. Drag to orbit; drag the timeline to scrub.
The interactive replay needs JavaScript and WebGL.
03 Alignment
Semantic alignment against SimFoundry and geometry-only ICP
Each method aligns eight cousin coffee machines to the demonstrated one. The demonstrated mug is then
dropped in MuJoCo at the goal each alignment gives, and it has to land on the tray.
SimFoundryGeometry-only ICPR2Sem2R (ours)
Three of the eight cousin machines, one per row. Each cousin is placed by the alignment in IsaacSim, then the
mug is released at the goal in MuJoCo. A tick marks a mug that comes to rest on the tray.
SimFoundry23% (29/128)
Geometry-only ICP37% (47/128)
R2Sem2R (ours)82% (105/128)
Mugs that rest on the tray: 8 cousin machines × 16 mugs per method. On the demonstrated machine
itself, 16 of 16.
SimFoundry: its released code scales the cousin to the size of the demonstrated machine, picks
its front with a VLM and copies the demonstrated pose. Geometry-only ICP: bidirectional ICP on the splats,
without DINOv3 features. R2Sem2R: semantic ICP, tray-weighted ICP and the upright projection.
A mug rests when it ends within 3 cm of the goal across the tray and 2.5 cm in height, tilted
less than 20°. SimFoundry’s front-picking step ran with a different VLM than the one in its release.
04 Rollouts
Real-robot rollouts
We fine-tune the π0.5 vision-language-action model on three kinds of data and evaluate on real
robots: teleoperation, 150 real demonstrations collected across several objects; R2R2R+, 2,000
generated episodes of the single demonstrated object pair, a strengthened Real2Render2Real run through our
tracker, grasp selection and simulators; and R2Sem2R, 2,000 generated episodes across digital cousins.
Clips are sped up (the factor is under each clip). The rates under each grid count every real-robot
trial of the evaluated policies, 50 per policy and 40 for the drill.
R2Sem2R on the real xArm7, all four tasks.
Mug on coffee machine · 3×Pour into bowl · 3×Hang on mug tree · 4×Drill reorient · 5×
The pour clip comes from an R2Sem2R policy trained on MuJoCo-generated data.
R2Sem2R on the real Franka Panda, all four tasks.
Mug on coffee machine · 3×Pour into bowl · 2×Hang on mug tree · 4×Drill reorient · 4×
Against the baselines on the xArm7.
On the real xArm7, ours sets the mug on the machine.
Mug on coffee machine. The demonstrated pair is the white mug and the black Keurig. Each success ends with the mug upright on the tray and the gripper released.
Against the baselines on the Franka Panda.
On the real Franka Panda, ours sets the mug on the machine.
R2Sem2R (ours) · success · 3×Teleop · success · 3×R2R2R+ · fails: never lifts · 3×R2R2R+ · fails: knocks it over · 2×
Mug on coffee machine. The demonstrated pair is the white mug and the black Keurig.
Hanging a mug on the Panda: the same mugs for both methods.
R2Sem2R (ours) · on the branch · 4×Teleop · hangs it · 2×R2Sem2R (ours) · hangs it · 4×Teleop · hangs it · 2×
Top: the white mug. Bottom: a brown mug. Success: the handle hooks over the low blue branch.
Pouring on the Panda: the same mug and bowl for both methods.
Top: the demonstrated red drill. Bottom: a yellow cousin drill. Successes end with the drill upright and the gripper released.
More R2Sem2R rollouts: other mugs, machines and drills.
Other mugs on other coffee machines, on the xArm7.
Cream · white · 3×Cream · pink · 3×Navy · red · 4×Navy · brown · 4×Teal · red · 3×Red · pink · 3×
R2Sem2R policies on cousin mugs and coffee machines (labels: machine · mug), trained on 1,320 to 2,000 generated episodes from IsaacSim and MuJoCo. Every clip ends with the mug upright on the tray and the gripper released.
Other mugs on other coffee machines, on the Franka Panda.
Keurig · orange · 3×Small black · pink · 3×Teal · white · 3×Teal · navy · 4×Cream · orange · 4×Cream · black · 3×Navy · black · 3×Navy · pink · 4×
Eight cousin pairs, all from the same R2Sem2R policy (labels: machine · mug). Each ends with the mug upright on the tray and the gripper released.
Other mugs on the tree, and a second drill, on the xArm7.
Navy mug · 4×Red mug · 3×Pink mug · 3×Drill set upright · 2×
Success: the mug hangs on the blue branch after the gripper releases it; the drill ends upright and released.
05 Results
Results
Real-robot success rate on the xArm7 (top) and Franka Panda (bottom). Whiskers: Wilson 95% interval.
Digital cousins beat a single object pair on every task and robot, by 26
to 54 points, and match or beat teleoperation on every one, with no human time beyond the one demonstration.
Object diversity versus data source on placement. One pair: 150 teleop demos or 2,000 R2R2R+ episodes on the demonstrated pair. Several objects: teleop on 4×4 pairs, or R2Sem2R. Whiskers: Wilson 95% interval.Success versus training-set size on xArm7 placement. Solid: several objects; dashed: demonstrated pair; bands: Wilson 95% interval.Demonstrations against wall-clock time (log-log) for 1 and 10 teleoperators or GPUs; R2R2R+ coincides with R2Sem2R and is omitted. Strips show where each method’s time goes, shaded as human or GPU time. Robot setup, shared by all methods, is not shown.
06 MuJoCo
A generated episode, replayed live in MuJoCo
Interactive: one of our generated MuJoCo episodes, simulated live. MuJoCo, compiled to
WebAssembly, re-simulates the scene the episode was generated in. The robot replays the episode's recorded joint
targets at 10 Hz while the mug, the table and the coffee machine are simulated with contact; as in our data
generation, a soft weld holds the mug once the finger pads close on the handle and releases it on the tray. The
generator keeps only episodes whose mug comes to rest on the tray, and each of the five object pairs ends on the tray
when replayed here. Drag to orbit. The arrows show the forces on the mug, longer for larger forces: from the fingers in red, the table in blue and
the coffee machine in orange. While the hand carries the mug, the red arrows at the fingertips show the force the
weld applies to hold it. The panel gives the totals; resting on the table or the tray, or carried at a steady pace,
they add up to the mug's weight, 2.9 N.
07 Limitations
Limitations
A photo per cousin
Every cousin needs at least one photograph. Generating a category from text alone is future work.
A known supporting surface
Monocular depth is up to scale, which we resolve per scene from the recovered ground plane, so the
demonstration must take place on a known supporting surface.
A clean grasp in the video
Grasp recovery needs a moment where the hand holds the object rigidly. Sliding or tool use without a
grasp falls outside its scope.
Rigid objects only
The tracker estimates one rigid pose per object, so soft and deformable objects are not supported.
Articulated objects such as drawers, doors and cabinets are the natural next step.
