Human2Any: Human-to-Robot Transfer via
Constraint-Aware Compositional Planning
Conference on Robot Learning (CoRL) 2026
Human2Any learns object-centric interaction priors from human videos, composes them with robot-side feasibility reasoning and motion planning, and deploys them on a Franka arm and an RBY-1 humanoid mobile robot without real-world robot training data.
Human2Any achieves successful real-world rollouts across diverse task contexts and robot embodiments.
The idea
Abstract
Human videos are a scalable source of supervision for robot manipulation, as they are abundant and naturally capture rich object interactions. However, transferring human demonstrations to robots remains challenging due to embodiment mismatch, scene variation, and robot-specific feasibility constraints. We present Human2Any, a framework for learning reusable object-centric interaction priors from human videos without requiring real-world robot demonstrations in the target task contexts. Human2Any represents manipulation through object-object interaction motion, capturing task-relevant scene changes while abstracting away embodiment-specific details. It composes learned interaction priors with robot-side feasibility reasoning and motion planning, allowing the same human-derived knowledge to adapt to different embodiments, scene geometries, and task contexts. We validate Human2Any across diverse manipulation settings, including real-world experiments on a Franka tabletop setup and an RBY-1 humanoid mobile robot, demonstrating robust interaction-centric manipulation without real-world robot training data.
The method
Method overview
Interactive
Planning visualizer
Drag to rotate · Scroll to zoom · Right-drag to pan
From human videos
Human data visualizer
Rainbow trails: 3D tracks on the tool, colored as in the video · Orange: tool · Green: target · Axes: tool pose
Drag to rotate · Scroll to zoom · Right-drag to pan
Experiments
Real-world results
Franka tabletop
Pour In Bowl
The robot avoids the kettle, grasps the cup by its handle, and rotates it during the transfer motion to align the cup for pouring into the bowl.
Hang Mug Tree
The robot aligns its grasp to avoid non-target objects and reorients the mug on the fly to accurately place the handle onto the mug tree across various object arrangements.
Sort Utensils
The robot sequentially places a bowl onto a plate and a utensil onto the bowl, demonstrating long-horizon skill composition for different scene layouts.
RBY-1 humanoid mobile robot
Pour Cup
The robot securely grasps the cup and leverages coordinated whole-body motion to pour a variety of cups into the target container.
Use Roller
The robot grasps the handle of the roller precisely and moves to press the target dough.
Experiments
Simulation results
Setup Table
Our framework can solve ultra-long-horizon tasks by directly reusing and composing learned models, adapting to new tasks and scenarios.
Sort Into Cabinet
Our framework can handle articulated objects by generating coordinated motions over movable parts, adapting to complex object structures and interaction dynamics.
Extension
VLM task planning
Combined with VLM-based task planning, our framework can be readily extended into a robust TAMP system with automated task decomposition and feasibility-aware motion generation.
Hang Mug Tree
GPT 5.2
Opus 5
Pour In Bowl
GPT 5.2
Opus 5
Sort Utensils
GPT 5.2
Opus 5
Under the hood
Diffusion visualization
We visualize the particles during the diffusion process, with each particle corresponding to a full end-effector trajectory and color-coded according to its score. The execution of the best final particle is demonstrated in the accompanying video.
No Steering
Ours
Drag to rotate · Scroll to zoom · Right-drag to pan
Read the research
Human2Any
Human-to-Robot Transfer via Constraint-Aware Compositional Planning
Conference on Robot Learning (CoRL) 2026
@inproceedings{cheng2026human2any,
title = {Human2Any: Human-to-Robot Transfer via
Constraint-Aware Compositional Planning},
booktitle = {Conference on Robot Learning (CoRL)},
author = {Cheng, Shuo and Zhang, Chuye and Cueva, Alfred and
Garrett, Caelan Reed and Mandlekar, Ajay and
Xu, Danfei},
year = {2026}
}