From Watch to Imagine

Steering Long-horizon Manipulation via

Human Demonstration and Future Envisionment

1HKUST(GZ), 2HKU, 3HKUST
*Co-first Author, ‡Project Lead, †Corresponding Author

This work introduces Super-Mimic, a hierarchical framework that goes beyond simple imitation of human actions, enabling task modifications and new skill transfers, and operates in a zero-shot manner without requiring task-specific training data.

Abstract

Super-Mimic Overview

Generalizing to long-horizon manipulation tasks in a zero-shot setting remains a central challenge in robotics. Current multimodal foundation based approaches, despite their capabilities, typically fail to decompose high-level commands into executable action sequences from static visual input alone. To address this challenge, we introduce Super-Mimic, a hierarchical framework that enables zero-shot robotic imitation by directly inferring procedural intent from unscripted human demonstration videos. Our framework is composed of two sequential modules. First, a Human Intent Translator (HIT) parses the input video using multimodal reasoning to produce a sequence of language-grounded subtasks. These subtasks then condition a Future Dynamics Predictor (FDP), which employs a generative model that synthesizes a physically plausible video rollout for each step. The resulting visual trajectories are dynamics-aware, explicitly modeling crucial object interactions and contact points to guide the low-level controller. We validate this approach through extensive experiments on a suite of long-horizon manipulation tasks, where Super-Mimic significantly outperforms state-of-the-art zero-shot methods by over 20%. These results establish that coupling video-driven intent parsing with prospective dynamics modeling is a highly effective strategy for developing general-purpose robotic systems.

Super-Mimic Framework

We propose Super-Mimic, a hierarchical framework that combines the HIT module for task planning and the FDP module for subtask execution to tackle zero-shot long-horizon robotic manipulation.

Super-Mimic Framework

The Human Intent Translator (HIT): This module parses complex human actions into transferable, language-guided robotic plans. It extracts key action frames from human videos and generates language-conditioned, transferable subtask plans through multimodal reasoning.

HIT Module

The Future Dynamics Predictor (FDP): This module employs video generation dynamics-aware visual trajectories that capture object interactions and contact constraints.

FDP Module

These imagined futures guide low-level control with enhanced robustness in cluttered, open-world environments.

Experimental Settings

We evaluate Super-Mimic on three challenging long-horizon scenarios, each comprising 5 to 8 subtasks:

  • Meal Preparation: A 5-subtask sorting task requiring the robot to categorize fruits and snack boxes into different containers.
  • Tidy Up the Desk: A 5-subtask task where the robot must use commonsense reasoning to sort trash from useful items into a bin and a basket.
  • Irregular Traversal: An 8-subtask task with non-logical pick-and-place motions, designed to be difficult to describe with language.
Experimental Platform

Our experimental platform: a 7-DoF xArm7 arm and a third-view Orbbec RGB-D camera.

Our experiments demonstrate that Super-Mimic significantly outperforms state-of-the-art zero-shot methods by over 20% on these tasks.

Experimental Results

We evaluate Super-Mimic on three challenging long-horizon scenarios and compare against state-of-the-art zero-shot methods ReKep and MOKA. The results show that Super-Mimic significantly outperforms the baselines by over 20%.

Experimental Results

Key Results:

  • Super-Mimic achieves 50% TSR on Meal Preparation, more than doubling the 20% TSR of the strongest baseline (MOKA).
  • On Irregular Traversal, Super-Mimic achieves 20% TSR while baselines fail entirely.
  • The framework demonstrates unique flexibility in Video + Constraints and Skill Generalization settings.

BibTeX

@article{ye2025watch,
    title={From Watch to Imagine: Steering Long-horizon Manipulation via Human Demonstration and Future Envisionment},
    author={Ye, Ke and Zhou, Jiaming and Qiu, Yuanfeng and Liu, Jiayi and Zhou, Shihui and Lin, Kun-Yu and Liang, Junwei},
    journal={arXiv preprint arXiv:2509.22205},
    year={2025}
}