Official project page for our ICLR 2026 paper:
Human-Object Interaction via Automatically Designed VLM-Guided Motion Policy
Paper | Project Page | InterPlay Dataset
We present a physics-based framework for long-horizon human-object interaction (HOI) synthesis. The framework uses vision-language models (VLMs) to convert scene observations and natural-language task instructions into structured Relative Movement Dynamics (RMD) plans. These plans are then used to automatically construct goal states and reward functions for reinforcement learning, enabling simulated humanoid characters to interact with static, dynamic, and articulated objects.
The InterPlay dataset is now available on Hugging Face:
https://huggingface.co/datasets/Marshalcy7/Interplay
The dataset provides structured indoor HOI scenes for layout-conditioned interaction reasoning, vision-language grounding, and scene-level task planning. The current release is packaged as interplay.zip.
Each data item is indexed by a shared scene id, such as dynamic_000, static_012, or hybrid_034. The same id aligns the scene layout, top-view rendering, and text instruction:
layout/dynamic_000.yaml
image/dynamic_000.png
text_instruction.json["dynamic_000"]
The release contains:
3d_assets/ # Processed 3D assets with mesh, URDF, and semantic config
layout/ # Multi-object scene layout files
image/ # Top-view rendered scene images
text_instruction.json # Natural-language task instructions
The dataset includes three task families: static_* for static furniture interactions such as sitting or lying, dynamic_* for object rearrangement tasks such as moving boxes or machines, and hybrid_* for mixed tasks combining static interaction and dynamic object manipulation.
The InterPlay dataset is released for non-commercial research use under CC BY-NC-SA 4.0. The dataset-specific annotations, layouts, interaction plans, top-view images, text instructions, and simulation metadata are provided under this license. The included 3D meshes are processed assets derived from 3D-FUTURE / 3D-FRONT and remain subject to the original 3D-FUTURE terms of use. Users should comply with the original asset licenses and cite the corresponding sources when using the 3D assets.
If you use this work, please cite:
@inproceedings{deng2026humanobject,
title={Human-Object Interaction via Automatically Designed {VLM}-Guided Motion Policy},
author={Zekai Deng and Ye Shi and Kaiyang Ji and Lan Xu and Shaoli Huang and Jingya Wang},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026}
}