Simulation-first benchmark for humanoid control

HumanoidArena:
Benchmarking Egocentric Hierarchical Whole-body Learning.

Taowen Wang1,* Zikang Xie1,* Bin Yang1,* Yunheng Wang1 Zizhao Yuan1 Yuetong Fang1 Yixiao Feng1 Yichi Wang2 Xingyu Chen1 Haodong Chen3 Qiwei Wu1 Weisheng Xu1 Lihan Chen4 Lusong Li5 Zecui Zeng5 Renjing Xu1

1 The Hong Kong University of Science and Technology (Guangzhou) 2 Beijing University of Technology 3 Harbin Institute of Technology, Shenzhen 4 Shenzhen MSU-BIT University 5 JD Explore Academy * Equal contribution

HumanoidArena is a simulation-first benchmark for egocentric hierarchical whole-body learning. It studies seven leg-critical HOI/HSI tasks where success depends on coordinated perception, foot placement, balance, posture adjustment, and whole-body motion, while exposing both perturbation-conditioned generalization and GMT-conditioned transfer.

Task suite
7 leg-critical HOI/HSI tasks
Evaluation axes
Visual, Semantic, Execution, Cross-GMT
Core stack
TWIST2, SONIC, IsaacLab, LeRobot

Overview figure

HumanoidArena overview figure

Abstract

Paper abstract.

Humanoid robots promise whole-body interaction in human-centered environments, but scalable policy learning remains difficult because task-level decision-making and whole-body dynamic execution are tightly coupled. A practical solution is hierarchical control, where a high-level policy predicts intermediate whole-body actions and low-level general motion trackers (GMTs) execute them as stable humanoid motion. However, existing benchmarks rarely evaluate the policy-tracker interface itself, leaving open whether intermediate whole-body actions are executable, robust under task distribution shifts, and transferable across different GMT backends. We introduce HumanoidArena, a simulation-first benchmark for egocentric hierarchical whole-body learning. The benchmark formulates policy learning as a hierarchical decision making problem: a high-level policy converts egocentric vision, proprioception, and instructions into a compact whole-body action, which is subsequently executed by a low-level GMT. Instead of treating the legs as planar transport tools, HumanoidArena emphasizes interactions where lower-body coordination is structurally necessary in task completion. We therefore design 7 leg-critical HOI/HSI tasks in which success requires foot placement, balance maintenance, posture adjustment, and whole-body reorientation. To further diagnose the hierarchical system, we evaluate policies from two complementary perspectives: perturbation-conditioned generalization and GMT-conditioned transfer. We benchmark representative imitation-learning and VLA-style policies under this shared interface. Experiments show that hierarchical control enables learned policies to solve diverse leg-critical interactions, but performance is strongly tracker-conditioned and cross-GMT transfer remains fragile. These results position HumanoidArena as a benchmark for studying transferable intermediate action representations and scalable egocentric whole-body policy learning.

Task gallery

Seven tasks across both GMTs.

Pipeline

From teleop to benchmark.

1. Shared capture and retargeting

The operator receives the humanoid's egocentric video stream through a PICO headset. Human motion is captured and retargeted through GMR into a shared 35D robot-space reference signal for real-time consumption.

2. Backend-specific action interpretation

Inside Isaac Lab, backend-specific action providers interpret that shared signal into executable G1 targets using either TWIST2 or SONIC, exposing different low-level tracking dynamics under one common upstream interface.

3. Recording and canonicalization

A recording manager serializes episodes into NPZ archives with egocentric observations, state, action, and replayable trajectory data. Demonstrations are normalized into a 64D canonical state and 40D intermediate whole-body action.

4. Conversion, training, evaluation

Raw recordings are converted into LeRobot-compatible datasets, then used to train high-level policies that are evaluated under in-GMT, cross-GMT, visual, semantic, and execution protocols.

SONIC inference additionally provides an experimental latent-output mode for VLA policies trained on SONIC encoder representations. In this mode, the policy outputs 64D SONIC latent actions instead of the default 40D semantic whole-body commands. The raw dataset also preserves SONIC latent64 representations for each demonstration episode.

Evaluation protocol

Four tests plus cross-GMT.

Example task

P&PBox makes the perturbation axes concrete.

Results

Success, failure, recovery.

* Clips are shown at the frame rate they were captured at; inference runs on the native video stream.

Leaderboard

How policy–tracker pairs compare.

Updated

All = mean SR across all seven tasks, (3 × HOI AVG + 4 × HSI AVG) / 7 AVG = suite average AFR = average fall rate (↓ is better) ± = standard deviation across seeds

Resources

Paper, code, and more.