ViT-H: Generative Video-To-Humanoid Control with Motion Tracking Feedback

This is the official implementation of ViT-H: Generative Video-To-Humanoid Control with Motion Tracking Feedback.

• Release inference code for offline video-to-humanoid control by November 1, 2026.
• Release pre-training code for offline video-to-humanoid control by November 1, 2026.
• Release deployment code for humanoid video teleoperation by December 1, 2026.
• Release pre-training code for humanoid video teleoperation by December 1, 2026.
• Release post-training code by December 1, 2026.

The offline release targets Linux, Python 3.10, CUDA 12.1, and PyTorch 2.3.

conda create -n vith python=3.10 -y
conda activate vith
pip install -r requirements.txt
pip install -e .

SAM3D preprocessing and inference require Fast-SAM-3D-Body. Clone it into third-party/Fast-SAM-3D-Body and follow its environment setup guide:

mkdir -p third-party
git clone https://github.com/yangtiming/Fast-SAM-3D-Body.git \
  third-party/Fast-SAM-3D-Body
bash third-party/Fast-SAM-3D-Body/setup_env.sh
conda activate fast_sam_3d_body

The SAM3D weights are distributed by the SAM3D project rather than in the ViT-H checkpoint repository.

DPVO camera estimation is optional. To enable --use-dpvo, install DPVO under third-party/DPVO:

git clone --recursive https://github.com/princeton-vl/DPVO.git \
  third-party/DPVO
wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip \
  -O /tmp/eigen-3.4.0.zip
unzip /tmp/eigen-3.4.0.zip -d third-party/DPVO/thirdparty
pip install ./third-party/DPVO

Create the checkpoint root:

mkdir -p inputs/checkpoints

Download SMPL and SMPL-X from their official websites after registration: SMPL and SMPL-X. Place the files as follows:

inputs/checkpoints/
└── body_models/
    ├── smpl/
    │   └── SMPL_{GENDER}.pkl
    └── smplx/
        └── SMPLX_{GENDER}.npz

The ViT-H checkpoint repository contains the released offline model, YOLO, ViTPose, HMR2, the optional DPVO weights, and the G1 tracker policy. Download the inference files into the project root:

hf download eerrr1/ViT-H-checkpoints \
  vith_offline_g1.ckpt \
  inputs/checkpoints/dpvo/dpvo.pth \
  inputs/checkpoints/hmr2/epoch=10-step=25000.ckpt \
  inputs/checkpoints/vitpose/vitpose-h-multi-coco.pth \
  inputs/checkpoints/yolo/yolov8x.pt \
  --repo-type dataset \
  --local-dir .

For G1 policy tracking, download the ONNX policy separately so that it stays in the ignored checkpoint directory:

hf download eerrr1/ViT-H-checkpoints \
  model_step_020000_g1.onnx \
  --repo-type dataset \
  --local-dir inputs/checkpoints/g1_tracker

The downloaded files follow this layout:

vith_offline_g1.ckpt
inputs/checkpoints/
├── dpvo/
│   └── dpvo.pth
├── hmr2/
│   └── epoch=10-step=25000.ckpt
├── vitpose/
│   └── vitpose-h-multi-coco.pth
└── yolo/
    └── yolov8x.pt

The released configuration uses SAM3D image features. The HMR2 checkpoint is included as an optional compatible image encoder. The SAM3D network and MHR files are obtained through Fast-SAM-3D-Body and stay inside third-party/Fast-SAM-3D-Body/checkpoints/.

We provide preprocessed data for training and evaluation. Due to licensing restrictions, we do not redistribute the original datasets. Please obtain the original data, including annotations and videos, directly from the respective dataset websites. By downloading our preprocessed data, you agree to comply with the terms of use of the corresponding datasets and to use the data for research purposes only.

The released full experiment uses AMASS, BEDLAM, BEDLAM2, Human3.6M, and BONES-SEED. Preprocessed data for the first four sources is distributed as four compressed archives: amass, bedlam, bedlam2, and h36m. Download the ViT-H training dataset and extract all archives into one directory.

hf download eerrr1/ViT-H \
  --repo-type dataset \
  --local-dir /path/to/training_set

cd /path/to/training_set
for archive in amass bedlam bedlam2 h36m; do
  tar --use-compress-program=unzstd -xf "${archive}.tar.zst"
done

export VITH_DATASET_ROOT=/path/to/training_set

The extracted layout is documented in docs/DATA_LAYOUT.md. Dataset components and body-model assets may have terms separate from this repository's code license.

BONES-SEED is available separately from its official Hugging Face repository. Accept its gated-data license on Hugging Face, then download the proportional SOMA motions and the matching G1 trajectories:

hf auth login
hf download bones-studio/seed \
  g1.tar.gz \
  soma_proportional.tar.gz \
  --repo-type dataset \
  --local-dir /path/to/bones-seed

cd /path/to/bones-seed
tar -xzf g1.tar.gz
tar -xzf soma_proportional.tar.gz

Convert the official 120 FPS files with the released BONES-SEED preprocessing pipeline. It extracts the 28 proportional SOMA joints, converts the provided G1 CSV trajectories through SONIC MotionLib/FK, applies the same motion-selection rules to both modalities, pairs them at 30 FPS, and writes session shards plus the training index. The G1 CSV files are already retargeted by BONES-SEED; this pipeline does not retarget the human BVH motions to G1.

cd /path/to/ViT-H-release
conda create -n bones-seed-preprocess python=3.10 -y
conda activate bones-seed-preprocess
pip install -r tools/bones_seed_preprocessing/requirements.txt

bash tools/bones_seed_preprocessing/run_preprocessing.sh \
  /path/to/bones-seed \
  /path/to/training_set/bones_seed \
  16

The training files are written to /path/to/training_set/bones_seed/paired/. The full conversion is CPU- and storage-intensive and must use a fresh output directory. See tools/bones_seed_preprocessing/README.md for the intermediate layout, output schema, filtering rules, and provenance.

The released experiment uses a 16-layer transformer with latent dimension 1024, 16 attention heads, 120-frame windows, and mixed-source batches of 32.

export VITH_DATASET_ROOT=/path/to/training_set
export NUM_DEVICES=8
export WANDB_MODE=offline
bash scripts/train.sh

The default training configuration uses all five sources and therefore checks for the converted BONES-SEED directory. The complete source and converted layouts are documented in docs/DATA_LAYOUT.md.

For Slurm clusters, adjust the site-specific directives in slurm/train.sbatch, export VITH_DATASET_ROOT, and submit:

sbatch slurm/train.sbatch

Model checkpoints are not stored in this repository. Pass the checkpoint path explicitly. The released model was trained with SAM3D image features, so inference requires both the SAM3D network checkpoint and its MHR body model:

python tools/infer_video.py \
  --video /path/to/input.mp4 \
  --ckpt /path/to/vith_offline_g1.ckpt \
  --sam3d-ckpt-path third-party/Fast-SAM-3D-Body/checkpoints/sam-3d-body-dinov3/model.ckpt \
  --sam3d-mhr-path third-party/Fast-SAM-3D-Body/checkpoints/sam-3d-body-dinov3/assets/mhr_model.pt \
  --sam3d-root third-party/Fast-SAM-3D-Body \
  --output outputs/inference/input_g1.npz

For the documented vith_offline_g1.ckpt, the command selects SAM3D automatically and refuses a conflicting encoder by default. Input videos are timestamp-resampled to 30 FPS while preserving their duration. Preprocessing caches are validated by hashes of the source video, effective settings, and model assets; use --force-recompute to rebuild all managed artifacts or --force-recompute-imgfeat to rebuild only image features. A user-provided preprocess/camera_K.pt is validated and is never deleted by either option. The selected SAM3D model_config.yaml and effective IMG_SIZE/LAYER_DTYPE are part of that identity. Concurrent jobs using the same video-stem cache directory are rejected by a process lock; use distinct --output-root and --output paths for simultaneous jobs. See cache behavior and regression tests for details.

Video preprocessing additionally requires the external detector, pose, body-model, and camera-estimation assets listed in docs/INFERENCE.md. Use --static-cam only for videos with a known static camera. Otherwise SimpleVO estimates camera motion by default; add --use-dpvo to use the installed DPVO package and dpvo.pth instead.

Generated G1 motions can optionally be tracked and evaluated in MuJoCo with the G1 motion tracker, whose entry point is python g1_tracker/tools/track_g1.py.

• hmr4d/: datasets, model, preprocessing, and shared utilities.
• motion_lib/: G1 kinematics and rotation utilities.
• scripts/train.sh: reproducible offline training entry point.
• tools/bones_seed_preprocessing/: BONES-SEED-to-training-data preprocessing.
• tools/infer_video.py: end-to-end video inference entry point.
• experiments/g1_training_set_exp1/: resolved experiment configuration and checksums.
• docs/: dataset and inference documentation.

If you find ViT-H helpful, please consider citing:

@article{he2026vith,
  title={ViT-H: Generative Video-To-Humanoid Control with Motion Tracking Feedback},
  author={He, Zimo and Bian, Tianyi and Yang, Zizhong and Li, Hongjie and Jiang, Nan and Cui, Jieming and Jia, Baoxiong and Wang, Yizhou and Huang, Siyuan and Zhu, Yixin},
  journal={arXiv preprint arXiv:2610.32966},
  year={2026}
}

We thank the authors of GVHMR, SAM 3D Body, Fast-SAM-3D-Body, YOLOv8, ViTPose, DPVO, HMR 2.0 / 4D-Humans, GMR, GEM, SONIC, HoloMotion, ScaleBFM, and Humanoid-GPT for their excellent prior work and publicly available code and models that supported this project.