• 🚀 EgoSim got accepted to ECCV 2026.

EgoSim is an egocentric world simulator for embodiment interaction generation. Given an initial 3D state and a sequence of actions, EgoSim generates temporally and spatially consistent egocentric observations with high-quality dexterous interactions. EgoSim also persistently updates a 3D scene state for continuous simulation.

Key features:

• Controllable egocentric video generation conditioned on 3D scene state and action sequences
• Updatable 3D memory for long-horizon continuous simulation
• Scalable data curation pipeline for scene-interaction pairs
• Few-shot generalization to in-the-wild real scenes and multiple embodiments

• Inference — run EgoSim-14B on Egodex and EgoVid datasets.
• Continuous simulation — multi-clip incremental generation with an updatable 3D scene state; see continuous_simulation/README.md.
• Data preparation — annotate raw egocentric videos to produce inference-ready assets; see data_process/README.md.
• Training — coming soon.

Requires Python 3.10+, CUDA 12.1+.

git clone https://github.com/jinkun-hao/EgoSim.git
cd EgoSim

conda create -n egosim python=3.10 -y
conda activate egosim

# Install PyTorch
pip install torch torchvision
# Install flash attention
pip install flash-attn --no-build-isolation
pip install -r requirements.txt

Download EgoSim-14B from HuggingFace:

huggingface-cli download wuzhi-hao/EgoSim --local-dir ./EgoSim-14B

Place the downloaded directory under the project root so the structure looks like:

EgoSim/
├── EgoSim-14B/
│   ├── diffusion_pytorch_model.safetensors
│   ├── Wan2.1_VAE.pth
│   ├── models_t5_umt5-xxl-enc-bf16.pth
│   ├── models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth
│   └── google/umt5-xxl/
├── egowm/
├── data_process/
└── ...

The VAE, T5, and CLIP weights are the same as Wan2.1-Fun-14B-InP. If you already have that model, you can symlink or copy those files.

Each sample requires three condition inputs alongside the source video, plus a text prompt:

All inputs are produced by the annotation pipeline in data_process/, which also generates the metadata.csv required by runner.py. See that README for environment setup, model checkpoints, and step-by-step instructions.

For quick testing, download the demo data from Google Drive and extract:

# Download mini_sample.zip and place it in the project root, then:
unzip demo_data.zip -d tests/samples/

All commands below assume you are inside the EgoSim repository root:

cd EgoSim

# Egodex — quick smoke test with bundled mini samples
PYTHONPATH=. python egowm/inference/runner.py \
  --dataset egodex \
  --model_root ./EgoSim-14B \
  --dataset_root tests/samples/demo_data/egodex \
  --metadata_path tests/samples/demo_data/egodex_metadata.csv \
  --output_dir output_egodex \
  --num_inference_steps 50 \
  --gpu_id 0

# EgoVid — quick smoke test with bundled mini samples
PYTHONPATH=. python egowm/inference/runner.py \
  --dataset egovid \
  --model_root ./EgoSim-14B \
  --dataset_root tests/samples/demo_data/egovid \
  --metadata_path tests/samples/demo_data/egovid_metadata.csv \
  --output_dir output_egovid \
  --num_inference_steps 50 \
  --gpu_id 0

Each sample produces two files in --output_dir:

• {id}.mp4 — generated video
• {id}_cmp.mp4 — side-by-side comparison: ego_prior | hand_keypoint | generated

For full datasets, replace dataset_root and metadata_path with your actual paths.

Key options:

Multi-clip incremental generation with updatable 3D scene state lives in continuous_simulation/. After installing the main project, follow that README to set up the scene environment and run recon_visualize or full modes.

This codebase is built upon the following open-source projects. We sincerely thank the authors for their contributions.

• Wan2.1
• DiffSynth Studio
• VACE

Scene-reconstruction dependencies for continuous_simulation/ are listed in that README.

@article{hao2026egosim,
  title={EgoSim: Egocentric World Simulator for Embodied Interaction Generation},
  author={Hao, Jinkun and Jia, Mingda and Wang, Ruiyan and Liu, Xihui and Yi, Ran and Ma, Lizhuang and Pang, Jiangmiao and Xu, Xudong},
  journal={arXiv preprint arXiv:2604.01001},
  year={2026}
}