Mu Huang1,2, Hui Wang3,2, Kerui Ren3,2, Linning Xu4,2, Yunsong Zhou2, Mulin Yu2, Bo Dai5,†, Jiangmiao Pang2
1Fudan University 2Shanghai Artificial Intelligence Laboratory 3Shanghai Jiao Tong University 4The Chinese University of Hong Kong 5The University of Hong Kong
†Corresponding author.
SoMA is a Gaussian splat neural simulator that models deformable object dynamics from real-world robot manipulation, enabling action-conditioned, stable long-horizon simulation with high-fidelity, multi-view-consistent rendering.
Clone SoMA
git clone https://github.com/Wrioste/SoMA.git
cd SoMACreate Environment (cuda11.8 & torch2.0.0)
conda create -n soma python=3.10
conda activate soma
conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 mkl==2023.1.0 -c pytorch -c nvidia
pip install mmcv-full==1.7.2 -f https://download.openmmlab.com/mmcv/dist/cu118/torch2.0.0/index.html
pip install -r requirements.txt
pip install -v -e .Install DGL:
pip install dgl==2.1.0 -f https://data.dgl.ai/wheels/cu118/repo.html
pip install --no-deps torchdata==0.6.1
pip install dglgo==0.0.2Install Gaussian Splatting dependencies:
git clone https://github.com/graphdeco-inria/gaussian-splatting.git --recursive
git submodule update --init --recursive
pip install --no-build-isolation gaussian-splatting/submodules/diff-gaussian-rasterization/
pip install --no-build-isolation gaussian-splatting/submodules/simple-knn/Install PyTorch3D dependencies:
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install -c bottler nvidiacub -y
conda install pytorch3d -c pytorch3d -yWe provide a sample cloth_lift_1 scene for validation. The code can also be adapted to PhyTwin, DROID, or other soft-body manipulation datasets after preprocessing.
-
Download the sample data.
-
tar -xzf soma_data_sample.tar.gz
Data preprocessing code is available under data_preprocess/; see data_preprocess/README.md for setup and usage.
Stage 1 trains the coarse Gaussian dynamics model with a temporal stride of k * dt. It is also used to generate the pred_stage1/ cache.
python tools/train.py \
configs/SoMA/cloth_lift_stage1.py \
--work_dir work_dirs/cloth_lift_stage1Generate the stage-1 cache, which stores deformed Gaussian splats at frames k, 2k, ..., nk.
python tools/test.py \
configs/SoMA/cloth_lift_stage1.py \
checkpoints/cloth_lift_stage1/epoch_15.pth \
--show-dir outputs/cloth_lift_stage1_cache \
--gpu-id 0Stage 2 trains the local dynamics model at the frame-level timestep dt from the stage-1 cache.
python tools/train.py \
configs/SoMA/cloth_lift_stage2.py \
--work_dir work_dirs/cloth_lift_stage2 \
--resume-from work_dirs/cloth_lift_stage1/epoch_15.pthRun a continuous stage-2 rollout from the initial Gaussian state with online cache updates at frame_gap boundaries. Use --test-rollout-mode segmented to evaluate with cached segment starts.
python tools/test.py \
configs/SoMA/cloth_lift_stage2.py \
checkpoints/cloth_lift_stage2/epoch_25.pth \
--show-dir outputs/cloth_lift_stage2_continuous \
--gpu-id 0 \
--test-rollout-mode continuousIf you find our work helpful, please consider citing:
@article{huang2026soma,
title={SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation},
author={Huang, Mu and Wang, Hui and Ren, Kerui and Xu, Linning and Zhou, Yunsong and Yu, Mulin and Dai, Bo and Pang, Jiangmiao},
journal={arXiv preprint arXiv:2602.02402},
year={2026}
}We will update the citation once the ICML 2026 proceedings version is available.
