Human mesh recovery offers substantial potential for detailed behavior analysis and understanding of complex human-environment interactions. In this paper, we propose a novel 4D Human Motion Reconstruction Virtual Reality System that integrates advanced 4D multi-view human mesh recovery and high-quality 3D environment reconstruction using 3D Gaussian Splatting (3DGS). Our system seamlessly combines detailed 4D human behavior capture with accurate 3D environment reconstruction, significantly extending traditional visual monitoring approaches. Visualization through an interactive Virtual Reality (VR) platform enables dynamic interaction representation using accurately reconstructed virtual environments and computer-generated (CG) avatars. Experimental results from realistic scenarios validate the effectiveness of our framework in providing immersive experiences and precise human-environment modeling, demonstrating a significant advancement in a practical human-centered representation approach. Our approach consistently outperforms existing state-of-the-art methods, achieving reductions in mesh errors of 24% in PVE and 32% in MPJPE on the CHI3D dataset, and 17% in MPJPE and 64% in translation error on the Hi4D dataset compared to other multi-view methods.

1. Install conda python virtual environment. (modified from https://github.com/naver/multi-hmr)

   conda env create -f conda.yaml
   conda activate multihmr
   

2. Install ffmpeg in the env created above.

   conda activate multihmr
   conda install conda-forge::ffmpeg
   

Please see Dataset Setup page for details.

Multi-HMR Pre-trained Checkpoints

Please download from MultiHMR. Place all files under checkpoints/multiHMR.

• multiHMR_896_L HuggingFace model (support zero-shot)
• multiHMR_672_L
• multiHMR_672_S (support zero-shot)

Our Multi-HMR Checkpoints (finetuned on CHI3D/Hi4D)

Please download our checkpoints from Google Drive. Place all files under checkpoints/saved.

SMPL

Please download the following files from SMPL.

• SMPL_NEUTRAL.npz: place under models/smpl.

SMPL-X

Please download the following files from SMPL-X.

• SMPLX_NEUTRAL.npz: place under models/smplx.

Please download from Google Drive.

• J_regressor_h36m.npy: place under models.
• smpl_mean_params.npz: place under models.
• smplx2smpl.pkl: place under models/smplx.

CHI3D

Please replace <YOUR_CHI3D_DATASET_PATH> to your downloaded CHI3D data path.

python train.py --exp_name model_large_672_chi3d_sv \
    --dataset chi3d \
    --dataset_dir <YOUR_CHI3D_DATASET_PATH> \
    --pretrained_path models/multiHMR/multiHMR_672_L.pt \
    --smplx_dir models \
    --smplx2smpl_path models/smplx/smplx2smpl.pkl \
    --j_regressor_h36m_path models/J_regressor_h36m.npy \
    --image_size 672 \
    --train_batchsize 8 \
    --epochs 5 \
    --backbone dinov2_vitl14 \
    --smplx_type smplx \
    --save_dir checkpoints/saved \
    --device 0

Hi4D (working in progress)

Please replace <YOUR_HI4D_DATASET_PATH> to your downloaded Hi4D data path.

python train.py --exp_name model_large_672_hi4d_sv \
    --dataset hi4d \
    --dataset_dir <YOUR_HI4D_DATASET_PATH> \
    --pretrained_path models/multiHMR/multiHMR_672_L.pt \
    --smplx_dir models \
    --smplx2smpl_path models/smplx/smplx2smpl.pkl \
    --j_regressor_h36m_path models/J_regressor_h36m.npy \
    --image_size 672 \
    --train_batchsize 8 \
    --epochs 5 \
    --backbone dinov2_vitl14 \
    --smplx_type smpl \
    --save_dir checkpoints/saved \
    --device 0

CHI3D

Please replace <YOUR_CHI3D_DATASET_PATH> to your downloaded CHI3D data path.

python test.py --exp_name model_large_672_chi3d_sv \
    --dataset chi3d \
    --dataset_dir <YOUR_CHI3D_DATASET_PATH \
    --pretrained_path models/multiHMR/multiHMR_672_L.pt \
    --smplx_dir models \
    --smplx2smpl_path models/smplx/smplx2smpl.pkl \
    --j_regressor_h36m_path models/J_regressor_h36m.npy \
    --image_size 672 \
    --backbone dinov2_vitl14 \
    --smplx_type smplx \
    --data_smplx_type smplx \
    --eval_mode finetune \
    --save_dir ./checkpoints/saved \
    --recenter \
    --device 0

Hi4D (working in progress)

Please replace <YOUR_HI4D_DATASET_PATH> to your downloaded Hi4D data path.

python test.py --exp_name model_large_672_hi4d_sv \
    --dataset hi4d \
    --dataset_dir <YOUR_HI4D_DATASET_PATH> \
    --pretrained_path models/multiHMR/multiHMR_672_L.pt \
    --smplx_dir models \
    --smplx2smpl_path models/smplx/smplx2smpl.pkl \
    --j_regressor_h36m_path models/smplx/smplx2smpl.pkl \
    --image_size 672 \
    --backbone dinov2_vitl14 \
    --smplx_type smpl \
    --data_smplx_type smpl \
    --eval_mode finetune \
    --save_dir checkpoints/saved \
    --device 0

Please see Quick Start page for details.

Please see Dataset Setup page for details.

Please see Visualization page for details.

CHI3D-Zeroshot

Hi4D-Zeroshot

CHI3D-finetuned

Hi4D-finetuned

5 views on Tesla V100

MultiHMR-Small-672

• Inference time: 61.12081336975098 ms
• FPS: 16.361038815869318

MultiHMR-Large-672

• Inference time: 244.543958902359 ms
• FPS: 4.08924434072517

• CHI3D single-view fine-tuning support
• CHI3D evaluation support
• CHI3D demo support
• Hi4D single-view fine-tuning support
• Hi4D evaluation support
• Hi4D demo support
• Panoptic demo support
• Shelf demo support
• Blender support
• Documentation

[1] MultiHMR: https://github.com/naver/multi-hmr