ACM Transactions on Graphics (TOG), Preseneted at SIGGRAPH Asia 2024

Heming Zhu · Fangneng Zhan · Christian Theobalt · Marc Habermann‡

Please find the Raw Data and Annotations, the Metadata, and the Checkpoints that is used in the paper from this link.

The dataset contains long, dense multi-view video sequences performing everyday motions. The training sequences are ~20 minutes long (25fps), and the testing sequences are ~10 minutes long (25fps). Each sequnce is equipped with Foreground Segementation, and Per-frame Groundtruth 3D reconstruction generated by NeuS2. The raw data follows the folder structure below

|--- Subject0001
    |---training                            # training poses
    |   |--- images 
    |   |       |--- stream***.mp4          # the raw images
    |   |--- foregroundSegmentation 
    |   |       |--- stream***.mp4          # the foreground segementations             
    |   |--- recon_neus2 
    |           |--- *.ply                  # the Neus2 reconstruction for each frame
    |---testing
        |---...                             

The raw data follows the folder structure below:

meta_data
|--- Subject0001
    |---skeletoolToGTPose                   # training poses
    |   |--- ... 
    |
    |---skeletoolToGTPoseTest               # Testing poses
    |   |--- ...
    | 
    |---cameras.calibration                 # 1K camera calibration
    |---ddc.character                       # character setting 
    |---ddc.obj                             # template mesh
    |---ddc_graph.obj                       # embedded graph
    |---ddc.skeleton                        # template skeleton
    |---ddc.png                             # template texture
    |---base_trans.pkl                      # translation for normalization
    |---dof_convert_dict.pkl                # dof with/without hands
    |---segmentation.txt                    # template segementation (hands / others) 
    |---segmentation_hand.txt               # template segementation (hands / others) 
    |---subdivision_dict_xyz_new.pkl        # subdivision information 

git clone [email protected]:kv2000/TriHuman.git --recursive

The code is tested on Python 3.9.21, pytorch 2.10, and cuda 12.1.

numpy==1.23.1 
pytorch3d==0.7.8
kaolin==0.17.0
kornia==0.7.4

Firstly, install the underlying clothed human body model,:fireworks: DeepCharacters Pytorch🎆, which also consists the dependencies that needed for this repo.

We may also need to install the cuda renderer for rendering the depth map for the template mesh, which allows for single mesh x multiple views simultaneously. Rendering the meshes(shaded/not shaded), normal maps, position maps, foreground masks, etc.

cd cuda_renderer
# -> may change the arch list according to the GPU that you may use
TORCH_CUDA_ARCH_LIST="7.5 8.0 8.6" python setup.py install 
cd ..

If you skip the Step 1, then you still need to install the following to install the skeleton related :D:

cd cuda_skeleton
# -> may change the arch list according to the GPU that you may use
TORCH_CUDA_ARCH_LIST="7.5 8.0 8.6" python setup.py install 
cd ..

Please download the dataset from this link, and extract the metadata and checkpoints to the ./trihuman_metadata/ folder. A sample structure is shown below:

./trihuman_metadata/
    |---Subject0010_loose_metadata
           |--- ....
    |---Subject0010_tight_checkpoints
           |--- explicit_checkpoint.pth
           |--- implicit_checkpoint.pth

Here we drop a simple code snippet to run the inference. For geometry inference, simply run the following code, the captions for the configuration files are detailed in: ./Geometry_Inference/confs_converted/Subject0010_loose/Subject0010_loose_train.conf

cd Geometry_Inference
bash scripts/Subject0010_loose/Subject0010_loose_train.sh

The results will be saved in ./results/Subject0010_loose/train/meshes/[frame_id].ply.

For rendering inference, simply run:

cd Rendering_Inference
bash scripts/Subject0010_loose/Subject0010_loose_train.sh

The results will be saved in ./results/Subject0010_loose/validations_val/[camera_id]/[frame_id].png.

If you find our dataset and source code is useful for your research, please consider citing our paper:

@article{zhu2023trihuman,
  title={Trihuman: a real-time and controllable tri-plane representation for detailed human geometry and appearance synthesis},
  author={Zhu, Heming and Zhan, Fangneng and Theobalt, Christian and Habermann, Marc},
  journal={ACM Transactions on Graphics},
  volume={44},
  number={1},
  pages={1--17},
  year={2024},
  publisher={ACM New York, NY}
}

For questions, clarifications, feel free to get in touch with:
Heming Zhu: [email protected]
Marc Habermann: [email protected]

Deep Characters Pyotrch is under CC-BY-NC license. The license applies to the pre-trained models and the metadata as well.

Christian Theobalt was supported by ERC Consolidator Grant 4DReply (No.770784). This project was also supported by the Saarbrucken Research Center for Visual Computing, Interaction, and AI.

Below are some resources that we benefit from (keep updating):

• 3D gaussian-splatting, bravo for the brilliant representation for real-time and high-quality rendering.
• DeepCharacters Pytorch for the human character model.
• Pytorch3D, Kaolin, and Kornia for the handy geometry library.
• NeuS for the project structure D:.