This is the official PyTorch implementation of the following publication:
EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation
Runsong Zhu, Jiaxin Guo, Xiaoyang Guo†, Zhengzhe Liu†, Ka-Hei Hui, Wei Yin, Kai Chen, Wei Chen, Weiqiang Ren, Yunhui Liu, Pheng-Ann Heng, Chi-Wing Fu.
ICML 2026
- Python 3.10+
- PyTorch 2.3.1 + CUDA 11.8
- Required checkpoints in
../checkpoints/EPS3D/ - CLIP checkpoint:
../checkpoints/ViT-B-32.pt - Test data:
../data/scannet_test/
Install dependencies:
pip install -r requirements.txtThe following model weights and data need to be downloaded and placed following the above directory structure:
| File | Link |
|---|---|
| EPS3D checkpoint | Download |
| CLIP weight (ViT-B-32.pt) | Download |
| ScanNet test data | Download |
Note: The provided checkpoint differs slightly from the original implementation details in paper: we freeze the appearance gaussian head weights from AnySplat and retrain only the perception-related modules, which achieves comparable performance on ScanNet with lower memory cost.
EPS3D/
├── code_eps3d/
│ ├── src/ # Core model code
│ │ └── model/
│ │ ├── model/
│ │ │ ├── eps3d.py # EPS3D base model
│ │ │ └── eps3d_panoptic.py # EPS3D panoptic model
│ │ ├── encoder/ # VGGT encoder + Gaussian adapter
│ │ └── decoder/ # CUDA splatting decoder
│ ├── scripts/
│ │ ├── run_eps3d_scannet.sh # ScanNet 8-view evaluation
│ │ ├── run_eps3d_scannet_2view.sh # ScanNet 2-view evaluation
│ │ ├── test_eps3d_panoptic.py # Main evaluation script
│ │ ├── evaluate_pq.py # PQ metric evaluation
│ │ └── data_utils/ # Data loading utilities
│ ├── submodules/ # Dependencies (dust3r, VGGT, etc.)
│ ├── config/ # Model configurations
│ └── lseg.py # LSeg feature extractor
├── data/
│ └── scannet_test/ # ScanNet test scenes
├── checkpoints/
│ ├── EPS3D/
│ │ ├── model.safetensors # EPS3D model weights
│ │ ├── config.json # EPS3D model config
│ │ └── demo_e200.ckpt # LSeg model
│ └── ViT-B-32.pt # CLIP weights
# 1. Create directories
mkdir -p checkpoints/ data/
# 2. Place the downloaded EPS3D checkpoint
mv path/to/EPS3D checkpoints/
# 3. Place the CLIP ViT-B-32 weights
mv path/to/ViT-B-32.pt checkpoints/
# 4. Download the LSeg demo model weights
gdown 1FTuHY1xPUkM-5gaDtMfgCl3D0gR89WV7 -O checkpoints/demo_e200.ckpt
# 5. Place the ScanNet test data
mv path/to/scannet_test data/cd scripts
# Novel-view (default)
bash run_eps3d_scannet.sh
# Context-view
bash run_eps3d_scannet.sh contextcd scripts
# Novel-view (default)
bash run_eps3d_scannet_2view.sh
# Context-view
bash run_eps3d_scannet_2view.sh context@misc{zhu2026eps3dendtoendfeedforward3d,
title={EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation},
author={Runsong Zhu and Jiaxin Guo and Xiaoyang Guo and Zhengzhe Liu and Ka-Hei Hui and Wei Yin and Kai Chen and Wei Chen and Weiqiang Ren and Yunhui Liu and Pheng-Ann Heng and Chi-Wing Fu},
year={2026},
eprint={2606.08980},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2606.08980},
}
This work is built on many great research works and open-source projects, thanks a lot to all the authors for sharing!
