🌐 MTPano: Multi-Task Panoramic Scene Understanding via Label-Free Integration of Dense Prediction Priors

MTPano is a multi-task foundation model for panoramic dense scene parsing (Semantic Segmentation, Depth Estimation, and Surface Normal Estimation).

• [2026/03] 🚀 We release the inference code and pretrained weights on Hugging Face! You can now run local inference.
• [2026/07] 🎓 The full training code and the label-free data generation pipeline are now released! See Training and Data Preparation.

This repository contains the official implementation of MTPano:

MTPano: Multi-Task Panoramic Scene Understanding via Label-Free Integration of Dense Prediction Priors
Jingdong Zhang, Xiaohang Zhan, Lingzhi Zhang, Yizhou Wang,
Zhengming Yu, Jionghao Wang, Wenping Wang, Xin Li

MTPano is a robust multi-task panoramic foundation model established by a label-free training pipeline. It addresses the critical challenges of geometric distortions and data scarcity in 360° vision:

• Label-Free Training Pipeline: We circumvent data scarcity by projecting panoramas into perspective patches, generating pseudo-labels using off-the-shelf perspective foundation models, and re-projecting them for patch-wise supervision.
• PD-BridgeNet: We propose the Panoramic Dual BridgeNet to tackle the interference between task types. It explicitly disentangles rotation-invariant (e.g., depth, segmentation) and rotation-variant (e.g., surface normals) features via geometry-aware modulation.

MTPano achieves state-of-the-art performance on multiple benchmarks including Structured3D and Stanford2D3D.

The overview of our proposed MTPano framework and PD-BridgeNet architecture.

The codebase is built with torch==2.5.0 and torchvision==0.20.0. We provide a convenient shell script to configure the environment in one click.

# Clone the repository
git clone https://github.com/Evergreen0929/MTPano.git
cd MTPano

# Run the setup script to install all dependencies
pip install torch==2.5.0 torchvision==0.20.0  
bash setup_env.sh

We host our pretrained model weights on Hugging Face. Currently, we provide two versions of weights: 140k and 408k.

Our inference script integrates huggingface_hub, which means you don't need to manually download them. The script will automatically fetch and cache the requested weights directly from jdzhang0929/MTPano during your first run.

You can easily run MTPano on a single panoramic image or a folder of images. The script will automatically generate Semantic Segmentation, Depth Maps, and Surface Normals.

python inference.py --input ./examples --output ./results

If you want to render a Flythrough Video, simply add the corresponding flags:

python inference.py \
    --input ./examples \
    --output ./results \
    --weight 408k \
    --save_video 

• --weight: Choose between 140k or 408k (defaults to 408k).
• --save_video: Renders a 15-second cinematic flythrough video showing the multi-task predictions.

MTPano uses two kinds of data: the label-free PanoMTDU pseudo-label set (for pre-training) and standard benchmark datasets (Structured3D, Stanford2D3D, Matterport3D, Deep360, SynPASS — for benchmark post-training and evaluation).

All download, format-conversion, and pseudo-label generation scripts live in preprocess_dataset/, with a detailed per-script guide in preprocess_dataset/README.md. In short:

• Benchmarks are downloaded and converted to a unified equirectangular (rgb / semantic / depth / normal, 1024×512) format; the train/val/test split JSONs are already shipped under data/db_info/.
• PanoMTDU is built label-free by projecting unlabeled panoramas into perspective patches, generating dense labels with off-the-shelf foundation models (MoGe for depth & normals, InternImage/Mask2Former for semantics), and re-projecting them back to the sphere (see preprocess_dataset/unsup/).

Point the code at your data root via an environment variable (defaults live in configs/mypath.py):

export MTPANO_DATA_ROOT=/path/to/your/data

Training is DDP-based and config-driven; each dataset has a launch script. All runs use the train_gt mode at the panoramic (512×1024) scale.

The most common use case is to (post-)train MTPano on a benchmark, initialized from a released MTPano weight via --trained_model — either a local checkpoint path, or the 140k / 408k tag to auto-fetch from Hugging Face:

bash run_structured3d_gt.sh     # Structured3D
bash run_stanford2d3d_gt.sh     # Stanford2D3D
bash run_matterport3d_gt.sh     # Matterport3D
bash run_deep360_gt.sh          # Deep360   (depth only)
bash run_synpass_gt.sh          # SynPASS   (semseg only)

Each script wraps a distributed launch:

python -m torch.distributed.launch --nproc_per_node=<N> --use_env main.py \
    --config_exp ./configs/<dataset>/<dataset>_dinov3L_dpt_gt.yml \
    --run_mode train --dataset_log <Name> \
    --trained_model 140k        # local checkpoint path, or 140k / 408k (auto-fetched from HF)

All hyperparameters (optimizer, max_iter, batch size, loss weights, active tasks) are defined in the corresponding configs/<dataset>/*_gt.yml. Evaluation runs automatically every val_interval iterations, reporting semantic mIoU, depth RMSE, and normal mean angular error.

To evaluate a trained checkpoint standalone:

python test.py \
    --config_exp ./configs/stanford2d3d/stanford2d3d_dinov3L_dpt_gt.yml \
    --trained_model /path/to/checkpoint.pth.tar

The released 140k / 408k weights are themselves produced by our label-free pre-training on the PanoMTDU pseudo-label dataset. If you want to reproduce MTPano from scratch, first build PanoMTDU by following the Data Preparation guide (preprocess_dataset/README.md), then launch:

bash run_panomtdu_gt.sh

If you find our work or this repository useful, please consider giving us a ⭐ star and citing our paper:

@article{zhang2026mtpano,
  title={MTPano: Multi-Task Panoramic Scene Understanding via Label-Free Integration of Dense Prediction Priors},
  author={Zhang, Jingdong and Zhan, Xiaohang and Zhang, Lingzhi and Wang, Yizhou and Yu, Zhengming and Wang, Jionghao and Wang, Wenping and Li, Xin},
  journal={arXiv preprint},
  year={2026}
}

If you have any questions, please feel free to reach out to Jingdong Zhang.

This project heavily relies on the excellent works from DINOv3, MoGe, and BridgeNet. We express our sincere gratitude to the authors for open-sourcing their code and models.

BridgeNet: This project proposes BridgeNet for Multi-task Dense Predictions, leveraging Bridge-Feature as intermediate representations.
HiTTs: This project targeting Partially-Supervisioned Multi-Task Dense Predictions with Hierarchical-Task-Tokens (HiTTs).