Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding

ViPS improves spatial understanding in MLLMs by bringing together complementary visual priors from multiple expert models.

• Efficient Prior Proxy estimates priors from VGGT, DepthAnything3, TraceAnything, RADIO, and Wan2.1 with one shared backbone.
• Dynamic Prior Fusion selects and injects these priors according to the input question.

Code

✅ ViPS Training (Qwen3-VL-8B)
✅ VSI-Bench Evaluation

Checkpoints

✅ ViPS (Qwen3-VL-8B)
✅ Prior Proxy

Tested with Python 3.10, CUDA 12.8, and PyTorch 2.8.0.

# Run from the repository root.
conda create -n vips python=3.10 -y
conda activate vips
conda install -c conda-forge ffmpeg=8.0

python -m pip install --upgrade pip setuptools wheel packaging ninja
python -m pip install torch==2.8.0 torchvision==0.23.0 \
  --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r requirements.txt
python -m pip install --no-deps -e .
MAX_JOBS=4 python -m pip install flash-attn==2.8.3 --no-build-isolation

Training: We follow VLM-3R and use its VSI-Bench training data.

Download the original RGB recordings from the official sources:

• ScanNet
• ScanNet++
• ARKitScenes

Evaluation: use the official VSI-Bench annotations and videos with the bundled LMMS-Eval fork.

Run the preparation commands from the repository root in the vips environment.

export VIPS_DATA_ROOT=/path/to/datasets

hf download Journey9ni/VLM-3R-DATA \
  --repo-type dataset \
  --revision aae3535e513d36be5fd0a1edbaab523c8a3c45e4 \
  --include "vsibench_train/*.json" \
  --local-dir "$VIPS_DATA_ROOT/VLM-3R-DATA"

python scripts/prepare_training_annotations.py \
  --input-dir "$VIPS_DATA_ROOT/VLM-3R-DATA/vsibench_train" \
  --output-dir "$VIPS_DATA_ROOT/vips/annotations"

mkdir -p data/train
ln -s "$VIPS_DATA_ROOT/vips/annotations" data/train/annotations

{
  "video": "<subset>/videos/<scene>.mp4",
  "conversations": [
    {"from": "human", "value": "<video>\n<question>"},
    {"from": "gpt", "value": "<answer>"}
  ]
}

$VIPS_DATA_ROOT/raw/
├── scannet/scans/<scene>/<scene>.sens
├── scannetpp/data/<scene>/iphone/rgb.mkv
└── arkitscenes/raw/{Training,Validation}/<scene>/<scene>.mov

python scripts/prepare_training_videos.py \
  --raw-root "$VIPS_DATA_ROOT/raw" \
  --output-dir "$VIPS_DATA_ROOT/vips/videos"

ln -s "$VIPS_DATA_ROOT/vips/videos" data/train/videos

export VIPS_DATA_ROOT=/path/to/datasets

hf download nyu-visionx/VSI-Bench \
  test.jsonl arkitscenes.zip scannet.zip scannetpp.zip \
  --repo-type dataset \
  --revision d7cb1a3960b79dd3e20d4990b83005e96e1bcd9d \
  --local-dir "$VIPS_DATA_ROOT/VSI-Bench"

mkdir -p "$VIPS_DATA_ROOT/VSI-Bench/videos" data/vsibench
for subset in arkitscenes scannet scannetpp; do
  unzip -n "$VIPS_DATA_ROOT/VSI-Bench/${subset}.zip" \
    -d "$VIPS_DATA_ROOT/VSI-Bench/videos"
done

ln -s "$VIPS_DATA_ROOT/VSI-Bench/test.jsonl" data/vsibench/test.jsonl
ln -s "$VIPS_DATA_ROOT/VSI-Bench/videos" data/vsibench/videos

hf download leeiioolin/ViPS --local-dir checkpoints
hf download Qwen/Qwen3-VL-8B-Instruct --local-dir models/Qwen3-VL-8B-Instruct

vips/
├── models/
│   └── Qwen3-VL-8B-Instruct/
├── checkpoints/
│   ├── vggt_multihead.pt
│   └── vips/
└── data/
    ├── train/
    │   ├── annotations/          # JSONL instruction data
    │   ├── videos/               # 32 pre-extracted frames per scene
    │   └── features/             # Generated by VGGT-MultiHead
    └── vsibench/
        ├── test.jsonl
        ├── videos/               # <dataset>/<scene_name>.mp4
        ├── features/             # Generated by VGGT-MultiHead
        └── cache/

The ViPS training and evaluation settings are defined in configs/vips.yaml.

ViPS prior fusion is implemented in modeling_qwen3_vl.py:

Use our proxy to extract prior features.

Generate training features:

bash scripts/extract_features.sh configs/vips.yaml train

Generate VSI-Bench evaluation features:

bash scripts/extract_features.sh configs/vips.yaml vsibench

Prior extraction uses one GPU by default. To shard the videos across multiple GPUs:

VIPS_NUM_GPUS=8 bash scripts/extract_features.sh configs/vips.yaml vsibench

bash scripts/train.sh configs/vips.yaml --output-dir outputs/train/vips

The launcher uses eight GPUs by default. Set VIPS_NUM_GPUS to use a different number:

VIPS_NUM_GPUS=4 bash scripts/train.sh configs/vips.yaml --output-dir outputs/train/vips

Evaluate ViPS on VSI-Bench with LMMS-Eval:

# Single GPU (default).
bash scripts/eval.sh configs/vips.yaml

# Eight GPUs.
VIPS_NUM_GPUS=8 bash scripts/eval.sh configs/vips.yaml

# Custom checkpoint.
bash scripts/eval.sh configs/vips.yaml --checkpoint /path/to/checkpoint

Results are saved to outputs/eval/vips/.

If you find ViPS useful in your research, please cite our paper:

@article{lin2026beyond,
  title={Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding},
  author={Lin, Xiao and Huang, Xiaohu and Han, Kai},
  journal={arXiv preprint arXiv:2607.15054},
  year={2026}
}

This project is built upon Qwen3-VL, VGGT, Hugging Face Transformers, and LMMS-Eval. We also thank the authors of DepthAnything3, TraceAnything, RADIO, Wan2.1, and VSI-Bench for making their work publicly available.

Original ViPS code is released under the MIT License; third-party code, model weights, and datasets retain their respective licenses.