SigLIP-HD

This codebase contains the official PyTorch implementation of SigLIP-HD:

SigLIP-HD by Fine-to-Coarse Supervision
Lihe Yang, Zhen Zhao, Hengshuang Zhao
International Conference on Learning Representations (ICLR), 2026

Note: This work was finished in mid-2025 and accepted by ICLR in early 2026. Sorry for the delay in open-sourcing.

This is a simple distillation framework for better performance: a frozen teacher runs on multi-scale images to produce high-quality features, and a student is optimized to match those features at a medium-scale image.

SigLIP-HD exhibits better performance than SigLIP 2 in MLLMs, especially in OCR scenarios.

Below is a brief comparison with the SigLIP 2-So400m/16-512px baseline. Refer to our paper for more details.

SigLIP-HD is fine-tuned from SigLIP 2-So400m/16-512px. It is a drop-in replacement: identical architecture and I/O to the original encoder. To use it in an MLLM, keep your existing SigLIP 2 pipeline and only change the vision-tower path to the SigLIP-HD checkpoint.

The model can be used from transformers:

import torch
from PIL import Image
from transformers import SiglipVisionModel, AutoImageProcessor

model = SiglipVisionModel.from_pretrained("LiheYoung/SigLIP-HD").eval()
processor = AutoImageProcessor.from_pretrained("LiheYoung/SigLIP-HD")

image = Image.open("example.jpg").convert("RGB")
inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    features = model(**inputs, output_hidden_states=True).hidden_states[-1]  # (1, 1024, 1152)

This is a vision encoder only (no text tower), intended for better visual representations in MLLMs.

pip install -r requirements.txt

We train on the 4.5M raw images from the Cambrian-1 collection. List all file paths in the file:

/path/to/cambrian/images/000001.jpg
/path/to/cambrian/images/000002.jpg
...

Single node:

# We use global batch size 512. Please modify the config correspondingly.
torchrun --nproc_per_node=8 siglip_hd/train/train.py \
    --config-file siglip_hd/configs/train/siglip_hd_from_siglip2_so400m.yaml \
    --output-dir exp/siglip_hd_from_siglip2_so400m

Multi-node with Slurm: see this script.

If you find this work useful, please consider citing:

@inproceedings{sigliphd,
  title={SigLIP-HD by Fine-to-Coarse Supervision},
  author={Yang, Lihe and Zhao, Zhen and Zhao, Hengshuang},
  booktitle={ICLR},
  year={2026}
}

We sincerely thank SigLIP 2 authors for open-sourcing their capable vision encoder and thank DINOv2 authors for open-sourcing their clean training code.

This project is released under the MIT license.