ALOE: Align Once to Explain

This is the public code repository for ALOE (ALign Once to Explain), our accepted CVPR 2026 poster "Align Once to Explain: Feature Alignment for Scalable B-cosification of Foundational Vision Transformers".

ALOE converts strong ViT-style foundation models into inherently interpretable B-cos visual backbones through one-time, label-free feature alignment. The aligned backbone can then be used as a drop-in encoder for downstream evaluation while producing model-inherent B-cos explanations.

Transform to B-cos — convert a foundation ViT encoder into a bias-free, dynamic-linear backbone. Align once — perform label-free feature alignment against the frozen teacher on unlabeled images. Deploy — freeze the aligned backbone for downstream transfer; explanations follow directly from the model's dynamic-linear summary.

• Label-free alignment from frozen supervised ViT, DINOv3, and SigLIP2 teachers.
• Public training and evaluation pipelines built with Hydra, Lightning, and Pixi.
• Published ALOE Hub checkpoints for DINOv3, SigLIP2, and supervised ViT families.
• Evaluation code for linear probing, k-NN, zero-shot transfer, GridPG, and pixel deletion.
• Explanation methods including B-cos, AttnLRP, integrated gradients, LeGrad, CheferCAM, LIME, and other Captum-based baselines.

All values below are from the paper and are reported on ImageNet-1k. Linear probe and k-NN measure recognition quality. GridPG measures localization quality; teacher GridPG uses the strongest reported teacher-side post-hoc baseline in the table, AttnLRP, while ALOE uses model-inherent B-cos attributions.

Across 10-dataset ViT-B/16 frozen-feature linear evaluation, ALOE improves over vanilla B-cosification by +13.24 points for supervised ViT, +7.62 points for SigLIP2, and +15.82 points for DINOv3 while staying close to the original teacher models.

The model-inherent attributions are object-centric and class-specific. The PCA visualizations show that alignment preserves the teacher's spatially structured feature geometry while making its evidence directly inspectable.

ALOEv2 is the multi-resolution DINOv3 follow-up. It fine-tunes the ALOE DINOv3 models with per-step 224/384/480-pixel sampling and corrects the selected distillation depths to include the final transformer block. This removes the train/evaluation resolution mismatch that hurt the original models on dense prediction while preserving classification quality and inherent B-cos explanations.

Multi-resolution alignment restores dense-feature quality close to the DINOv3 teacher across correspondence, surface normals, and monocular depth.

The public ALOE model set contains eight backbones:

Seven ALOE checkpoints include their trained ImageNet-1k linear-probe classifier:

ALOEv2 provides matching DINOv3 backbones and ImageNet-1k classifiers:

Published ALOE checkpoints use custom Hugging Face transformers code, so load them with trust_remote_code=True.

from transformers import AutoImageProcessor, AutoModel

repo_id = "rmaser/aloe-dinov3-base"
processor = AutoImageProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

Load a checkpoint with its ImageNet-1k classifier through the classification auto class:

from transformers import AutoImageProcessor, AutoModelForImageClassification

repo_id = "rmaser/aloe-dinov3-base-in1k-lp"
processor = AutoImageProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForImageClassification.from_pretrained(
    repo_id,
    trust_remote_code=True,
)
model.eval()

Classifier checkpoints expose model-inherent explanations directly through model.explain(...):

from PIL import Image
from transformers import AutoImageProcessor, AutoModelForImageClassification

repo_id = "rmaser/aloe-dinov3-base-in1k-lp"
processor = AutoImageProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForImageClassification.from_pretrained(
    repo_id,
    trust_remote_code=True,
)
model.eval()

image = Image.open("image.jpg").convert("RGB")
pixel_values = processor(images=image, return_tensors="pt").pixel_values

result = model.explain(pixel_values, idx=None)
class_idx = int(result["explained_class_idx"][0])
print(f"Predicted ImageNet-1k class index: {class_idx}")

rgba = (result["explanation"][0] * 255).astype("uint8")
Image.fromarray(rgba).save("explanation.png")

idx=None explains the predicted class. Pass an ImageNet-1k class index to idx to explain a specific class instead.

ALOE publishes vision encoders only. Pair a SigLIP2-family checkpoint with the original SigLIP2 text encoder named in its config, then pass normalized text features to explain_language_features to explain image-text cosine similarity:

import torch
import torch.nn.functional as F
from PIL import Image
from transformers import AutoImageProcessor, AutoModel, AutoTokenizer, Siglip2TextModel

repo_id = "rmaser/aloe-siglip2-base"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

processor = AutoImageProcessor.from_pretrained(repo_id, trust_remote_code=True)
image_model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).to(device)
image_model.eval()

text_model_id = image_model.config.aloe_base_model_name
tokenizer = AutoTokenizer.from_pretrained(text_model_id)
text_model = Siglip2TextModel.from_pretrained(text_model_id).to(device)
text_model.eval()

labels = ["a person eating spaghetti", "a person playing guitar", "a person running"]
prompts = [f"This is a photo of {label}.".lower() for label in labels]
tokens = tokenizer(
    prompts,
    padding="max_length",
    truncation=True,
    max_length=64,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    text_features = F.normalize(text_model(**tokens).pooler_output, dim=-1)

image = Image.open("image.jpg").convert("RGB")
pixel_values = processor(images=image, return_tensors="pt").pixel_values.to(device)

explanation = image_model.explain_language_features(
    pixel_values,
    text_features,
    idx=None,
)
# explanation["explanation"]         — RGBA attribution overlay, (1, H, W, 4)
# explanation["contribution_map"]    — input×gradient map, (1, 1, H, W)
# explanation["explained_class_idx"] — index into labels

idx=None explains the highest-scoring label; pass a label index to explain a specific prompt. Explanation calls currently expect one input image at a time.

For code that needs this repository's Hydra/model-factory path, use aloe_model_loader.py. It is intentionally thin and delegates to src.models.model_loader.

This repository uses Pixi and is configured for Linux CUDA environments.

export SCRATCH=/path/to/scratch
pixi install

SCRATCH is required. Pixi resolves project paths from it:

Public training tasks are defined in pixi.toml and launch Hydra multiruns through scripts/train.py.

pixi run train_native_distill_siglip2_base
pixi run train_native_distill_siglip2_large
pixi run train_native_distill_siglip2_so400m
pixi run train_native_distill_siglip2_so400m_432

pixi run train_native_distill_dinov3_small
pixi run train_native_distill_dinov3_base
pixi run train_native_distill_dinov3_large

pixi run train_native_distill_google_vit_base

The training configs live under configs/experiment/native/. They freeze the teacher, train an ALOE B-cos student, and align global and token-level features with the objectives and model factories defined under configs/objective/, configs/loss/, configs/model/factory/, and src/training/.

Run linear probing and k-NN over the published Hub checkpoints:

pixi run eval_native_hf_all_hub_lp

Run classifier/explanation evaluation over published ALOE classifiers:

pixi run eval_native_classifier
pixi run eval_native_classifier_distilled_supervised

Run baseline HF backbone evaluation:

pixi run eval_hf_baseline

The public Hub evaluation sweep covers the eight published ALOE backbones and the main classification datasets used in the paper: ImageNet-1k, Caltech101, Stanford Cars, CIFAR-10, CIFAR-100, DTD, Oxford Flowers102, Food101, SUN397, and FGVC Aircraft.

Important config groups:

This project is licensed under the Creative Commons Attribution-NonCommercial ShareAlike 4.0 International License. The methods described in this repository are patent pending.

@inproceedings{maser2026align,
  title = {Align Once to Explain: Feature Alignment for Scalable B-cosification of Foundational Vision Transformers},
  author = {Maser, Raphael and Gairola, Siddhartha and Rao, Sukrut and Schiele, Bernt},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2026},
  note = {Poster}
}