Hidden in plain sight: VLMs overlook their visual representations

Stephanie Fu, Tyler Bonnen, Devin Guillory, Trevor Darrell

UC Berkeley

To start, clone this repository and create the environment with Mamba (or other favorite Conda-flavored package manager):

mamba env create -f environment.yaml
mamba activate vlm_readout

# Example command to run Semantic Correspondnece on DINOv2
python eval_runner.py \ 
    task=semantic_correspondence \
    backbone=dinov2_l14_reg \
    vlm_id="dinov2-224px+7b"

# Example command to run Depth Estimation with pretrained probe on CLIP
python eval_runner.py \
    task=cvbench_depth \
    backbone=clip_l14 \
    vlm_id="clip-224px+7b" \
    dpt_path="probes/clip_vitlopenai_14_dense-depth.pt"

Task options: semantic_correspondence, visual_correspondence, functional_correspondence, art_style, mochi, cvbench_depth

Model options:

• backbone=dinov2_l14_reg vlm_id="dinov2-224px+7b"
• backbone=vit_l16 vlm_id="in1k-224px+7b"
• backbone=siglip_l14 vlm_id="siglip-224px+7b"
• backbone=clip_l14 vlm_id="clip-224px+7b"

blind_vlm=True will pass in a blank image in place of the benchmark image. The text prompt will be unaffected.

To use our pretrained depth probes, please download them here and place them in probes/.

After running the evaluations, plot results (like Figures 2-3) with:

python plot_main_results.py # Single-layer results (like Figures 2-3)
python plot_multilayer_results.py # Multi-layer results (like Figure 5)
python plot_answer_distributions.py # Answer distributions (like Figure 4)

Our training datasets are generated with scripts in data_gen. Note that depending on random seed, the exact datasets may vary slightly (e.g., points chosen for correspondence or exact image files drawn from larger dataset).

# Generate art style examples, configurable in configs/gen_data.yaml
python gen_data.py dataset=art_style

If you find this work useful in your research, please cite our paper:

@misc{fu2025hiddenplainsightvlms,
      title={Hidden in plain sight: VLMs overlook their visual representations}, 
      author={Stephanie Fu and Tyler Bonnen and Devin Guillory and Trevor Darrell},
      year={2025},
      eprint={2506.08008},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2506.08008}, 
}