Bio-VQA

How Finely Can Vision–Language Models See Biology?
Bio-VQA: A Benchmark for Taxonomic, Morphological, Spatial and Individual Reasoning

S M Rayeed · Qian Ma · Charles V. Stewart · Yao Ma

🌐 Project page  |  🤗 HF Dataset  |  💻 Github Repo  |  📄 Paper: WACV 2027 (link will be added here)

• 2026-11 — Camera-ready version for WACV 2027. This README, the numbers in every component README and the environment notes are updated to match the camera-ready paper.
• 2026-11 — The full benchmark, Bio-VQA-Lite (500 questions per task, 12,000 in total), and the per-item answers of all 18 models are public on Hugging Face: imageomics/Bio-VQA.

In biodiversity monitoring, a single image raises several questions: which taxon, how many, which trait, and whether this individual has been seen before. In practice each is answered by its own specialist model, or by a field expert. Bio-VQA asks whether one generalist vision–language model (VLM) can answer all four.

Bio-VQA holds 1,861,608 questions over 216,715 images from 11 public datasets, organized into four components and 24 tasks. Every question is generated only where an existing annotation supplies the ground truth, so every answer can be checked against its source. Every task is bounded by an image-blind floor and, where one exists, by a domain-specialist ceiling. Scoring is graded, so a near miss counts for more than a distant one, and each graded score is reported next to the exact match it modifies.

C3 draws on WAID, Boxes on Bees, 2018-NEON-Beetles and DePondFi, and also on four of the C4 catalogs (WhaleSharkID, HyenaID, LeopardID, GZCD) for localization and viewpoint. That is why the component sources add up to more than 11.

Where to find what

• Quick start
• The data at a glance
• Repository layout
• Pipeline
  • Stage overview
  • C1 — Hierarchical taxonomic reasoning
  • C2 — Fine-grained attribute grounding
  • C3 — Quantitative spatial reasoning
  • C4 — Individual re-identification
  • Release tooling (Hugging Face and Lite)
• Environment
• Evaluation protocol
• Metrics
• Results
  • Full results table
  • Findings by component
  • Cross-component structure
• Reproducibility
• Limitations
• License
• Citation
• References

Bio-VQA-Lite is a stratified sample of 500 questions per task, 12,000 in total. It is drawn with a fixed seed and is meant for cheap evaluation and for development. Details are on the dataset card.

from datasets import load_dataset

lite = load_dataset("imageomics/Bio-VQA", "lite")   # C1.csv, C2.csv, C3.csv, C4.csv
print(lite)

# Read one image that is stored inside a zip shard (C2 marker renders, C3, C4)
import io, zipfile
from huggingface_hub import hf_hub_download
from PIL import Image

def load_image(component, image_zip, image_in_zip):
    path = hf_hub_download("imageomics/Bio-VQA", f"{component}/{image_zip}", repo_type="dataset")
    with zipfile.ZipFile(path) as z:
        return Image.open(io.BytesIO(z.read(image_in_zip))).convert("RGB")

C1 images come from TreeOfLife-200M and C2 photographs come from CUB-200-2011. Neither is re-hosted. See Images on the dataset card, and Release/fetch_c1_images.py for C1.

Every scorer in this repository reads a question file that has one extra column named <model>_answer. That column holds your model's raw, unparsed reply. Parsing and normalization happen inside the scorer, and they are the same for every model.

# 1) append a column  <mymodel>_answer  to the component question file(s)
# 2) run that component's scorer
python 1.Hierarchical_taxonomic_reasoning/Evaluation/Evaluate_build_synonyms.py ...   # once, before C1 scoring
python 1.Hierarchical_taxonomic_reasoning/Evaluation/Evaluate_run.py --base_dir <dir with C1 csvs> --parquet <ToL catalog> --synonyms <synonym map>
python 2.Fine-grained_attribute_grounding/Evaluation/evaluate.py --base_dir <dir with C2 csvs>
python 3.Quantitative_spatial_reasoning/Reasoning_Tasks/<Task>/3.Evaluate/evaluate.py --results_dir <dir>
python 4.Individual_reidentification/Evaluation/Evaluate.py --base_dir <C4 root>

Each Evaluation/ or 3.Evaluate/ folder has a README with every flag spelled out. C1 note: build the synonym map first. Without it, soft match silently falls back to hard match and no error is raised.

Each component has one runner script per model (Runs/, Experiment-Runs/, 2.Experiment_Runs/, Experiment_Runs/). The launch commands are written out in each runner folder's README. See Pipeline and Environment.

The predictions/ folder on Hugging Face holds the per-item answers of all 18 models and both specialists, exactly as scored in the paper. Point a scorer at those files to regenerate any table.

This is a short summary. The dataset card has the full description, file layout, column reference and licensing.

C1 asks the reptiles group seven rank questions, not eight, because GBIF has no order rank for Squamata. So that group contributes 8 question types instead of 9.

Bio-VQA/
├── README.md                                  ← you are here
├── requirements.txt                           ← shared environment (see "Environment")
├── LICENSE                                    ← MIT, code only
│
├── 1.Hierarchical_taxonomic_reasoning/        ← C1
│   ├── README.md
│   ├── Tree-of-Life-200M/                     ← data: download.py · verify.py · generateQA.py · stat.py
│   │   └── Resolve_common_names/              ← non-English common names → English (iNaturalist, Wikidata)
│   ├── Runs/                                  ← 18 runners: run_<model>.py
│   ├── Post-run/                              ← Merge_csv_chunks.py · Fix_Llama_Answers.py · Assign_difficulty/
│   └── Evaluation/                            ← Evaluate_build_synonyms.py · Evaluate_check_synonyms.py · Evaluate_run.py
│                                                 run_pyBioCLIP.py · prefetch_pybioclip.py  (BioCLIP-2 ceiling)
│
├── 2.Fine-grained_attribute_grounding/        ← C2
│   ├── README.md
│   ├── Data-Preparation/                      ← 1.preprocess.py · 2.prepareQA.py · 3.stat.py · taxon_name_mapping.json
│   ├── Experiment-Runs/                       ← per-family runners: run_qwen.py · run_InternVL.py · run_gemma.py · …
│   └── Evaluation/                            ← evaluate.py (CAGS) · evaluate_soft.py · sensitivity_sweep.py
│       └── Human Study/                       ← 306-item, 3-annotator validation of CAGS
│
├── 3.Quantitative_spatial_reasoning/          ← C3
│   ├── README.md
│   ├── Datasets/                              ← Aerial/ (WAID) · Bees/ · Beetles/ · Fish/ (DePondFi) · Zebra/ (GZCD)
│   └── Reasoning_Tasks/
│       ├── Counting/          ├── Counting_Relative/   ├── Distance_Absolute/   ├── Distance_Relative/
│       ├── Localization/      ├── Relative_Position/   ├── Viewpoint/           └── OCR/
│       └── <task>/  1.Generate_QA/  →  2.Experiment_Runs/ (Preprocess/ + 18 runners)  →  3.Evaluate/
│
├── 4.Individual_reidentification/             ← C4
│   ├── README.md
│   ├── Data_Preparation/                      ← download.py · extract_embeddings.py (DINOv2) · generate_QA.py
│   ├── Experiment_Runs/                       ← 18 VLM runners + megadescriptor.py (specialist)
│   └── Evaluation/                            ← Evaluate.py (accuracy, d′, c, bootstrap CIs)
│
└── Release/                                   ← how the Hugging Face release was built
    ├── README.md
    ├── make_lite.py                           ← Bio-VQA-Lite (500 per task, seeded, stratified)
    ├── pack_images.py                         ← images → zip shards + image_zip / image_in_zip columns
    └── fetch_c1_images.py                     ← fetch C1 images from TreeOfLife-200M source URLs

Each component folder and most subfolders have their own README with construction details, every command, and results. No shell scripts are committed; every launch command is written out in a README.

 public dataset ──► 1. acquire & filter ──► 2. generate QA ──► 3. preprocess / shard ──► 4. inference (18 VLMs + specialist)
 (annotations)        (download, verify,      (seeded; only      (manifest, row_id,         (greedy, one attempt,
                       resolve names)          where GT exists)   pre-run audit)             raw string stored)
                                                                                                  │
                    6. tables & figures ◄── 5. score (one scorer per component; floors, graded metrics, CIs) ◄┘

Every component follows these six stages. The folder names differ a little between components (shown below), but what each stage does is the same.

📁 1.Hierarchical_taxonomic_reasoning/  ·  🤗 config C1_taxonomy

1. Sample. download.py runs DuckDB queries over the TreeOfLife-200M catalog. It takes 10,000 images from each of 10 organismal groups (amphibians, angiosperms, arachnids, birds, bryophytes, fish, gymnosperms, insects, mammals, reptiles). Three constraints apply: (a) the lineage must be complete from phylum to species, with a common name; (b) the image must meet a minimum resolution (--min_size_kb); (c) there is a per-species cap (--images_per_species), so abundant species do not take over a group. The sample spans 626 families, 2,311 genera and 3,403 species — 6× VLM4Bio's genus coverage and 4× its species coverage.
2. Verify. verify.py checks that a manifest row exists if and only if its image is on disk and every rank the generator needs is non-null.
3. Resolve names. GBIF vernacular names are not filtered by language. They include Swedish, Danish and Norwegian names. Resolve_common_names/ resolves every non-English common name to English against iNaturalist and Wikidata.
4. Generate QA. generateQA.py asks each image for its phylum, class, order, family, genus, species epithet, scientific name and common name. It also asks the common name a second time as 4-way multiple choice, with distractors drawn uniformly from taxa in the same class as the target.
5. Run. There are 18 runners in Runs/. The BioCLIP-2 ceiling is Evaluation/run_pyBioCLIP.py, in three label-set configurations: tol (the full TreeOfLife-200M label set, used in the paper's main table), kingdom and group.
6. Post-run. Merge chunks, repair Llama-3.2 answer formatting, and assign Easy / Medium / Hard tiers by how common the target taxon is in TreeOfLife-200M (top third, middle third and bottom third by image count at each rank).
7. Score. Exact match, soft exact match (any GBIF Backbone synonym of the target counts as correct, so Felis concolor = Puma concolor), Wu–Palmer, Lin similarity, and error severity.

📁 2.Fine-grained_attribute_grounding/  ·  🤗 config C2_attributes

1. Invert the attributes. CUB-200-2011 stores 312 binary attributes per image as part · property · value (has_wing_color::brown). Asking 312 sparse yes/no questions per image would be uninformative, since a constant no scores well. So 1.preprocess.py and 2.prepareQA.py merge each part and property into one question (What is the wing color?) and take the marked value as the answer. Annotator certainty is used to select and filter candidate traits.
2. Distractors. These are other values attested for the same part and property. Any value that is also true of the image is excluded, so exactly one option is correct.
3. Marker localization. A marker is rendered onto each of the 11,788 images, and the question asks which body part it marks. Two controlled distractors are added: one shares the marker's color but not its shape, the other shares its shape but not its color. The model has to bind both cues to answer. The renders double the image count to 23,576.
4. Ten question families in four categories. See the data at a glance.
5. Score. CAGS (chance-adjusted graded score), using CIEDE2000 color distance for colors and annotation co-occurrence for parts. It is 0 at chance and 1 at perfect for any number of options. sensitivity_sweep.py reruns the ranking under 11 kernel and hyperparameter settings. Human Study/ holds the 306-item, 3-annotator validation.

📁 3.Quantitative_spatial_reasoning/  ·  🤗 config C3_spatial

The tasks follow AVA-Bench's decomposition of spatial reasoning into atomic skills, so a failure can be traced to a specific ability. The source datasets are aggregated into ten 500-image subsets. These are count-stratified, so sparse and dense scenes are balanced, and images whose annotation completeness cannot be verified are dropped. Each task runs only where its ground truth exists:

Every task folder has the same three stages: 1.Generate_QA/ → 2.Experiment_Runs/ (with Preprocess/split_qa.py and the 18 runners) → 3.Evaluate/ (merge_results.py, evaluate.py). Within each task, the block marked # HARNESS is byte-identical across the 18 runners. You can check this:

cd 3.Quantitative_spatial_reasoning/Reasoning_Tasks/Counting/2.Experiment_Runs
for f in *.py; do sed -n '/^# HARNESS/,$p' "$f" | md5sum; done | sort -u | wc -l     # prints 1

Counting tiers. Every counting canvas has the same pixel budget. Easy fills the canvas with one image. Medium splits it into two panels from different datasets. Hard splits it into two panels from the same dataset. OCR is a designed negative control: locating the label is spatial, but the answer depends on transcription.

📁 4.Individual_reidentification/  ·  🤗 config C4_reidentification

1. Catalogs. WhaleSharkID (spot pattern), BelugaID (scarring, dorsal ridge), HyenaID (spot pattern), LeopardID (rosettes) and GZCD Grevy's zebra (stripes).
2. Embed. extract_embeddings.py computes DINOv2 embeddings. Retrieval runs within viewpoint groups, because viewpoint dominates the embedding space.
3. Pair (generate_QA.py, seeded). Each catalog gives 1,000 balanced pairs per condition (500 same and 500 different), and same-individual pairs are drawn first because they are scarce.
   • Easy: a same pair is an animal with its nearest match; a different pair is two animals that look nothing alike. Appearance and identity agree.
   • Random: both pairs are drawn without using appearance, so appearance carries no signal.
   • Hard (adversarial): a different pair is two nearest neighbors; a same pair is an animal with its furthest sighting. Appearance and identity disagree.
4. Ask. The two photographs are placed side by side on one canvas, and the question is: "This image shows two images placed side by side. Is the main subject in the left image the very same individual animal as the main subject in the right image?" The runner adds "Answer with one word: Yes or No. Do not explain." The prompt never reveals that the set is balanced.
5. Specialist. megadescriptor.py runs MegaDescriptor on Grevy's zebra.
6. Score. Evaluate.py reports accuracy and d′ / criterion c from signal detection theory, with a log-linear correction, plus 95% bootstrap CIs (2,000 resamples per cell), balanced accuracy, MCC and Cohen's κ.

📁 Release/ — the scripts that built the Hugging Face release:

All 18 models run in one shared Python 3.12 environment (requirements.txt). The one exception is Molmo2-8B: AI2 ships Molmo2 as custom code pinned to transformers==4.57.1, so it has its own environment.

# shared environment (17 models, data preparation, scoring)
python3.12 -m venv bio-vqa && source bio-vqa/bin/activate
pip install -r requirements.txt

# Molmo2-8B only
python3.12 -m venv bio-vqa-molmo2 && source bio-vqa-molmo2/bin/activate
pip install -r requirements.txt "transformers==4.57.1"

Hardware. C1 image download is network- and I/O-bound, so use CPU nodes. Inference needs one H100-class GPU per job; the 30B–38B models need two. Some runner READMEs show SLURM launch lines that build a per-job virtual environment in $TMPDIR. That is a scheduling convenience; it installs the same packages.

The exact checkpoint of each model is listed in the supplementary material (Table S2) and in each runner's MODEL BLOCK. Examples: Qwen/Qwen3-VL-32B-Instruct, OpenGVLab/InternVL3-38B-Instruct, google/gemma-3-12b-it, allenai/Molmo2-8B, mistral-community/pixtral-12b, microsoft/Phi-3.5-vision-instruct, meta-llama/Llama-3.2-11B-Vision-Instruct, llava-hf/llava-v1.6-vicuna-7b-hf, llava-hf/llava-v1.6-mistral-7b-hf.

The 18 models span 3.75B–38.4B parameters across eleven families: Qwen2.5-VL (3B, 7B), Qwen3-VL (8B, 30B-A3B MoE, 32B), InternVL3 (8B, 38B), InternVL3.5 (8B, 30B-A3B MoE, 38B), Gemma-3-12B, Gemma-4-12B, Molmo2-8B, Pixtral-12B, Phi-3.5-Vision, Llama-3.2-11B-Vision, and LLaVA-NeXT (Vicuna-7B, Mistral-7B).

Specialists. BioCLIP-2 for C1, which is trained on TreeOfLife-200M and ranks a closed label set, so treat it as an optimistic upper bound. MegaDescriptor for C4, on Grevy's zebra.

Exact match is reported for every task. Every graded metric is reported next to the exact match it modifies, so no claim rests on the graded score alone.

A lift of 0 means the model does no better than an image-blind rule; negative lift means it does worse. Two C3 tasks (localization and absolute distance) have field means below their floors.

18 models × 24 tasks (Table 2 of the paper). Bold marks the best of the 18 models in each column.

Columns. C1 (exact match): phylum, class, order, family, genus, species epithet, scientific name, common name (open-ended). C2 (CAGS): relative salience, morphological attributes, anatomical localization, naming. C3: counting (CAS), relative counting, absolute distance (share within 20%), relative distance, localization (mean IoU), relative position, viewpoint, ocr (exact match). C4: d′ and accuracy under random pairing. Floor is the score an image-blind rule reaches; for C1 it is the most frequent taxon at each rank. CAGS and d′ are chance-corrected, so their floors are zero. ¹ Domain specialist for taxonomy, full TreeOfLife-200M label set. ² Domain specialist for re-identification, on Grevy's zebra only.

Per-item answers for every cell are in predictions/ on Hugging Face. Leaderboards with confidence intervals, per-group, per-tier and per-dataset breakdowns are in each component README and in the supplement.

General visual capabilities do not transfer to biological organisms, and scale does not close the gap. The failures below hold across a tenfold range of model size (3.75B → 38.4B).

† not significant after Holm correction. At n = 18 the confidence intervals are wide (median width .66).

• Tasks agree more within a component than across components: mean ρ .749 vs .534 (label permutation, p = .0005). Internal agreement is .841 for C1, .662 for C3, .645 for C2 and .617 for C4.
• C4 stands alone. Knowing how well a model names an organism says nothing about whether it can tell two individuals of that organism apart.
• C2 and C3 overlap (ρ = .927). Both ask a model to find something fine-grained. They are reported separately because they diagnose different failures, but they should not be treated as independent evidence.
• A strong model is strong everywhere; below the top, rankings reorder. One factor explains 62.2% of the variance. InternVL3-38B, Qwen3-VL-30B and Qwen3-VL-32B are top five in every component. The median model moves 5.5 of 18 places between components (Phi-3.5-Vision: 8th percentile on C1, 64th on C4).

Given a manifest, everything downstream reproduces exactly. Use the released question files rather than expecting to regenerate the C1 image set byte for byte.

• Open-weight models only; we cannot say whether proprietary models (e.g. GPT-5) behave the same way.
• One decoding protocol (zero-shot, greedy, no chain-of-thought). Reasoning modes may score higher.
• Images come from public datasets that a model may have seen in pretraining, including BioCLIP-2's training corpus for C1.
• No human performance reference. Human raters validate C2 scoring only; C1, C3 and C4 ground truth have no human audit.
• Floors are rule-based; a text-only VLM may set a higher one.
• Two models (Molmo2-8B, Phi-3.5-Vision) mostly copy the example box on localization.
• Species-epithet answers are truncated in the released runs, so species results are analyzed using the scientific name.
• The CAGS anatomy kernel over-credits laterality confusions.
• Localization is evaluated on the Easy tier (3,988 items).

• Code in this repository: MIT.
• Benchmark annotations (questions, answers, Lite, predictions) on Hugging Face: CC BY-NC 4.0.
• Images keep the license of their source dataset. See the dataset card for per-source terms. Check them before redistributing any image.

If you use Bio-VQA, please cite:

@inproceedings{rayeed2027biovqa,
  title     = {How Finely Can Vision--Language Models See Biology? {Bio-VQA}: A Benchmark for Taxonomic, Morphological, Spatial and Individual Reasoning},
  author    = {Rayeed, S M and Ma, Qian and Stewart, Charles V. and Ma, Yao},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}

Links: 🌐 Project page · 🤗 Dataset · 💻 Code

Every source dataset, baseline, model, metric and benchmark referenced by Bio-VQA or by any component README. The key column is the BibTeX key used in the paper, so entries can be cross-referenced against the manuscript's .bib directly.

All source imagery retains its original license. This repository distributes code only; the benchmark data are on Hugging Face. Verify licensing before redistributing any downloaded imagery.

Eleven public datasets supply every image and every ground-truth annotation. A question is generated only where an existing annotation supplies the answer.

BioCLIP-2 and MegaDescriptor are the measured ceilings for C1 and C4. DINOv2 is used for C4 pair mining, not as a baseline. The remaining entries are the re-identification literature the component situates itself against.

Eighteen open-weight models from 3.75B to 38.4B parameters. Two of the 30B models are mixture-of-experts activating 3B parameters per token.

Every graded metric is reported alongside the exact match it modifies.

_{Bio-VQA · project page · dataset · images retain their original licenses}