MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments

Han Wang^1*, David Wan^1* Hyunji Lee^1*, Thinh Pham², Mikaela Cankosyan², Weiyuan Chen²,
Elias Stengel-Eskin³, Tu Vu², Mohit Bansal^1
1UNC Chapel Hill, ²Virginia Tech, ³UT Austin
* Equal contribution

MERRIN is a benchmark designed to evaluate whether search-augmented models can autonomously determine which non-text modalities (images, videos, charts) to retrieve and reason over when answering questions. All 162 questions in MERRIN satisfy three criteria:

1. Natural text input — questions are plain language without explicit references to specific modality sources (e.g., "In the first episode of Rick and Morty Season 8")
2. Non-text modality required — correct answers strictly require visual, video, or audio evidence
3. Single unambiguous short answer — enables automatic evaluation

MERRIN tests the full pipeline from modality-autonomous retrieval to cross-modal reasoning in noisy web environments.

The MERRIN benchmakr contains 162 human-annotated questions spanning diverse modalities.

The benchmark is available on 🤗 HuggingFace. The question, answer, and resources fields are encrypted to prevent data contamination in LLM training corpora.

python load_dataset.py --output data/questions/MERRIN.jsonl

This downloads the dataset from HuggingFace, decrypts the encrypted fields, and saves a JSONL file for evaluation.

We evaluate models under three settings with increasing tool access:

We currently support three model providers:

Adding open-source models via vLLM: First serve the model with vLLM:

# Example: serve Qwen3-4B-Thinking on port 8000
vllm serve Qwen/Qwen3-4B-Thinking-2507 --port 8000 --max-model-len 262144 --reasoning-parser deepseek_r1

# For agent framework (tool calling), add:
vllm serve Qwen/Qwen3-4B-Thinking-2507 --port 8000 --max-model-len 262144 --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser deepseek_r1

Then register the model in src/evaluate/config.py:

MODELS: dict[str, ModelConfig] = {
    # ... existing models ...

    # Add your model here
    "your-model-name": ModelConfig(
        "vllm",                           # provider
        "org/model-name",                 # HuggingFace model ID (must match vLLM)
        False,                            # supports_search (False for open-source)
        base_url="http://localhost:8000/v1",  # vLLM endpoint
    ),
}

Adding other API models: For models with OpenAI-compatible APIs, use the "openai" provider with a custom base_url. For Gemini models, use the "gemini" provider.

git clone https://github.com/HanNight/MERRIN.git
cd MERRIN
pip install -r requirements.txt

Set the following environment variables as needed:

export GOOGLE_API_KEY="your-google-api-key"          # For Gemini models
export AZURE_OPENAI_API_KEY="your-azure-api-key"     # For GPT models (Azure)
export SERPER_API_KEY="your-serper-api-key"           # For agent framework web search

The --conditions argument controls which search tools the model can access:

# No Search (all providers)
python -m src.evaluate.run evaluate \
    --questions data/questions/MERRIN.jsonl \
    --output-dir experiments/results \
    --models gemini-3-pro-preview \
    --conditions no_search \
    --concurrency 10

# Native Search — Gemini (google_search + url_context)
python -m src.evaluate.run evaluate \
    --questions data/questions/MERRIN.jsonl \
    --output-dir experiments/results \
    --models gemini-3-pro-preview \
    --conditions with_url_context \
    --concurrency 10

# Native Search — OpenAI (web_search)
python -m src.evaluate.run evaluate \
    --questions data/questions/MERRIN.jsonl \
    --output-dir experiments/results \
    --models gpt-5.4-mini \
    --conditions with_search \
    --thinking-level high \
    --concurrency 10

python -m src.evaluate.agent_runner \
    --questions data/questions/MERRIN.jsonl \
    --output-dir experiments/results/agent \
    --model gemini-3-pro-preview \
    --tools web_search visit_webpage search_video watch_video \
    --concurrency 5

Available agent tools:

• web_search — Google search via Serper API (default, 10 results)
• web_search_custom — Configurable number of results (use --web-search-num N)
• visit_webpage — Fetch and read web pages including images (powered by Gemini)
• search_video — Search for YouTube videos via Serper API
• watch_video — Process YouTube videos for visual and audio understanding (powered by Gemini)

Additional options:

• --thinking-level {none,low,medium,high,xhigh} — Set reasoning effort
• --tool-model MODEL — Gemini model used by visit_webpage and watch_video (default: gemini-3-flash-preview)
• --run-id N — Run ID for multi-run experiments (appends _run-N to output)
• --video-num N — Number of videos returned by search_video (default: 3)
• --web-search-num N — Number of results for web_search_custom (default: 10)

# LLM-as-Judge scoring
python -m src.evaluate.run score \
    experiments/results/gemini-3-pro-preview_no_search.jsonl \
    --judge-model gemini-3-flash-preview

If you find MERRIN useful, please cite our paper:

@article{wang2026merrin,
  title={MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments},
  author={Han Wang and David Wan and Hyunji Lee and Thinh Pham and Mikaela Cankosyan and Weiyuan Chen and Elias Stengel-Eskin and Tu Vu and Mohit Bansal},
  year={2026},
  journal={arXiv preprint arXiv:2604.13418}
}