BenchBrowser helps you check whether benchmarks are a valid proxy for your intent. It retrieves candidate benchmark items aligned to your use case, lets you curate an evidence set, and provides tools to inspect coverage (content validity) and consistency across benchmarks (convergent validity). Access the tool here.
pip install -r requirements.txt
- Core Components (
src/components/)
| File | Description |
|---|---|
| retriever.py | Implements dense (BGE+FAISS) and sparse (BM25) retrieval systems with configuration management. |
| evaluator.py | Evaluation framework for scoring retrieved samples using LLM judges (OpenAI, Gemini, Anthropic) and computing precision, NDCG, and recall metrics. |
| embedder.py | Embedding generation utilities for converting text to vector representations using various embedding models. |
| processors.py | Data processors for different benchmark formats (truthfulQA, ifeval, arc, mmlu, bbh) that load data and compute task-specific accuracies. |
| subsamplers.py | Implements sampling strategies for retrieved results including inverse MRR subsampling and re-ranking. |
| refiner.py | Query refinement utilities to improve retrieval quality through query expansion and reformulation. |
| utils.py | General utility functions used across components. |
- Shorthand Processing Pipeline (
src/shorthand/)
The shorthand pipeline transforms test cases into compact representations. See src/shorthand/use.MD for detailed usage.
- Main Analysis Scripts are in (
src/) - Ablation Studies (
ablations/)
To run this codebase, you will need to download/set:
-
Models:
<path-to-bge-en-icl-model>: We use the BAAI/bge-en-icl model.<model-to-finetune>/``<path-to-finetuned-model>: We use the meta-llama/Meta-Llama-3-8B-Instruct model.
-
Data Files:
<path-to-benchmark-templates>: Directory containing benchmark templates (already provided). This is a folder of dummy predictions used to generate the samples that are stored in the database.- Various accuracy maps and task mappings (see individual script arguments)
-
API Keys:
- OpenAI API key (for GPT judges and embeddings)
- DeepInfra API key (for inference) using BGE.
- Gemini API key (for Gemini judges)
Refer to individual script documentation and the shorthand pipeline guide (src/shorthand/use.MD) for specific usage instructions.
