PitchBench

Benchmark suite for evaluating pitch and acoustic perception in audio language models (ALMs). Probes pitch identification, temporal localisation, chord recognition, melodic contour, robustness to audio effects, and more. The paper comparison uses MIDI, SPN and Hz; D8/F1/F2 use Ordered Note F1 and other tasks use accuracy.

Table 1: all 28 tasks × 8 models is the canonical paper comparison, including DSP and Basic Pitch. CSV, LaTeX, and source mapping are generated together. The reproduction guide documents the metrics, sample coverage, and underlying evidence.

PYTHONPATH=src python -m pitchbench.analysis.table1 --check

This checks the committed table against its evidence without model calls, audio, or third-party Python packages. Omit --check to regenerate all formats.

Regenerate Figures 1–4 with the paper layout:

PYTHONPATH=src python -m pitchbench.analysis.publication

The figure generator uses saved evidence and bundled fonts; it makes no model calls.

Skip this step when evaluating an existing Parquet dataset.

A GM soundfont is also required. Place any .sf2 or .sf3 in data/soundfonts/ — no further configuration needed.

Linux / WSL:

sudo apt install fluid-soundfont-gm
mkdir -p data/soundfonts
cp /usr/share/sounds/sf2/FluidR3_GM.sf2 data/soundfonts/

macOS / cross-platform:

curl -L "http://ftp.debian.org/debian/pool/main/f/fluid-soundfont/fluid-soundfont-gm_3.1-5.3_all.deb" \
     -o /tmp/fluid.deb
cd /tmp && ar x fluid.deb && tar xf data.tar.* ./usr/share/sounds/sf2/FluidR3_GM.sf2
mkdir -p data/soundfonts && cp /tmp/usr/share/sounds/sf2/FluidR3_GM.sf2 data/soundfonts/

SF2 resolution order: PITCHBENCH_SF2 env var → data/soundfonts/ → /usr/share/sounds/sf2/FluidR3_GM.sf2.

python3 -m venv .venv
source .venv/bin/activate        # Linux / macOS
# .venv\Scripts\activate         # Windows
pip install -e .               # install the version in this checkout

Use the source installation above for the scoring protocol and reproduction commands documented here.

For development tools:

uv sync --extra dev              # installs pytest + pytest-mock

Requires Python ≥ 3.12.

OpenRouter — add to .env:

OPENROUTER_KEY=sk-or-...

Use --model openrouter/<provider>/<model>. Some models may require a ~ prefix, e.g. openrouter/~google/gemini-flash-latest.

DashScope — add to .env:

DASHSCOPE_API_KEY=sk-...

Use --model dashscope/<model>.

Local server — use an OpenAI-compatible audio endpoint:

--model http://localhost:8001 --name my-local-model

PitchBench separates stimulus generation from model evaluation. Generate once, evaluate any number of models.

pitchbench generate paper
pitchbench generate a          # single category
pitchbench generate a1         # single experiment

Writes WAV files and a Parquet dataset to data/generated/<exp>/. Re-running is safe: existing rows are skipped.

data/generated/pitchbench_a1_single_pitch_id/
  *.wav               # one file per condition
  _questions.parquet  # HuggingFace-compatible schema
  _questions.csv      # human-readable companion

pitchbench --list
pitchbench evaluate paper --model openrouter/<provider>/<model>
pitchbench evaluate a1  --model openrouter/<provider>/<model>

# Quick test (20 stimuli, stratified)
pitchbench evaluate a1  --model openrouter/<provider>/<model> --sample-n 20 --sample-seed 0

Results land in results/evaluation/<model_slug>/<YYYYMMDD_HHMMSS>/.

paper selects exactly the 28 Table 1 tasks and queries MIDI/SPN/Hz only. It rejects missing tasks before reporting overall. all includes extra tasks such as Y1 and the additional reference variants.

Evaluation uses the stored stimuli; it does not require local audio synthesis or regenerate conditions to filter out instruments. Condition filtering belongs to analyze presets. D8/F1/F2 export Ordered Note F1 through the task summary, CSV and overall score. B3/B4/B5 export their timing accuracy as task-level scores; the paper overall is the equal-weight mean of all 28 tasks. Numeric summaries and CSVs retain unrounded scores; rounding is applied only for display.

In tmux (recommended for long runs):

tmux new-session -d -s mymodel "source .venv/bin/activate && \
  pitchbench evaluate paper --model openrouter/<provider>/<model> 2>&1 | tee logs/eval_mymodel_\$(date +%Y%m%d_%H%M%S).log"

Without any model specification, the system defaults to the localhost:8001 endpoint. See the Audio Flamingo Next server example for a local backend.

pitchbench analyze --preset q1 --model openrouter/<provider>/<model>

Writes to results/analysis/<model_slug>/<run>/ and generates a summary of the ablations.

Batch (multiple models):

python -m pitchbench.analysis.run_analysis q1 \
    --models openrouter/<provider>/<model1> \
             openrouter/<provider>/<model2> \
             dashscope/<model3>

results/evaluation/<model_slug>/<YYYYMMDD_HHMMSS>/
  pitchbench_<exp>/
    results_<model>.json      # metadata, git commit, per-item responses
    results_<model>.csv       # one row per stimulus
    accuracies_<model>.csv    # aggregate metrics
    plots/                    # per-format CI plots

results/evaluation/summary/ is populated automatically after each run (aggregated_accuracies.csv, accuracy plots by instrument / notation / note).

The paper benchmark contains 28 tasks across categories A–F. The package also provides four optional tasks: D7b, D7c, D7d, and Y1.

• Waveforms (always available): sine, sawtooth, square, triangle
• GM instruments (FluidSynth): piano, electric_keyboard, guitar, flute, trumpet, trombone, clarinet, oboe, violin, cello, organ, bass, synth_lead, synth_pad, voice
• Backgrounds (e2): white_noise synthesised; real recordings in data/preloaded/background/<name>.mp3

src/pitchbench/
  config.py                    # runtime constants and env wiring
  configs/
    benchmark_config.py        # paper-locked data-gen parameters
    analysis_config.py         # analysis-mode presets/overrides
    plot_config.py             # labels/colors for figures
    user_config.py             # user-overridable settings
  sound/
    engine.py                  # central audio synthesis engine
  model/
    query.py                   # ALM query facade (local/OpenRouter/DashScope)
    dispatcher.py              # provider-aware request dispatch
    cost.py                    # API usage/cost tracking
  experiments/
    run.py                     # pitchbench CLI
    scripts/                   # one .py per experiment (32 total)
    helpers/
      cat_a.py ... cat_f.py    # per-category generate + evaluate helpers
      audit.py                 # condition/result audit utilities
      data.py                  # Parquet I/O
      music.py                 # pitch/notation conversion helpers
      plots.py                 # experiment plotting helpers
      results.py               # result writers, aggregators, summary CSVs
      sampling.py              # stratified sampling
      setup.py                 # experiment setup helpers
      timing_layout.py         # timing-grid utilities
  analysis/
    analyze.py                 # core analysis pipeline
    a1.py                      # A1 line plots + heatmaps
    ablation.py                # ablation summaries
    combine.py                 # combine multi-run CSV outputs
    overview.py                # overview plots/tables
    run_analysis.py            # batch analysis CLI
    replay.py                  # canonical 28-task raw-answer replay
    table1.py                  # Table 1 exports from verified evidence
    publication.py             # generate manuscript Figures 1–4
    fonts/                     # bundled publication fonts and license
  baselines/
    evaluation.py              # frozen-dataset baseline runner
    runtime.py                 # DSP and Basic Pitch adapters
data/
  preloaded/                   # background recordings (gitignored)
  generated/                   # created by `pitchbench generate`
configs/
  paper_baselines.yaml         # reusable baseline recipe
  baselines-requirements.txt    # separate pinned ONNX inference environment
paper/
  evaluation/                  # compressed saved ALM answers (replay inputs)
  analysis/                    # separate diagnostic sample and numeric evidence
  figures-and-tables/          # publication tables and A1 figures
results/
  table1-recomputed/           # full-precision paper scores and provenance

Set PITCHBENCH_ROOT to override the project root for data/ and results/.

uv run pytest

Tests cover CLI resolution, Parquet I/O, result aggregation, sequence scoring, baseline adapters, Table 1 consistency, sampling, and API routing. No network calls or audio generation required.

@misc{dujardin2026pitchbench,
  title         = {PitchBench: Measuring Pitch Hearing in Audio-Language Models},
  author        = {Milan Liessens Dujardin and Song-Ze Yu and Craver Corbyn Thomas-Smith and David M. Chan and Karina Nguyen},
  year          = {2026},
  eprint        = {2605.26176},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2605.26176},
}

MIT — see LICENSE.