NEEDLE: Which search API moves the needle for AI agents?

We make each query set fresh from public sources: News each hour, the other verticals each day.

For Scholar we sample papers from arXiv (computer, physical, life, and social sciences) and Europe PMC (health sciences). Four age bands reach from the last week back to approximately five years. Each paper gives one query per slice, so the four Scholar slices rank engines on the same papers. The gaps between the slices are the point: title vs body isolates full-text indexing, body vs clue isolates keyword-vs-prose handling, and tot isolates semantic retrieval.

We leak-check each LLM-written Scholar query: at least two content words must not appear in the title or the abstract. We keep a paper only if each slice produced a valid query. The known-item and tip-of-the-tongue framing follows Exa's "SOTA Search Over Academic Publications" (Jul 2026).

The AgenticRare stream merges search queries from three public agent-trajectory releases: the anonymized DeepResearchGym logs (14M+ search requests from real agentic clients; "Agentic Search in the Wild", SIGIR '26), the OpenResearcher deep-research trajectories (TIGER-AI-Lab, 2026), and LRAT-Train ("Learning to Retrieve from Agent Trajectories", SIGIR '26).

We send each engine in a run the same query text. We pin the per-engine request shapes in git (Appendix F). Each engine has one request in flight at a time, two on Scholar, so the latency percentiles are comparable across engines. On News, AgenticRare and Scholar the runner queries all engines in parallel; on Finance and Legal it queries them one after another. We do not time each engine's connection warm-up, the requests that open its first connections in a run: they pay one-time connection setup and, in a parallel run, every engine starting at once. The latency figures therefore show performance on repeated requests. A later request that has to reconnect is timed, and reports count it in latency.reconnects. Engines see the live web minutes apart in a run, not at the same instant.

We judge an engine on its own ranking, titles, and snippets. We do not fetch pages and we do not re-rank. We cap the evidence at the same snippet length for each engine, so engines that return more content get no extra signal to the judge or the answer matchers. We judge each distinct combination of URL, title, snippet, and date once per query. We use that rating again where the identical result occurs for other engines (Appendix E gives the full judging setup).

A failed search scores zero when the engine's side caused it (timeout, server error, content filter, a rejected query); an agent gets nothing back either. We remove a failed search from the mean only when our own account or plan caused it (HTTP 401, 402, 403 or 429, a plan limit, or a message about our credits, quota, billing or API key). We pace each engine to its rate limit and retry a 429 with exponential backoff. We also remove failed judgements. Reports list search_errors, split into account_errors and service_errors, and judge_errors.

We make fresh News queries each hour and new known-answer queries each day from public sources (Appendix B), so there is no fixed benchmark set to overfit. We archive the exact queries with each run (Appendix G).

Each vertical also reports ultimate, a synthetic engine. It pools all engines' results per query and ranks them oracle-style: best judge rating first on the judged verticals, and the gold item first on the known-answer verticals. Its score shows what one engine could get if it always returned the best results that the full field found. This separates "the engine ranked poorly" from "no engine found better results". We do not include ultimate in the overlap and uniqueness statistics.

We judge with openai/gpt-6-luna via OpenRouter, high reasoning effort, temperature 0, one pointwise call per document. We record the model for each run as judge_model in its archived report. The judgement browser shows it when you select a run, so judge changes stay visible in the history.

The judge does not see the engine identity. The prompt contains the query, the document title, the URL, the published date, and the snippet — not the engine name. We judge each distinct combination of URL, title, snippet, and date once per query and use that rating again where the identical result occurs for other engines. We judge the same URL with a different snippet separately, with the same blind judge and rubric.

A judge preference for a snippet style would be a shared bias across engines. We publish the per-result reasoning with each run, so you can audit this.

Rubric: 0–4 Needs Met scale (FailsM / SM / HM / FullyM) with hard caps for a wrong entity, stale coverage of dated events, and search-operator compliance ("quoted phrases", site:). The full prompt is one template in the repo: src/needle/shared/prompts/judgement.jinja.

Known limitation: we operate NEEDLE ourselves, and we did not yet do a systematic human-agreement audit of the judge. But you can check the numbers: the judge is blind and shared across engines, each rating includes its reasoning, and we publish the queries, prompts, and raw results. Thus anyone can score any rating again.