Many teams evaluate web search APIs by comparing model answers to a static gold set. When the gold labels are published in a public dataset or already stored in the model’s parameters, the agent can simply fetch the answer or recall it from memory, making the test measure memorization rather than true retrieval ability. This creates an inflated score and hides real weaknesses in ranking, coverage, or latency.
NEEDLE solves this problem by treating the benchmark as a live, open‑source harness that rebuilds its query stream from fresh public sources instead of freezing a fixed list. News queries are regenerated every hour from RSS feeds and Google Trends; finance, scholar, legal, and rare‑entity queries are rebuilt daily from SEC XBRL, arXiv, Europe PMC, CourtListener, and public agent logs. Fifteen search APIs are run against the exact same query text under a single protocol: one request at a time, evidence clipped to 2 000 characters, and no page fetch or re‑ranking. Scores are judged blindly by an LLM that never sees the engine name.
Instead of ranking engines against each other, NEEDLE compares each result list to an empirical ceiling called ultimate—a pooled oracle that orders the combined returns of all participants. The gap to ultimate reveals whether a shortfall is due to poor ranking or a shared retrieval limitation across the market. Metrics differ by vertical: nDCG@5 for news and deep‑tail, answer‑recall@5 for finance, and identifier match for scholar and legal. Latency is reported as p50/p95 per call, reflecting the dozens of searches an agent typically issues.
The approach eliminates over‑fitting, surfaces real‑world performance, and provides a reproducible way to track progress as models and APIs evolve. #AI #SearchBenchmark #LLM #AgenticAI #NEEDLE #OpenSource