Speed on Artificial Analysis
From the Artificial Analysis Search Index, published 18 August 2026:- Fastest search call on the Artificial Analysis Search Index: 0.34 s.
- Lowest time per agent task on that board (15.1 s), about 1 s more than running with no search at all.
realtime mode. See the Cost table on Search for the two modes. Source: Announcing the Artificial Analysis Search Index.
NEEDLE
We publish NEEDLE, the open benchmark we run, live against 20+ search APIs; anyone can re-score the raw results. Keenable runs NEEDLE, so check any NEEDLE number against the raw data below. The code is MIT licensed and the results data is CC BY 4.0. Every engine in a run gets the same query text. NEEDLE scores the ranking, titles and snippets each engine returns. It doesn’t fetch pages or re-rank, the judge doesn’t see engine names, and every engine’s evidence is capped at the same snippet length. Queries are generated fresh from public sources, so there is no fixed set to tune for. The NEEDLE dashboard has the full comparison protocol and the current results.Verticals
Engines
On 19 November 2026, the NEEDLE dashboard, for the open benchmark we run, listed 25 engines. These are their CLI ids, which--engines takes, with the dashboard name where it differs: keenable-realtime (shown as Keenable), perplexity, perplexity-fast, exa, exa-lite, brave-llmcontext, parallel (shown as parallel-advanced), context (shown as context.dev), brave, parallel-fast, you, linkup, parallel-turbo, tinyfish, deepseek, claude-search, kagi, octen, chatgpt-search, firecrawl, bing (shown as bing/searchapi), google (shown as google/serper), tavily, jina, seltz.
A vendor with more than one mode or endpoint appears once per mode, such as exa and exa-lite. The dashboard’s Keenable row is keenable-realtime, Keenable in realtime mode. The NEEDLE CLI also has keenable, which runs pro mode, and engines the dashboard doesn’t show. The NEEDLE README lists each engine’s API key variable.
Public data
Everything behind a NEEDLE number is public:Re-score a run
Run Keenable again on the exact queries of a published run, then compare your report with the one in the run folder. You need uv and git. This re-runs Keenable on the exact queries of the Legal run of 19 November 2026, 12:44 UTC:Terminal
legal.json, a report with recall@5 and MRR@5 for keenable-realtime, broken down by suite (case law or regulation code), query syntax and court. Compare it with the published legal.json in the run folder, or with legal/keenable-realtime.json for that engine alone. Your numbers can differ, because the live web changes between runs.
- Without
KEENABLE_API_KEY, the CLI of NEEDLE, the open benchmark we run, calls Keenable’s keyless endpoint, which allows 1,000 requests per hour per IP address. See Rate limits. - To add other engines, pass them to
--enginesand set each engine’s API key. The NEEDLE README lists the variables. - Legal and Scholar score by ID match, so re-running them on a published query file needs no LLM key. News and AgenticRare use an LLM judge, so re-running them needs
OPENROUTER_API_KEY. So does generating new queries for most verticals. - To re-score without searching again, read each engine’s per-query results from the run folder and apply the metric code in
src/needle/shared/metrics.py.
Next steps
- Evaluate search APIs: build a gold set and compare providers with the same settings.
- Search: parameters, modes and cost.
- Migrate to Keenable: map another provider’s parameters to Keenable’s.