Skip to main content
Find out which search API fits your agent by running your own queries through each provider with the same settings. You build a gold set, run a harness that records every result, judge the results blind and compare the scores. For public numbers, see Benchmarks. Its NEEDLE section follows the same rules as this guide.

Build a gold set

Take queries from your agent’s logs, in the words your agent sends. A query you write by hand for the test tells you less about your agent than one it sent last week. Each line of gold.jsonl is one query:
gold.jsonl
Mix two kinds of query. A known-item query has an answer page, so you list it in gold_urls and score recall with no judge. An open query, such as news, has many good answers, so an LLM judge rates the results.

Run the harness

This harness sends every query in gold.jsonl to Keenable with fixed settings and writes each result to keenable.results.jsonl. Install the SDK with pip install keenable or npm install keenable, then run python eval_keenable.py gold.jsonl or npx tsx eval-keenable.ts gold.jsonl:
You see recall@5 and MRR@10 over the queries that have gold_urls, then the median latency:
Example output · results vary
Both harnesses work with the published SDKs and run without an API key. Write the same run function for each provider you compare. Keep K, the text budget and the query text the same, and write each provider’s results to its own file. These settings keep the comparison fair: In Python, pass mode="realtime" or disable_cards=True. In TypeScript, pass mode: "realtime" or disableCards: true; both need TypeScript SDK 0.1.5 or later, so check npm view keenable version first. See Search for every parameter.

Judge the results

Open queries need a judge. Give an LLM one result at a time with this prompt, and fill the placeholders from the results file. When published_at is null, write unknown:
Judge prompt
Map each rating to a gain and score nDCG@5 for each provider: These are the same gains as NEEDLE, the open benchmark we run. NEEDLE also penalizes redundant results, so its nDCG can come out lower; see src/needle/shared/metrics.py. For the ideal ranking, pool every provider’s results for the query, keep each URL once with its best rating and sort by rating. A provider that returns the best results the whole field found then scores 1.
  • Remove provider names from what the judge sees, and shuffle the order of providers.
  • Judge each distinct URL, title and snippet once per query, and reuse that rating when another provider returns the same result.
  • Use the same judge model and prompt for every provider.
  • Read a sample of judgements yourself before you trust the totals.

Common pitfalls

Next steps

  • Benchmarks: Keenable’s speed on Artificial Analysis and the open NEEDLE data.
  • Search API reference: every request parameter and response field.
  • Migrate to Keenable: map another provider’s parameters to Keenable’s.
  • Errors: what each status code means and how to fix it.