Build a gold set
Take queries from your agent’s logs, in the words your agent sends. A query you write by hand for the test tells you less about your agent than one it sent last week. Each line ofgold.jsonl is one query:
gold.jsonl
Mix two kinds of query. A known-item query has an answer page, so you list it in
gold_urls and score recall with no judge. An open query, such as news, has many good answers, so an LLM judge rates the results.
Run the harness
This harness sends every query ingold.jsonl to Keenable with fixed settings and writes each result to keenable.results.jsonl. Install the SDK with pip install keenable or npm install keenable, then run python eval_keenable.py gold.jsonl or npx tsx eval-keenable.ts gold.jsonl:
gold_urls, then the median latency:
Example output · results vary
run function for each provider you compare. Keep K, the text budget and the query text the same, and write each provider’s results to its own file.
These settings keep the comparison fair:
In Python, pass
mode="realtime" or disable_cards=True. In TypeScript, pass mode: "realtime" or disableCards: true; both need TypeScript SDK 0.1.5 or later, so check npm view keenable version first. See Search for every parameter.
Judge the results
Open queries need a judge. Give an LLM one result at a time with this prompt, and fill the placeholders from the results file. Whenpublished_at is null, write unknown:
Judge prompt
These are the same gains as NEEDLE, the open benchmark we run. NEEDLE also penalizes redundant results, so its nDCG can come out lower; see
src/needle/shared/metrics.py. For the ideal ranking, pool every provider’s results for the query, keep each URL once with its best rating and sort by rating. A provider that returns the best results the whole field found then scores 1.
- Remove provider names from what the judge sees, and shuffle the order of providers.
- Judge each distinct URL, title and snippet once per query, and reuse that rating when another provider returns the same result.
- Use the same judge model and prompt for every provider.
- Read a sample of judgements yourself before you trust the totals.
Common pitfalls
Next steps
- Benchmarks: Keenable’s speed on Artificial Analysis and the open NEEDLE data.
- Search API reference: every request parameter and response field.
- Migrate to Keenable: map another provider’s parameters to Keenable’s.
- Errors: what each status code means and how to fix it.