> ## Documentation Index
> Fetch the complete documentation index at: https://docs.keenable.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Every page is listed in https://docs.keenable.ai/llms.txt. To pick a product for a job, read https://docs.keenable.ai/choose.md
> Setup steps for agents, covering the SDKs, the MCP server and the CLI: https://docs.keenable.ai/skill.md
> Keenable search and fetch work without an API key. To add them to code, use the official SDK for the project's language: https://docs.keenable.ai/sdks.md
> Without a key, call POST https://api.keenable.ai/v1/search/public or GET https://api.keenable.ai/v1/fetch/public with an X-Keenable-Title header that names your app.
> With a key, send it in the X-API-Key header and read it from the KEENABLE_API_KEY environment variable. Never ask a user to paste a key into chat. Every access method: https://docs.keenable.ai/auth.md
> To get an API key without the console, use device login: https://docs.keenable.ai/authentication.md#device-login
> MCP server: https://api.keenable.ai/mcp (no key, or X-API-Key; keyless Codex uses https://api.keenable.ai/mcp-public), or https://api.keenable.ai/mcp-oauth for OAuth sign-in. These docs are an MCP server at https://docs.keenable.ai/mcp and the OpenAPI spec is at https://docs.keenable.ai/api-reference/openapi.json
> Keenable searches and fetches public web pages. If a question needs no web information, skip the call.
> On 402 with an API key (code insufficient_credits), stop and ask a human to add credits; don't retry. On 429, wait the number of seconds in Retry-After, then retry. Before you ship, follow https://docs.keenable.ai/production.md
> Every error, with its cause and fix, is in https://docs.keenable.ai/errors.md and moving from another search API is covered in https://docs.keenable.ai/migrate.md
> Before you compare Keenable with another search API, read https://docs.keenable.ai/guides/evaluate-search-apis.md

# Evaluate search APIs on your own queries

> Compare web search APIs fairly: build a gold set, run the same queries with equal settings, judge results blind and avoid the pitfalls that skew scores.

Find out which search API fits your agent by running your own queries through each provider with the same settings. You build a gold set, run a harness that records every result, judge the results blind and compare the scores.

For public numbers, see [Benchmarks](https://docs.keenable.ai/benchmarks.md). Its NEEDLE section follows the same rules as this guide.

## Build a gold set

Take queries from your agent's logs, in the words your agent sends. A query you write by hand for the test tells you less about your agent than one it sent last week.

Each line of `gold.jsonl` is one query:

```json gold.jsonl theme={"theme":{"light":"github-light-high-contrast","dark":"dark-plus"}}
{"id": "q1", "query": "asyncio TaskGroup reference in the Python standard library docs", "gold_urls": ["https://docs.python.org/3/library/asyncio-task.html", "https://docs.python.org/3.12/library/asyncio-task.html"]}
{"id": "q2", "query": "RFC that defines HTTP semantics and status codes", "gold_urls": ["https://www.rfc-editor.org/rfc/rfc9110", "https://httpwg.org/specs/rfc9110.html"]}
{"id": "q3", "query": "latest news on EU AI Act enforcement", "published_after": "7d"}
```

| Field | Required | What it holds |
| - | - | - |
| `id` | Yes | A stable id, so you can join results from different runs and providers |
| `query` | Yes | The query text, sent unchanged to every provider |
| `gold_urls` | No | Every URL that answers the query. Queries with gold URLs are scored without a judge. |
| `published_after` | No | A recency bound for news queries, such as `7d`. Send the same bound to every provider. |

Mix two kinds of query. A known-item query has an answer page, so you list it in `gold_urls` and score recall with no judge. An open query, such as news, has many good answers, so an LLM judge rates the results.

## Run the harness

This harness sends every query in `gold.jsonl` to Keenable with fixed settings and writes each result to `keenable.results.jsonl`. Install the SDK with `pip install keenable` or `npm install keenable`, then run `python eval_keenable.py gold.jsonl` or `npx tsx eval-keenable.ts gold.jsonl`:

```typescript eval-keenable.ts theme={"theme":{"light":"github-light-high-contrast","dark":"dark-plus"}}
import { appendFileSync, readFileSync, writeFileSync } from "node:fs";
import { Keenable, KeenableRateLimitError } from "keenable";

const K = 10; // the same max_results for every provider
const SNIPPET_CHARS = 2000; // the same text budget per result for every provider

interface GoldItem {
  id: string;
  query: string;
  gold_urls?: string[];
  published_after?: string;
}

interface Row {
  id: string;
  query: string;
  latency_ms: number;
  results: { rank: number; url: string; title: string; snippet: string; published_at: string | null }[];
  first_hit?: number | null;
}

const client = new Keenable(); // reads KEENABLE_API_KEY; without it, keyless limits apply

function normalize(url: string): string {
  let out = url.split("#")[0].replace(/\/+$/, "").toLowerCase();
  for (const prefix of ["https://", "http://", "www."]) {
    if (out.startsWith(prefix)) out = out.slice(prefix.length);
  }
  return out;
}

async function run(item: GoldItem): Promise<Row> {
  const start = performance.now();
  const response = await client.search(item.query, {
    maxResults: K,
    snippetMaxLength: SNIPPET_CHARS,
    publishedAfter: item.published_after,
  });
  const latency_ms = Math.round(performance.now() - start);
  const results = response.results.map((r, i) => ({
    rank: i + 1,
    url: r.url,
    title: r.title,
    snippet: r.snippet,
    published_at: r.publishedAt ?? null,
  }));
  const row: Row = { id: item.id, query: item.query, latency_ms, results };
  if (item.gold_urls?.length) {
    const gold = new Set(item.gold_urls.map(normalize));
    row.first_hit = results.find((r) => gold.has(normalize(r.url)))?.rank ?? null;
  }
  return row;
}

async function main() {
  const goldSet: GoldItem[] = readFileSync(process.argv[2], "utf8")
    .split("\n")
    .filter((line) => line.trim())
    .map((line) => JSON.parse(line));
  await client.search(goldSet[0].query, { maxResults: 1 }); // warm-up, not timed

  const rows: Row[] = [];
  let dropped = 0;
  writeFileSync("keenable.results.jsonl", "");
  for (const item of goldSet) {
    try {
      const row = await run(item);
      rows.push(row);
      appendFileSync("keenable.results.jsonl", JSON.stringify(row) + "\n"); // saved as it goes
    } catch (err) {
      if (!(err instanceof KeenableRateLimitError)) throw err;
      dropped++; // a 429 is not a miss: leave the query out of the scores
    }
  }
  if (dropped) console.log(`dropped ${dropped} rate-limited queries`);

  const ranks = rows.filter((r) => r.first_hit !== undefined).map((r) => r.first_hit ?? null);
  if (ranks.length) {
    const recall5 = ranks.filter((r) => r !== null && r <= 5).length / ranks.length;
    const mrr = ranks.reduce<number>((sum, r) => sum + (r ? 1 / r : 0), 0) / ranks.length;
    console.log(`recall@5 ${recall5.toFixed(2)}`);
    console.log(`MRR@${K} ${mrr.toFixed(2)}`);
  }
  const latencies = rows.map((r) => r.latency_ms).sort((a, b) => a - b);
  const mid = Math.floor(latencies.length / 2);
  const p50 = latencies.length % 2 ? latencies[mid] : (latencies[mid - 1] + latencies[mid]) / 2;
  console.log(`latency p50 ${p50} ms over ${latencies.length} searches`);
}

main();
```

```python eval_keenable.py theme={"theme":{"light":"github-light-high-contrast","dark":"dark-plus"}}
import json
import statistics
import sys
import time

from keenable import Keenable, KeenableRateLimitError

K = 10  # the same max_results for every provider
SNIPPET_CHARS = 2000  # the same text budget per result for every provider

client = Keenable()  # reads KEENABLE_API_KEY; without it, keyless limits apply


def normalize(url):
    url = url.split("#")[0].rstrip("/").lower()
    for prefix in ("https://", "http://", "www."):
        url = url.removeprefix(prefix)
    return url


def run(item):
    start = time.perf_counter()
    response = client.search(
        item["query"],
        max_results=K,
        snippet_max_length=SNIPPET_CHARS,
        published_after=item.get("published_after"),
    )
    latency_ms = round((time.perf_counter() - start) * 1000)
    results = [
        {"rank": i + 1, "url": r.url, "title": r.title, "snippet": r.snippet, "published_at": r.published_at}
        for i, r in enumerate(response.results)
    ]
    return latency_ms, results


def first_hit(results, gold_urls):
    gold = {normalize(u) for u in gold_urls}
    return next((r["rank"] for r in results if normalize(r["url"]) in gold), None)


with open(sys.argv[1]) as f:
    gold_set = [json.loads(line) for line in f if line.strip()]
client.search(gold_set[0]["query"], max_results=1)  # warm-up, not timed

latencies, ranks, dropped = [], [], 0
with open("keenable.results.jsonl", "w") as out:
    for item in gold_set:
        try:
            latency_ms, results = run(item)
        except KeenableRateLimitError:
            dropped += 1  # a 429 is not a miss: leave the query out of the scores
            continue
        latencies.append(latency_ms)
        row = {"id": item["id"], "query": item["query"], "latency_ms": latency_ms, "results": results}
        if item.get("gold_urls"):
            row["first_hit"] = first_hit(results, item["gold_urls"])
            ranks.append(row["first_hit"])
        out.write(json.dumps(row) + "\n")  # saved as it goes
        out.flush()

if dropped:
    print(f"dropped {dropped} rate-limited queries")

if ranks:
    print(f"recall@5 {sum(1 for r in ranks if r and r <= 5) / len(ranks):.2f}")
    print(f"MRR@{K} {sum(1 / r for r in ranks if r) / len(ranks):.2f}")
print(f"latency p50 {statistics.median(latencies)} ms over {len(latencies)} searches")
```

You see recall\@5 and MRR\@10 over the queries that have `gold_urls`, then the median latency:

```text Example output · results vary theme={"theme":{"light":"github-light-high-contrast","dark":"dark-plus"}}
recall@5 0.50
MRR@10 0.50
latency p50 247 ms over 3 searches
```

Both harnesses work with the published SDKs and run without an API key. Write the same `run` function for each provider you compare. Keep `K`, the text budget and the query text the same, and write each provider's results to its own file.

These settings keep the comparison fair:

| Setting | Keenable parameter | Keep it equal because |
| - | - | - |
| Results per query | `max_results`, 1 to 50, default 10 | A provider that returns more results gets more chances at a hit |
| Text per result | `snippet_max_length`, 180 to 10,000 characters | A judge rates a long snippet higher than a short one with the same page behind it |
| Recency | `published_after` | A news query without a bound returns older pages from every provider |
| Mode | `mode`: `pro` (default) or `realtime` | The modes trade depth for speed. Score each mode as its own entry. |
| Cards | `disable_cards: true` | Cards are results Keenable writes for prices, exchange rates or company facts. Turn them off to compare web pages only. |

In Python, pass `mode="realtime"` or `disable_cards=True`. In TypeScript, pass `mode: "realtime"` or `disableCards: true`; both need TypeScript SDK 0.1.5 or later, so check `npm view keenable version` first. See [Search](https://docs.keenable.ai/products/search.md#key-parameters) for every parameter.

## Judge the results

Open queries need a judge. Give an LLM one result at a time with this prompt, and fill the placeholders from the results file. When `published_at` is null, write `unknown`:

```text Judge prompt theme={"theme":{"light":"github-light-high-contrast","dark":"dark-plus"}}
You rate one web search result for one query. Today is {today}.

Query: {query}
Result title: {title}
Result URL: {url}
Published: {published_at}
Result text: {snippet}

Rate how well this result meets the need behind the query, using only the
title, URL and text above. Do not guess what the full page says.

4: fully meets the need; a user would stop here
3: highly useful; answers most of the need
2: somewhat useful; related but incomplete or partly outdated
1: barely related
0: unrelated, spam, broken or outdated for a time-sensitive query

Reply with JSON only: {"rating": <0-4>, "reason": "<one sentence>"}
```

Map each rating to a gain and score nDCG\@5 for each provider:

| Rating | Gain |
| - | - |
| 4 | 1.0 |
| 3 | 0.667 |
| 2 | 0.117 |
| 1 or 0 | 0 |

These are the same gains as NEEDLE, the open benchmark we run. NEEDLE also penalizes redundant results, so its nDCG can come out lower; see `src/needle/shared/metrics.py`. For the ideal ranking, pool every provider's results for the query, keep each URL once with its best rating and sort by rating. A provider that returns the best results the whole field found then scores 1.

* Remove provider names from what the judge sees, and shuffle the order of providers.
* Judge each distinct URL, title and snippet once per query, and reuse that rating when another provider returns the same result.
* Use the same judge model and prompt for every provider.
* Read a sample of judgements yourself before you trust the totals.

## Common pitfalls

| Pitfall | What goes wrong | Fix |
| - | - | - |
| Unequal `max_results` | Default result counts differ between providers, and the one returning more wins on recall | Set the same count on every provider. Keenable's default is 10. |
| No recency bound on news | Without a bound, news queries return older coverage and the judge marks it outdated | Send `published_after`, such as `7d`, on time-sensitive queries. It also leaves out pages with no known publish date. |
| `query_time` used for recency | `query_time` searches the web as it stood at that moment. It sets no lower bound, so newer pages are excluded and older ones stay. | Use `published_after` for recency |
| Reworded queries | Tuning the wording for one provider measures your rewrite, not the provider | Send the same query text to every provider. Keenable takes operators such as `site:` in `query`, so translate them only where a provider needs a parameter instead. |
| Keyless rate limits | Keenable allows 1,000 keyless requests per hour per IP address, and a large gold set reaches that cap. Further calls return `429` with `public_rate_limit_hourly`. | Send an [API key](https://docs.keenable.ai/api-keys.md): your organization gets 10 requests per second and no hourly cap. See [Rate limits](https://docs.keenable.ai/rate-limits.md). |
| Rate-limit errors scored as misses | A `429` from your own account counts as a zero for the provider, though the search never ran | Retry after `Retry-After`, or drop the query from that provider's mean and report how many you dropped, as the harness does. Score timeouts and server errors as zero. |
| Unequal text budget | A provider that returns more text per result gives the judge more evidence | Cap every provider's snippet at the same length before judging |
| Timing the first call | The first request pays for connection setup | Send a warm-up call first, as the harness does, and keep one request in flight per provider |
| One gold URL per query | The same page lives at several URLs, such as a versioned docs path, and a correct result counts as a miss | List every URL that answers the query in `gold_urls` |
| Reading `description` | On Keenable, `description` is deprecated and always empty | Read page text from `snippet`. See [Versioning](https://docs.keenable.ai/versioning.md). |

## Next steps

* [Benchmarks](https://docs.keenable.ai/benchmarks.md): Keenable's speed on Artificial Analysis and the open NEEDLE data.
* [Search API reference](https://docs.keenable.ai/api-reference/search.md): every request parameter and response field.
* [Migrate to Keenable](https://docs.keenable.ai/migrate.md): map another provider's parameters to Keenable's.
* [Errors](https://docs.keenable.ai/errors.md): what each status code means and how to fix it.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.