> ## Documentation Index
> Fetch the complete documentation index at: https://docs.keenable.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Keenable search and fetch work without an API key. To add them to code, use the official SDK for the project's language: https://docs.keenable.ai/sdks.md
> Read an API key from the KEENABLE_API_KEY environment variable. Never ask a user to paste a key into chat; send them to https://docs.keenable.ai/api-keys.md
> On 402 with an API key (code insufficient_credits), stop and ask a human to add credits; don't retry. On the pay-per-request paths a 402 is the payment offer: pay it and retry, see https://docs.keenable.ai/pay-per-request.md. On 429, wait the number of seconds in Retry-After, then retry.
> Every error, with its cause and fix: https://docs.keenable.ai/errors.md

# Keenable crawlers

> Keenable sends web requests as KeenableBot (index crawling) and Keenable-User (a page a user asked for). The user-agent strings, the published IP lists to verify them, and the robots.txt rules to allow or block KeenableBot.

If you run a website, this page tells you how to recognize requests from Keenable in your logs, check that they are real, and allow or block them.

Keenable.ai, Inc. identifies its requests with two user-agent tokens. Both point to the public crawler page at [keenable.ai/bot ↗](https://keenable.ai/bot), which is the reference for site owners.

## User agents

| Token | What it is used for |
| - | - |
| `KeenableBot` | Crawling. KeenableBot collects publicly available pages to build and maintain the Keenable search index. It also fetches your `robots.txt`. |
| `Keenable-User` | A request made on behalf of a person, or an agent acting for one, who asked for a specific page. For example, a [Fetch](https://docs.keenable.ai/products/fetch.md) call that reads the page live. Keenable-User does not crawl the web automatically. |

KeenableBot sends this `User-Agent` header:

```text theme={"system"}
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; KeenableBot/1.0; +https://keenable.ai/bot)
```

Keenable-User sends this `User-Agent` header:

```text theme={"system"}
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Keenable-User/1.0; +https://keenable.ai/bot)
```

Match on the token, `KeenableBot` or `Keenable-User`, not on the full string.

## Verify a request

Anyone can copy a user-agent header, so check the source address as well. Requests that carry a Keenable token come from a fixed set of IP addresses. Keenable publishes one list per token as JSON, in the same format as Googlebot's list:

| Token | IP list |
| - | - |
| `KeenableBot` | `https://app.keenable.ai/bot/keenablebot.json` |
| `Keenable-User` | `https://app.keenable.ai/bot/keenable-user.json` |

This command downloads the list for Keenable-User:

```bash theme={"system"}
curl -s https://app.keenable.ai/bot/keenable-user.json
```

You get a `creationTime` and a `prefixes` array. Each entry holds one `ipv4Prefix` in CIDR notation:

```json theme={"system"}
{
  "creationTime": "2026-10-09T12:04:36.000000",
  "prefixes": [
    {
      "ipv4Prefix": "34.31.116.125/32"
    },
    {
      "ipv4Prefix": "34.31.229.240/32"
    }
  ]
}
```

The output above is trimmed. If a request names a Keenable token and its source address is not in that token's list, the request is not from Keenable. Download the lists again on a schedule, because the addresses can change.

> [!NOTE]
>
> Keenable requests don't carry HTTP Message Signatures (Web Bot Auth) yet, so verify by IP address. Each token already has a key directory, at `https://keenablebot.keenable.ai/.well-known/http-message-signatures-directory` and `https://keenable-user.keenable.ai/.well-known/http-message-signatures-directory`.

## Allow or block KeenableBot

KeenableBot follows the Robots Exclusion Protocol (RFC 9309). Use the `KeenableBot` token in your `robots.txt`. The match is not case-sensitive. If a group names `KeenableBot`, KeenableBot uses that group and ignores the `User-agent: *` group.

This rule blocks KeenableBot from your whole site:

```text theme={"system"}
User-agent: KeenableBot
Disallow: /
```

Keenable stops indexing your site, so your pages no longer appear in Keenable search results.

This rule keeps KeenableBot out of one directory and allows the rest:

```text theme={"system"}
User-agent: KeenableBot
Disallow: /private/
```

To allow KeenableBot on a site that blocks other crawlers, add a `KeenableBot` group with `Allow: /`.

A new rule takes effect after KeenableBot next fetches your `robots.txt`.

## Block at your server or firewall

To refuse either agent at your web server, firewall or CDN, match the token in the `User-Agent` header, the addresses in that token's IP list, or both. To allow Keenable through a bot filter, allowlist the same addresses.

## What Keenable does with crawled pages

Keenable builds a search index from the pages KeenableBot collects. The index answers queries from AI agents and applications through [Search](https://docs.keenable.ai/products/search.md) and [Fetch](https://docs.keenable.ai/products/fetch.md). Results link to the source page.

## Contact

Email [abuse@keenable.ai](mailto:abuse@keenable.ai) to report excessive load, ask for a lower crawl rate, request removal of specific content, or report a crawler that claims to be Keenable. Include the hostname, the timestamps and the source addresses so Keenable can tell you whether the traffic is its own.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.