> ## Documentation Index
> Fetch the complete documentation index at: https://docs.keenable.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# cognee

> Keenable as the web-fetch backend behind cognee's URL ingestion and scheduled scraper: pages come back as clean Markdown before they enter graph memory.

*Fetch backend*

cognee turns what an agent reads into persistent, graph-backed memory. From cognee 1.5.0 it can fetch URLs through Keenable: pass a URL to `remember()` or `add()`, and cognee asks Keenable for the page as Markdown instead of crawling the HTML itself. The rest of the pipeline (chunking, embeddings, entity extraction, recall) does not change.

→ [View the change on GitHub](https://github.com/topoteretes/cognee/pull/4421) · [cognee's URL-ingestion guide](https://docs.cognee.ai/guides/web-url-ingestion)

## Install

### No separate install

The backend lives in cognee itself (`cognee/tasks/web_scraper/`) and uses the `httpx` client cognee already ships, so there is no Keenable package to add:

```bash theme={"system"}
pip install "cognee>=1.5.0"
```

### Set the key

```bash theme={"system"}
export KEENABLE_API_KEY="keen_..."
```

<Note>
  This is one of the few integrations where the key is not optional. cognee picks Keenable **because** `KEENABLE_API_KEY` is set, and it calls the keyed `/v1/fetch` endpoint, so a missing key means a different backend is chosen, not a keyless call. Create a key at [keenable.ai/console](https://keenable.ai/console).
</Note>

### Check the precedence

cognee chooses a fetch backend from the environment, in this order:

1. Tavily, if `TAVILY_API_KEY` is set
2. Keenable, if `KEENABLE_API_KEY` is set
3. The built-in crawler

Tavily wins when both keys are present. If a Tavily key is already in your environment, unset it to route URL ingestion through Keenable, or select the backend explicitly (below).

## Use

### Give cognee a URL

```python theme={"system"}
import cognee

await cognee.remember("https://docs.example.com/pricing")
await cognee.recall("what does the enterprise tier cost?")
```

`remember()` fetches the page through Keenable, ingests the Markdown, and builds the graph; `recall()` answers from what was learned, without touching the original HTML again.

### Fetch pages that are not indexed

By default Keenable returns its indexed copy of a page and rejects URLs it has not indexed. For arbitrary URLs, turn on live fetching:

```bash theme={"system"}
export KEENABLE_LIVE_FETCH="true"
```

With live fetching on, every request goes to the source page. Leave it off when you only ingest pages Keenable already indexes and want the faster cached path.

### Select the backend explicitly

When calling cognee's fetch layer directly, the backend is a parameter and the environment precedence does not apply:

```python theme={"system"}
from cognee.tasks.web_scraper import fetch_page_content

pages = await fetch_page_content(urls, preferred_tool="keenable")
```

### Narrow the page with an extraction prompt

`KeenableConfig` carries the per-call options. `prompt` makes Keenable return only the part of the page you ask for instead of the whole document:

```python theme={"system"}
from cognee.tasks.web_scraper import fetch_page_content
from cognee.tasks.web_scraper.config import KeenableConfig

cfg = KeenableConfig(live=True, prompt="Extract the pricing table and plan limits only.")
pages = await fetch_page_content(url, preferred_tool="keenable", keenable_config=cfg)
```

### Track a page over time

cognee's scheduled scraper runs through the same fetch path (it needs `apscheduler`):

```python theme={"system"}
from cognee.tasks.web_scraper import cron_web_scraper_task
from cognee.tasks.web_scraper.config import KeenableConfig

await cron_web_scraper_task(
    url=["https://docs.example.com/changelog"],
    schedule="0 6 * * *",
    job_name="daily_changelog",
    keenable_config=KeenableConfig(live=True),
    tavily_api_key=None,  # defaults to TAVILY_API_KEY, which would win
)
```

Each run writes `WebPage` nodes into the graph, linked `is_part_of` to a `WebSite`, which the `ScrapingJob` links to with `is_scraping`. Every page carries a SHA-256 `content_hash`, so the job can tell whether a page changed between runs. Unlike `remember()`, this path indexes the nodes directly and skips chunking and entity extraction; what the two share is the Keenable fetch.

## How it behaves

* **Request.** `GET /v1/fetch?url=...` with `X-API-Key`, plus `live=true` and `prompt=...` when set. cognee keeps the `content` field of the response.
* **Concurrency.** Up to 5 URLs in flight at once (`KeenableConfig.concurrency`), 30-second timeout per request (`timeout`, 1 to 120).
* **Failures are isolated.** A URL that fails is skipped with a warning and the rest of the batch continues. If every URL fails, the first error is raised instead of returning an empty result, so a bad key surfaces as an authentication error and not as a silent empty ingest.
* **Logs stay clean.** Warnings identify a failed URL by its position in the batch and the exception class, never by the URL, so credentials in query strings cannot leak into logs.

## Configuration

| Setting           | Environment variable  | `KeenableConfig` field | Default                   |
| ----------------- | --------------------- | ---------------------- | ------------------------- |
| API key           | `KEENABLE_API_KEY`    | `api_key`              | none (required)           |
| Base URL          | `KEENABLE_BASE_URL`   | `base_url`             | `https://api.keenable.ai` |
| Live fetch        | `KEENABLE_LIVE_FETCH` | `live`                 | `false`                   |
| Extraction prompt | —                     | `prompt`               | none (max 2000 chars)     |
| Concurrency       | —                     | `concurrency`          | `5`                       |
| Timeout (s)       | —                     | `timeout`              | `30`                      |

The environment variables cover the memory path (`remember()`, `add()`). The `KeenableConfig` fields cover direct calls to `fetch_page_content`, `web_scraper_task` and `cron_web_scraper_task`, and override the environment for that call.