Skip to main content
Fetch backend cognee turns what an agent reads into persistent, graph-backed memory. From cognee 1.5.0 it can fetch URLs through Keenable: pass a URL to remember() or add(), and cognee asks Keenable for the page as Markdown instead of crawling the HTML itself. The rest of the pipeline (chunking, embeddings, entity extraction, recall) does not change. → View the change on GitHub · cognee’s URL-ingestion guide

​
Install

​
No separate install

The backend lives in cognee itself (cognee/tasks/web_scraper/) and uses the httpx client cognee already ships, so there is no Keenable package to add:

​
Set the key

This is one of the few integrations where the key is not optional. cognee picks Keenable because KEENABLE_API_KEY is set, and it calls the keyed /v1/fetch endpoint, so a missing key means a different backend is chosen, not a keyless call. Create a key at keenable.ai/console.

​
Check the precedence

cognee chooses a fetch backend from the environment, in this order:
  1. Tavily, if TAVILY_API_KEY is set
  2. Keenable, if KEENABLE_API_KEY is set
  3. The built-in crawler
Tavily wins when both keys are present. If a Tavily key is already in your environment, unset it to route URL ingestion through Keenable, or select the backend explicitly (below).

​
Use

​
Give cognee a URL

remember() fetches the page through Keenable, ingests the Markdown, and builds the graph; recall() answers from what was learned, without touching the original HTML again.

​
Fetch pages that are not indexed

By default Keenable returns its indexed copy of a page and rejects URLs it has not indexed. For arbitrary URLs, turn on live fetching:
With live fetching on, every request goes to the source page. Leave it off when you only ingest pages Keenable already indexes and want the faster cached path.

​
Select the backend explicitly

When calling cognee’s fetch layer directly, the backend is a parameter and the environment precedence does not apply:

​
Narrow the page with an extraction prompt

KeenableConfig carries the per-call options. prompt makes Keenable return only the part of the page you ask for instead of the whole document:

​
Track a page over time

cognee’s scheduled scraper runs through the same fetch path (it needs apscheduler):
Each run writes WebPage nodes into the graph, linked is_part_of to a WebSite, which the ScrapingJob links to with is_scraping. Every page carries a SHA-256 content_hash, so the job can tell whether a page changed between runs. Unlike remember(), this path indexes the nodes directly and skips chunking and entity extraction; what the two share is the Keenable fetch.

​
How it behaves

  • Request. GET /v1/fetch?url=... with X-API-Key, plus live=true and prompt=... when set. cognee keeps the content field of the response.
  • Concurrency. Up to 5 URLs in flight at once (KeenableConfig.concurrency), 30-second timeout per request (timeout, 1 to 120).
  • Failures are isolated. A URL that fails is skipped with a warning and the rest of the batch continues. If every URL fails, the first error is raised instead of returning an empty result, so a bad key surfaces as an authentication error and not as a silent empty ingest.
  • Logs stay clean. Warnings identify a failed URL by its position in the batch and the exception class, never by the URL, so credentials in query strings cannot leak into logs.

​
Configuration

The environment variables cover the memory path (remember(), add()). The KeenableConfig fields cover direct calls to fetch_page_content, web_scraper_task and cron_web_scraper_task, and override the environment for that call.