Skip to main content
Fetch backend From version 1.5.0, cognee can fetch URLs through Keenable, so pages enter its graph memory as clean Markdown. Pass a URL to remember() or add(), and cognee asks Keenable for the page as Markdown instead of crawling the HTML itself. The rest of the pipeline (chunking, embeddings, entity extraction, recall) doesn’t change. → View the change on GitHub · cognee’s URL-ingestion guide

​
Install

​
Install cognee

The Keenable backend is part of cognee, so there’s no Keenable package to add:

​
Set the key

cognee needs the key. It picks Keenable only if KEENABLE_API_KEY is set, and it calls the keyed /v1/fetch endpoint. If the key is missing, cognee uses a different backend instead of calling Keenable without a key. Create a key at app.keenable.ai/console.

​
Check the precedence

cognee chooses a fetch backend from the environment, in this order:
  1. Tavily, if TAVILY_API_KEY is set
  2. Keenable, if KEENABLE_API_KEY is set
  3. The built-in crawler
If both keys are set, cognee uses Tavily. To route URL ingestion through Keenable, unset TAVILY_API_KEY or select the backend explicitly.

​
Use

​
Give cognee a URL

Pass the URL to remember(), then query what cognee learned:
remember() fetches the page through Keenable, ingests the Markdown, and builds the graph. recall() answers from the graph without fetching the page again.

​
Fetch pages that are not indexed

By default Keenable returns its indexed copy of a page and rejects URLs it has not indexed. To fetch any URL, turn on live fetching:
With live fetching on, every request goes to the source page. Leave it off when you only ingest pages Keenable already indexes and want the faster cached path.

​
Select the backend explicitly

If you call cognee’s fetch layer directly, pass the backend as a parameter. The environment precedence doesn’t apply:

​
Narrow the page with an extraction prompt

KeenableConfig carries the per-call options. prompt makes Keenable return only the part of the page you ask for instead of the whole document:

​
Track a page over time

cognee’s scheduled scraper runs through the same fetch path (it needs apscheduler):
Each run writes WebPage nodes into the graph, linked is_part_of to a WebSite. The ScrapingJob links to the WebSite with is_scraping. Every page carries a SHA-256 content_hash, so the job can tell whether a page changed between runs. Unlike remember(), this path indexes the nodes directly and skips chunking and entity extraction.

​
Fetch behavior

  • cognee sends GET /v1/fetch?url=... with X-API-Key, plus live=true and prompt=... when set, and keeps the content field of the response.
  • Up to 5 URLs are in flight at once (KeenableConfig.concurrency), with a 30-second timeout per request (timeout, 1 to 120).
  • If a URL fails, cognee logs a warning, skips it and continues with the rest of the batch. If every URL fails, cognee raises the first error instead of returning an empty result, so a bad key shows up as an authentication error.
  • Warnings identify a failed URL by its position in the batch and the exception class, never by the URL, so credentials in query strings can’t leak into logs.

​
Configuration

The environment variables cover the memory path (remember(), add()). The KeenableConfig fields cover direct calls to fetch_page_content, web_scraper_task and cron_web_scraper_task, and override the environment for that call.