Knowledge Base

Crawl a Documentation Site into an Upstash Vector RAG Index

A crawl-to-structure-to-store pipeline that recursively crawls an entire docs site into clean markdown with Firecrawl, chunks it, and upserts embeddings into Upstash Vector for serverless retrieval.

What This Builds

This recipe turns any documentation website into a searchable knowledge base. Firecrawl’s crawl endpoint recursively walks a site and returns each page as clean, LLM-ready markdown (code blocks and structure preserved, navigation and boilerplate stripped). Those pages are chunked, embedded, and written into Upstash Vector, a serverless vector database that bills per request and needs no servers to run.

Once indexed, a small query function embeds the user’s question, queries Upstash Vector for the nearest chunks, and feeds them to an LLM. Because the whole thing is serverless, it pairs naturally with a scheduled re-crawl to keep the index fresh as the docs change.

The Stack

  • Firecrawl provides /crawl to recursively fetch a whole site and /scrape for single pages, returning markdown or structured JSON. It handles JavaScript rendering and pagination so you do not hand-roll a crawler.
  • Upstash Vector is the store. It is HTTP/REST-based and serverless, so it works from edge functions and scheduled workers without connection pooling. It can also embed text for you, or accept your own vectors.
  • OpenAI (or any embedding model) generates embeddings and the final grounded answer.

Step-by-Step Outline

  1. Create a Firecrawl API key and an Upstash Vector index (choose dimensions to match your embedding model, or use Upstash’s built-in embedding).
  2. Kick off a crawl: call Firecrawl crawl with the docs root URL and a page limit. Poll until it completes; you receive an array of pages, each with markdown and source metadata.
  3. Chunk each page’s markdown (e.g. by headings or a fixed token window) and attach metadata: source URL, title, and a stable id derived from the URL + chunk index.
  4. Embed each chunk and upsert into Upstash Vector, keyed by that stable id so a later re-crawl overwrites rather than duplicates.
  5. To query, embed the question, call Upstash Vector query with topK, then pass the returned chunks plus their source URLs to the LLM for a cited answer.
  6. Optionally wrap steps 2-4 in a scheduled job (cron) that re-crawls periodically and re-upserts; stable ids make the refresh idempotent.

Why This Shape Works

Firecrawl removes the two worst parts of building a docs index: rendering JS-heavy pages and cleaning markup into usable text. Upstash Vector removes the operational tax of running a vector DB for a low-traffic internal tool. Keying upserts by a URL-derived id means the same crawl can run on a schedule without bloating the index, giving you a self-updating knowledge base for cents.

Source

Based on Firecrawl’s crawl-endpoint guide and the Upstash Vector docs/examples for RAG. Firecrawl: https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl — Upstash Vector examples: https://upstash.com/docs/vector/examples