Incrementally Sync a Source into a pgvector Index Without Re-Embedding

What This Builds

This recipe builds the maintenance layer that production RAG systems need: incremental indexing. Naively re-embedding a whole corpus on every sync is slow and expensive, and deleting-then-reinserting leaves windows where the index is empty. LangChain’s indexing API uses a RecordManager to track content hashes, so on each run it embeds only new or changed chunks, skips unchanged ones, and deletes vectors whose source content disappeared.

The store here is Supabase Postgres with the pgvector extension, so the vectors live next to your relational data and the record-manager bookkeeping table sits in the same database.

The Stack

• LangChain provides the index(...) function plus SQLRecordManager, which records each chunk’s hash and source. Cleanup modes (incremental / full) decide how stale vectors are removed.
• Supabase (Postgres + pgvector) is the vector store via LangChain’s PGVector/SupabaseVectorStore, and also hosts the record-manager table.
• OpenAI (or any embeddings model) embeds only the chunks that the record manager flags as new or changed.

Step-by-Step Outline

1. Enable the vector extension in Supabase and create the vector store table; point a SQLRecordManager at the same Postgres connection with a unique namespace.
2. record_manager.create_schema() once to set up the bookkeeping table.
3. In the sync job, load the source documents (files, a crawl, a DB export) and split them into chunks. Give each document a stable source in metadata — the record manager needs it for incremental cleanup.
4. Call index(docs, record_manager, vector_store, cleanup="incremental", source_id_key="source"). LangChain hashes each chunk, embeds only num_added / num_updated ones, skips the rest, and deletes vectors for chunks no longer present from each touched source.
5. Read the returned counts (num_added, num_updated, num_skipped, num_deleted) and log them so you can see drift over time.
6. Run the job on a schedule (cron / a scheduled function). Because indexing is hash-based and idempotent, repeated runs are cheap and safe.

Why This Shape Works

The expensive operations in RAG ingestion are embedding calls and write amplification. Hash-based incremental indexing makes a re-sync proportional to what actually changed, not to corpus size, and incremental cleanup keeps the index consistent without an empty window. Keeping both the vectors and the record-manager table in the same Postgres means the whole sync is one transaction-friendly system rather than two services to keep in step.

Source

Based on the LangChain indexing API docs and the walkthrough “Building a Production-Ready RAG System with Incremental Indexing” (uses SQLRecordManager with a pgvector store): https://pub.towardsai.net/building-a-production-ready-rag-system-with-incremental-indexing-ee42cfbfef7f — LangChain indexing API: https://python.langchain.com/docs/how_to/indexing/