The RAG pipeline

1. Chunk documents into overlapping segments
2. Embed chunks with a model (e.g., text-embedding-3-small)
3. Store vectors + metadata in a vector DB
4. Retrieve top-k by cosine similarity
5. Generate answer with LLM + retrieved context

Where it fails

| Stage | Common mistake | Fix | |-------|----------------|-----| | Chunking | Fixed 512-token chunks | Semantic chunking by heading/section | | Embedding | Generic model | Domain-adapted or fine-tuned | | Retrieval | Top-k only | Hybrid: BM25 + vector + rerank | | Grounding | No citations | Require [doc_id] in every claim |

Vector store evaluation

| Need | Choose | |------|--------| | < 1M vectors, Postgres already in stack | pgvector | | 1M–10M, need filtering + hybrid search | Qdrant | | 10M+, multi-tenant, managed | Pinecone / Weaviate Cloud | | Self-hosted, strict data residency | Qdrant / Milvus on Kubernetes |

Our default

For most B2B SaaS: pgvector (already have Postgres). Chunk by semantic boundaries, hybrid retrieve with BM25, rerank with a cross-encoder, cite every claim.