Large language models know a great deal, but not your policies, products or latest data. Retrieval-augmented generation (RAG) bridges that gap by searching a knowledge source at query time and passing the most relevant passages to the model along with the question.
How the pipeline works
- Ingest — documents are cleaned, split into chunks and enriched with metadata.
- Index — chunks are embedded as vectors and stored, often alongside a keyword index.
- Retrieve — the user's question is used to find the most relevant chunks.
- Generate — the model answers using the retrieved context, ideally with citations.
Design choices that matter
Chunking
Chunks that are too small lose context; chunks that are too large dilute relevance. Splitting along the document's natural structure — sections and headings — usually beats fixed character counts.
Hybrid search
Combining semantic vector search with keyword search catches both paraphrased questions and exact terms such as product codes or policy numbers.
Re-ranking
A second-stage re-ranker can reorder candidate passages for relevance, improving precision without enlarging the context.
Key takeaways
- RAG retrieves relevant documents and supplies them to the model as context.
- Retrieval quality usually matters more than model choice.
- Chunking, metadata and hybrid search are high-leverage design decisions.
- Evaluate retrieval and generation separately.
