Pipeline

A RAG system has two phases. Indexing prepares your documents ahead of time, offline. Retrieval runs on every question. Keeping the two apart makes the rest of RAG easier to reason about, because most problems belong clearly to one phase or the other.

Indexing phase#

Indexing turns raw documents into searchable vectors:

1. Load your documents: files, pages, tickets, database rows.
2. Chunk each document into smaller pieces. See chunking.
3. Embed each chunk into a vector with an embedding model.
4. Store each vector in a vector database, together with the original chunk text and metadata such as source, title and access rules.

This runs once up front, and again whenever documents change. Store the chunk text next to the vector, because the vector is only for searching. The text is what goes into the prompt.

Retrieval phase#

Retrieval runs when a user asks a question:

1. Embed the question with the same embedding model used for indexing.
2. Search the vector database for the top k most similar chunks.
3. Build the prompt: the retrieved chunks plus the question.
4. Generate: the model reads the chunks and answers, citing its sources.

Step 1 must use the same model as indexing, or the vectors are not comparable, as covered in embeddings. Step 4 only works well with clear answer rules in the prompt.

Short line#

Indexing runs offline: load, chunk, embed and store chunks with their text and metadata. At question time: embed the question with the same model, retrieve the top k chunks, and prompt the model to answer from them with citations.

Key takeaways#

• Indexing is offline: load, chunk, embed, store. It reruns when documents change.
• Store the chunk text and metadata with each vector. The text goes into the prompt.
• Retrieval runs per question: embed, search top k, build the prompt, generate.
• Embed the question with the same model used to index the documents.