epochs

Learn · Module 09 · Search and retrieval · Lesson 1 of 5

RAG in one picture

A model doesn't know your documents. Retrieval-augmented generation looks things up first, then lets the model answer from what it found.

5 minute read

In short
  • RAG retrieves relevant passages first, then asks the model to answer from them.
  • Indexing is chunk, embed, store; answering is retrieve, generate.
  • When RAG is wrong, first find out whether retrieval or generation failed.

In plain words

A language model knows only what was in its training data, up to a cutoff date. It has never seen your company's policies, yesterday's tickets or a customer's contract. Ask anyway, and it may answer confidently and wrongly.

Retrieval-augmented generation, or RAG, fixes this the way an open-book exam does. Before the model answers, a search step finds the most relevant passages and puts them in the prompt. The model's job changes from remembering to reading.

It has two phases. Indexing happens once, ahead of time. Answering happens for every question.

phasestepwhat happens
IndexingChunkSplit documents into passages
IndexingEmbedTurn each passage into a vector
IndexingStoreSave vectors in a vector database
AnsweringRetrieveEmbed the question, find the closest passages
AnsweringGenerateGive the passages and question to the model

Worked example: a leave-policy question

A company handbook has been split into passages and indexed. An employee asks: 'How many days of leave can I carry over?'

  1. 1
    Embed the question and score every passage

    Cosine similarity between the question vector and each passage vector.

    0.81  "Up to 10 unused leave days carry over to the next year..."
    0.64  "Leave requests need manager approval 5 days ahead..."
    0.22  "The office is closed on national holidays..."
  2. 2
    Keep the top k

    With k = 2, the first two passages go into the prompt. The third is dropped.

  3. 3
    Build the prompt

    Instructions, then the passages, then the question.

    Answer using only the passages below. If they don't contain the answer, say you don't know. Cite the passage you used.
    
    [1] Up to 10 unused leave days carry over...
    [2] Leave requests need manager approval...
    
    Question: How many days of leave can I carry over?
  4. 4
    Generate

    The model reads the passages and answers from them.

    You can carry over up to 10 unused leave days to the next year. [1]

Count the cost too. If each passage is about 200 tokens and you retrieve three, the prompt is roughly 3 × 200 + 20 for the question + 80 for instructions = 700 tokens. Retrieving more passages raises the chance of including the right one, and also raises cost and distraction.

Your turn

A RAG system gives a wrong answer, although the correct passage exists in the database. What should you check first?

Why not just fine-tune, or paste everything in?

Fine-tuning teaches style and skills well, but it's a poor way to store facts: it's slow to update, and the model can't point to where an answer came from. RAG updates the moment you add a document, and can cite its source.

Pasting every document into a large context window works for small collections. It gets expensive and slow as they grow, and models pay less attention to details buried in very long prompts. RAG keeps the prompt small and relevant.

Common mistakes

  • Blaming the model for bad answers without checking whether retrieval found the right passages.
  • Skipping citations, so users and reviewers can't verify answers.
  • Using RAG for questions that need reasoning over the whole dataset, like 'how many orders last month', which a database query answers better.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

Explain RAG and when you'd use it.

RAG, retrieval-augmented generation, answers questions using your own documents. You split documents into chunks, embed them, and store them in a search index. When a question comes in, you retrieve the most relevant chunks, often with hybrid keyword and vector search plus a reranker, and put them in the prompt with instructions to answer only from them and cite sources. I'd use it when answers depend on private or frequently changing information, like company policies or product docs, and when traceability matters. It's cheaper to keep current than fine-tuning, since you just update documents. The main thing to get right is retrieval quality, so I'd evaluate retrieval and answers separately.

More to think about
  • RAG or fine-tuning: how do you choose?
  • Your RAG system hallucinates. How do you debug it?

Go further

Was this lesson clear?