epochs

Learn · Module 09 · Search and retrieval · Lesson 2 of 5

Chunking documents

Before you can search documents, you have to cut them into passages. Where you cut decides what your system can find.

4 minute read

In short
  • Small chunks are precise but can lose context; large chunks keep context but blur the match.
  • Overlap stops sentences being cut in half, at the cost of storing more.
  • Cut on the document's own structure, and make each chunk understandable when read alone.

In plain words

You can't embed a 200-page handbook as one vector. A single vector can't represent that many different topics, and you couldn't fit the handbook into a prompt anyway. So you split documents into passages, called chunks, and embed each one.

Chunk size is a trade. Small chunks are precise: each is about one thing, so its vector is a clear match for the right question. But they can lose the surrounding context that makes them understandable. Large chunks keep context, but mix topics, so their vectors are blurry and they use up more of the prompt.

Worked example: fixed-size chunks with overlap

The simplest method cuts every N tokens. Take a 1,000-token document and chunks of 200 tokens.

  1. 1
    No overlap

    Chunks start every 200 tokens. That gives 5 chunks.

    starts: 0, 200, 400, 600, 800
  2. 2
    The problem

    A sentence running from token 190 to token 210 is cut in half. Half is in chunk 1, half in chunk 2, and neither half makes sense alone.

    chunk 1: 0–199   chunk 2: 200–399
  3. 3
    Add an overlap of 50 tokens

    Each chunk now starts 150 tokens after the previous one, so neighbours share 50 tokens.

    starts: 0, 150, 300, 450, 600, 750, 900   (7 chunks)
  4. 4
    The sentence is whole again

    The second chunk now covers tokens 150 to 349, which includes all of 190 to 210.

    chunk 2: 150–349

Overlap has a price. Seven chunks instead of five means storing 1,300 tokens for a 1,000-token document, 30% more to embed, store and search.

Your turn

With 200-token chunks and a 50-token overlap, how far apart do chunks start?

Better places to cut

Cutting by token count ignores what the text means. Better methods cut where the author already did: at headings, paragraphs, list items, or functions in code. A common recipe is to split on the largest structure first, and only split further when a piece is still too big.

A chunk also needs to make sense when read alone. 'This limit doesn't apply to contractors' is useless without knowing which limit. A cheap fix is to add the document title and section heading to the start of every chunk before embedding it.

There's no best chunk size. It depends on your documents and your questions. Start with a few hundred tokens and a small overlap, then measure whether the right passages are being retrieved, and adjust.

Common mistakes

  • Splitting at a fixed character count that cuts sentences, tables or code blocks in half.
  • Choosing one chunk size without testing it against real questions.
  • Throwing away headings and titles, which carry context the chunk needs.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

How do you decide on a chunk size?

There's no universal right size, so I'd test it. The trade-off is that small chunks are precise but may lack context, while large chunks keep context but dilute the embedding and waste prompt space. I'd start by splitting on the document's natural structure, like sections and paragraphs, aiming for a few hundred tokens with a small overlap, and attach the document title and section heading to each chunk. Then I'd build a set of real questions with known source passages and measure hit rate and MRR for a few chunk sizes. The best size depends on the content: FAQs suit small chunks, while long policy documents often need bigger ones or a parent-child approach.

More to think about
  • What's the purpose of chunk overlap?
  • Retrieval finds the right document but the wrong part. What would you change?

Go further

Was this lesson clear?