Area 02 · RAG · Concept 03 of 20

Chunking Strategies

Split documents into pieces (by size, sentence or heading) small enough to retrieve precisely.

01–10 · The basePhase 2 · Weeks 7–10≈ 1.5 hours

The idea

You rarely embed a whole document. A 40-page manual as one vector matches everything vaguely and nothing precisely. Chunking splits it into passages, usually a few hundred tokens each, and each chunk gets its own embedding.

Common strategies: fixed size with overlap (simple, robust), sentence or paragraph boundaries (keeps thoughts whole), and structure-aware splitting by headings or sections (best for docs, wikis and code). Overlap of 10–20% stops answers being cut in half at a boundary.

Store metadata with every chunk: document ID, title, section heading, page number, URL. You need it for filtering and citations. The right chunk size depends on your questions; test two or three sizes against your eval set.

Why it matters

Chunking quality caps retrieval quality. Bad chunks cannot be fixed by a better model later in the pipeline.

Picture it

Words to know

  • ChunkA passage of a document stored and retrieved as one unit.
  • OverlapText repeated at the end of one chunk and start of the next.

Try it without code

Chunk a page by hand

  1. Take a long help-centre article.
  2. Split it into chunks so each one answers one question.
  3. Write the question each chunk answers. Chunks that answer none are too small or too mixed.

Use it in code

A heading-aware markdown chunker with size limit and overlap.

python
import re

def chunk_markdown(text, max_chars=1500, overlap=200):
    sections = re.split(r"(?m)^(?=#{1,3} )", text)
    chunks = []
    for sec in sections:
        heading = sec.splitlines()[0] if sec.strip() else ""
        start = 0
        while start < len(sec):
            piece = sec[start:start + max_chars]
            chunks.append({"heading": heading.strip("# "), "text": piece})
            start += max_chars - overlap
    return [c for c in chunks if c["text"].strip()]

for c in chunk_markdown(open("README.md").read())[:3]:
    print(c["heading"], "|", len(c["text"]), "chars")
PRACTICE · TICK IT AFTER THIS

Chunk the same document at 500, 1,000 and 2,000 characters and compare which gives the best top-3 results for 10 questions.

Common mistakes

  • Splitting tables and code blocks in half.
  • Dropping headings, so a chunk loses its context.
  • No metadata, so you cannot cite.

Go deeper