The idea
You rarely embed a whole document. A 40-page manual as one vector matches everything vaguely and nothing precisely. Chunking splits it into passages, usually a few hundred tokens each, and each chunk gets its own embedding.
Common strategies: fixed size with overlap (simple, robust), sentence or paragraph boundaries (keeps thoughts whole), and structure-aware splitting by headings or sections (best for docs, wikis and code). Overlap of 10–20% stops answers being cut in half at a boundary.
Store metadata with every chunk: document ID, title, section heading, page number, URL. You need it for filtering and citations. The right chunk size depends on your questions; test two or three sizes against your eval set.
Chunking quality caps retrieval quality. Bad chunks cannot be fixed by a better model later in the pipeline.
Picture it
Words to know
- ChunkA passage of a document stored and retrieved as one unit.
- OverlapText repeated at the end of one chunk and start of the next.
Try it without code
Chunk a page by hand
- Take a long help-centre article.
- Split it into chunks so each one answers one question.
- Write the question each chunk answers. Chunks that answer none are too small or too mixed.
Use it in code
A heading-aware markdown chunker with size limit and overlap.
import re
def chunk_markdown(text, max_chars=1500, overlap=200):
sections = re.split(r"(?m)^(?=#{1,3} )", text)
chunks = []
for sec in sections:
heading = sec.splitlines()[0] if sec.strip() else ""
start = 0
while start < len(sec):
piece = sec[start:start + max_chars]
chunks.append({"heading": heading.strip("# "), "text": piece})
start += max_chars - overlap
return [c for c in chunks if c["text"].strip()]
for c in chunk_markdown(open("README.md").read())[:3]:
print(c["heading"], "|", len(c["text"]), "chars")Chunk the same document at 500, 1,000 and 2,000 characters and compare which gives the best top-3 results for 10 questions.
Common mistakes
- Splitting tables and code blocks in half.
- Dropping headings, so a chunk loses its context.
- No metadata, so you cannot cite.