Building AI Agents That Survive Production · Part 17 of 44

Chunking Strategies That Survive Production

Fixed-size, semantic, and structure-aware chunking measured against real retrieval failures. The uncomfortable finding: your chunking choice barely moves whether the answer reaches the model, and hugely moves how much junk you pay for to get it there. Also, your chunk size is probably a library default nobody chose.

Your chunk size is a default you never chose

Open the ingestion code in most RAG systems and the chunk size is whatever the library shipped with. LangChain's TextSplitter defaults to 4,000 characters with 200 of overlap. LlamaIndex's SentenceSplitter defaults to 1,024 tokens with 200 tokens of overlap. Different documents, same vector store, and in most repositories neither number was chosen by anyone on the team.

It is worse than an untuned knob, because the two are not in the same unit. LangChain measures length with len() unless you hand it a tokenizer, so chunk_size=4000 is characters - roughly a thousand tokens of English. Teams compare their 4,000 against someone else's 512 and think they are eight times apart when they are closer to two.

There is no best chunk size. There is a trade, and most teams picked a side of it by accident at import time.

# The two defaults most pipelines inherit without reading

from langchain_text_splitters import RecursiveCharacterTextSplitter
RecursiveCharacterTextSplitter()        # chunk_size=4000 CHARACTERS, overlap=200

from llama_index.core.node_parser import SentenceSplitter
SentenceSplitter()                      # chunk_size=1024 TOKENS, overlap=200

# Same keyword, different unit. State the unit you mean:
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=200,          # now tokens
    chunk_overlap=0,
)

The trade, settled with numbers

A large chunk is more likely to contain the answer - that is recall. But its embedding averages everything inside it, so it matches queries vaguely, and you pay for all of it in context. A small chunk embeds sharply and wastes little, but the answer may straddle a boundary and never be retrieved at all.

Chroma's chunking evaluation put token-level numbers on this. Measuring intersection-over-union between retrieved tokens and the tokens that were actually relevant, 200-token recursive chunks scored 7.0; the widely copied 800-token-with-400-overlap configuration scored 1.4 - roughly five times as much of the payload wasted. Recall, meanwhile, spanned only eight to nine points across every chunker they tested.

Read that together and the conclusion is blunt. Chunking barely changes whether the answer reaches the model; it hugely changes how much junk rides along. If you are tuning chunking for accuracy you are mostly tuning cost - and cost is paid on every query, forever.

chunk  overlap   recall@10   token IoU   tokens/query
  150       0%        0.94        0.081          1,610
  250       0%        0.96        0.070          2,480
  250      10%        0.96        0.063          2,720
  400       0%        0.96        0.048          4,020
  800      50%        0.97        0.014         11,900   <- the default

recall barely moves; you pay 7x the context per query for +0.03

Fixed-size, done properly, is hard to beat

Recursive character splitting with a sensible separator hierarchy, at roughly 200 tokens, won no single column in that evaluation and was never bad in one either. It is the baseline every fancier strategy has to beat, and most do not.

The phrase doing the work is "done properly", and it means the separator list. Recursive splitting tries separators in order - paragraph, line, sentence, word - falling through only when a chunk is still too large. That default list is tuned for prose. Point it at Python, SQL, logs or CSV and it cuts inside a function body or a row, because nothing in the list is a boundary in that format.

from langchain_text_splitters import RecursiveCharacterTextSplitter

PROSE  = ["

", "
", ". ", " ", ""]
PYTHON = ["
class ", "
def ", "

", "
", " ", ""]
MARKUP = ["
## ", "
### ", "

", "
", " ", ""]

def splitter_for(kind: str, size: int = 200, overlap: int = 20):
    return RecursiveCharacterTextSplitter.from_tiktoken_encoder(
        encoding_name="cl100k_base",
        separators={"prose": PROSE, "python": PYTHON, "markup": MARKUP}[kind],
        chunk_size=size,
        chunk_overlap=overlap,
        keep_separator=True,       # do not silently eat the heading you split on
    )

chunks = splitter_for("markup").split_text(open("docs/auth.md").read())

Semantic chunking is not the upgrade it is sold as

The pitch is appealing: embed each sentence, measure the distance to the next, break where the distance spikes, and chunks follow topics instead of a character count.

Measured, the standard implementation was the worst performer in the study - 83.6 recall, against 91.9 for an LLM-driven chunker and 91.3 for a clustering variant at 400 tokens. A modified version of the same percentile approach reached 87.1. The family is not worthless; the version in every tutorial is its weakest member.

The break condition is a hyperparameter with no natural value - a percentile on a distance distribution that differs per corpus and per embedding model. And adjacent-sentence distance is noisy: one transitional sentence reads as a topic change and splits an argument in half. You also pay an embedding pass over every sentence at ingest, and pay it again each time you retune.

Structure-aware chunking is where the real wins are

Most documents already say where the boundaries are. Markdown has headings, HTML has sections, code has definitions, tables have rows, logs have entries, contracts have numbered clauses. Guessing boundaries from embedding distance while ignoring the ones written into the document is the strange choice.

Structure also marks the cuts that are unrecoverable. A chunk ending mid-row in a table, or mid-body in a function, is not a slightly worse chunk - no reranker and no larger context window repairs it, because the information needed to read it was deleted at ingest.

The second half is carrying structure forward. Prepend the heading path and store it as metadata. "API Reference > Authentication > Rotating keys" makes a 200-token chunk self-locating without making it bigger, and gives you the filters to narrow retrieval before searching.

import re, tiktoken
ENC = tiktoken.get_encoding("cl100k_base")

def chunk_markdown(md: str, max_tokens: int = 200):
    """Split on headings, size-cap within a section, keep the heading path."""
    path, buf, out = [], [], []

    def flush():
        body = "
".join(buf).strip()
        buf.clear()
        if body:
            out.append({"text": " > ".join(path) + "

" + body,
                        "section": " > ".join(path),
                        "depth": len(path)})

    for line in md.splitlines():
        m = re.match(r"^(#{1,6})s+(.*)", line)
        if m:
            flush()
            level = len(m.group(1))
            path[:] = path[: level - 1] + [m.group(2).strip()]
            continue
        buf.append(line)
        if len(ENC.encode("
".join(buf))) >= max_tokens:
            flush()
    flush()
    return out

Overlap is a patch, and it is not free

Overlap exists so an answer sitting on a boundary appears whole in at least one chunk. It works, and it is why the 800/400 configuration spread so far.

It also duplicates tokens into the index. They are embedded twice, stored twice, and when both neighbours are retrieved they occupy context twice - which is why removing overlap generally improved intersection-over-union in the evaluation. At 800/400, half of every chunk is a copy of the one beside it.

The counter-evidence matters too: with the small-context all-MiniLM-L6-v2, recall fell from 82.4 to 77.1 when overlap was removed. Overlap earns its keep when the chunk is large relative to what the embedding model can see. The rule that survives review - zero to fifteen percent of chunk size, zero once you prepend heading context, and never a number you cannot derive from the size you chose.

Contextualize the chunk instead of enlarging it

The real reason people reach for bigger chunks is that small ones lose their referents. "The service rejects requests above this limit" is useless when the chunk names neither the service nor the limit. Enlarging fixes that by brute force, and hands back the precision you were buying.

The cheaper fix: before embedding, have a small model write one or two sentences situating the chunk inside its document, and prepend them. Anthropic's published numbers are the strongest in the area - a 35% reduction in top-20 retrieval failure rate from contextual embeddings alone (5.7% to 3.7%), 49% with a contextual BM25 index alongside (to 2.9%), and 67% with reranking on top (to 1.9%). Ingest cost is about $1.02 per million document tokens, one-time, on their stated assumptions of 800-token chunks and prompt caching on the document.

This is the only technique here that bends the trade rather than sliding along it. The chunk stays small, so precision holds, while the embedding gets enough to be unambiguous, so recall rises too.

PROMPT = """Here is the whole document:
<document>{doc}</document>

Here is a chunk from it:
<chunk>{chunk}</chunk>

In 1-2 sentences, say what this chunk is about and where it sits in the
document, naming any entity the chunk refers to only by pronoun or shorthand.
Answer with the context only."""

def contextualize(doc: str, chunk: str, client) -> str:
    msg = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=120,
        messages=[{"role": "user", "content": [
            # cache the document: it is reused for every chunk of it
            {"type": "text", "text": PROMPT.format(doc=doc, chunk=chunk),
             "cache_control": {"type": "ephemeral"}},
        ]}],
    )
    return msg.content[0].text.strip() + "

" + chunk   # embed THIS

Measure retrieval, not chunks

Every number above came from someone else's corpus, so none of it applies unmeasured. The harness that settles it for yours is small: fifty real questions, each with the span of source text that answers it marked by hand. Not generated by the model that will later answer them, or you are measuring agreement with yourself.

Then two numbers per configuration - recall at k, meaning whether the gold span survived into the retrieved window, and token IoU, meaning how much of that window was the gold span rather than filler. Sweep 150, 250, 400 and 800 tokens against overlaps of zero and ten percent. That is the table from the second section, and it tells you where you stand before anyone argues about strategies.

It is also the interview answer. Asked how you would chunk a corpus, the weak reply names a strategy; the strong one names the trade and the measurement - small for precision, large for recall, and an eval with gold spans to decide where this corpus sits on that line.

The supporting points are the ones that show you ran it. Recursive splitting at a couple of hundred tokens is a hard baseline. Embedding-distance semantic chunking has measured worse than what it replaced. A chunk cut mid-row cannot be repaired downstream. Overlap should be derived, not inherited. And contextualizing a small chunk improves both sides at once.

Then the line that separates shipping from reading: the number that actually changed when you fixed chunking was cost per query, not accuracy.

PRACTISE IT · SYSTEM DESIGN LABS

Work the same ideas as a full interview case study, with pressure questions and model answers.

Keep reading

Related articles