Chunking
How you split your documents decides what retrieval can find. It is the most underrated lever in the whole pipeline — and often the single biggest quality jump available.
Why it matters
Chunk too large and the answer is buried and diluted inside a wall of unrelated text, which weakens the retrieval signal and wastes context. Chunk too small and you sever the sentence from the context that gave it meaning, so retrieval finds a fragment that no longer answers anything. There is a sweet spot, and most bad RAG is simply sitting on the wrong side of it.
The strategies, from crude to careful
Fixed size
Splits every N tokens. Simple, but cuts mid-sentence and mid-rule — exactly the boundary-cut failure from Lesson 2.
Recursive
Sensible defaultSplits on structure first: paragraphs, then sentences, then words. Breaks at natural seams.
Semantic
Splits where the meaning shifts. Higher quality, slower, more machinery.
Document-aware
Splits on headers and sections — ideal for policies, wikis, and markdown.
Tip
Always overlap
Let adjacent chunks share a little text at their edges so a rule that straddles a boundary survives in at least one whole chunk. Overlap is cheap insurance against the boundary cut.
The fix you will see land
Re-chunk the Northwind handbook with a recursive splitter and overlap, then re-run the query that failed in Lesson 2 ("Can I work from home full time?"). The whole remote-work rule now lives in one chunk, retrieval returns it, and the answer is correct. Same question, same model, better chunking, right answer.
The Claude Code method here
Chunking is a decision your team will make once and want applied consistently across every project. That is a natural home for a skill with a bundled script. Prose instructions are weak for deterministic work like splitting text on precise rules, so the `chunking` skill carries an actual Python script in its folder, and the `SKILL.md` tells Claude to run it rather than reason token by token about where to split.
Claude runs the script without pulling it into context, so the operation is fast, deterministic, and identical every time. Prose for judgment, scripts for anything that must be exact and repeatable.
Watch out
Common mistakes
- Using fixed-size splits with zero overlap on policy documents.
- Chasing a better embedding model when chunking is the real problem.
- Letting the agent improvise splits instead of running your team's script.
Go deeper (optional)