Chunking

A document is divided into passages, each embedded and stored separately. At query time, whole chunks come back — so a chunk is simultaneously the unit of search and the unit of context. The naive approach splits every fixed number of characters. Better approaches split on structure: paragraphs, sections, headings, or whatever boundary the document already has. As a starting point rather than a rule, chunks of roughly 200 to 800 tokens with 10 to 20 per cent overlap suit most prose; tune from there against your own retrieval results.

Overlap means repeating the tail of one chunk at the head of the next, so a sentence spanning a boundary appears whole in at least one of them.

It is the setting with the largest effect on retrieval quality and the one most often left at its default. Chunks that are too large dilute the embedding — a passage covering four topics matches all of them weakly and none strongly. Chunks that are too small lose the context that made them meaningful, and retrieve a sentence with no idea what it referred to.

When a retrieval system returns technically related but useless passages, chunking is the first thing to examine, ahead of the embedding model or the database.

Splitting on character count regardless of structure. Cutting a table in half, separating a heading from the section it introduces, or breaking mid-sentence produces chunks that read as nonsense to an embedding model and retrieve accordingly.

The second is keeping one strategy for every document type. A policy document, a chat log and an API reference have completely different natural units, and the same splitter will serve at most one of them well.