Almost every disappointing RAG system is disappointing for the same reason, and it is not the model. The right document never made it into the prompt.
By OneShopAI · · 5 min read
The answers are vague. They are confidently wrong about details in your own documentation. They cite the wrong section, or they answer from general knowledge when the documents say something more specific. Every one of those looks like the model underperforming, and the natural response is to try a bigger one, rewrite the system prompt, or turn the temperature down.
None of that helps, because the model is answering correctly from what it was given. It was given the wrong three chunks.
A retrieval-augmented system is two systems in a trench coat, and they fail differently. Test them apart. Take fifty real questions and, for each, note which chunk actually contains the answer. Then ask only one thing of your retrieval step: did that chunk come back in the top results?
That number is your ceiling. If the right chunk is retrieved sixty per cent of the time, no model, prompt or temperature setting can produce a system that is right more than sixty per cent of the time. Teams routinely spend weeks on the generation half of the system while the retrieval half sits at that number, unmeasured.
When retrieval underperforms, the reflex is to blame the vector database or the embedding model. Occasionally that is right. Far more often it is how the documents were split. A chunk cut mid-sentence embeds a fragment of an idea. A chunk stripped of the heading it sat under loses the thing that made it findable — "Rate limits" at the top of the section is often the only place the topic is named. A chunk that is too large buries one useful line under six irrelevant paragraphs and drags the embedding towards the average of all of them.
Before changing infrastructure, read twenty of your chunks as text. If you cannot tell what a chunk is about by reading it cold, neither can a retrieval system.
Embeddings capture meaning, which is precisely why they are unreliable on identifiers. Product codes, error numbers, version strings, surnames and rare acronyms have little semantic content — the thing that makes them useful is that they match exactly. A pure vector search will happily return the conceptually similar error instead of the one you asked about.
Hybrid search, combining keyword matching with vector similarity, consistently beats either alone on real corpora. If your users search by identifier at all, this is not an optimisation.
When retrieval returns nothing relevant, most systems answer anyway from the model's general knowledge, in the same confident register as a grounded answer. That is the worst possible failure mode: indistinguishable from success, and specifically wrong about your own product.
Instruct explicitly that "the documents provided do not cover this" is a valid answer, then check that it is ever actually given. A system that has never once said it does not know is not a system that always knows.