RAG Is a Search Problem First
When a retrieval-augmented assistant gives a bad answer, the model usually isn't the culprit. The search behind it is. Lessons from shipping RAG to real users.
Saji Nammari3 min read
Retrieval-augmented generation, or RAG, is the most common way to put an organization's own knowledge behind an AI assistant. The idea is simple: find the documents relevant to a question, hand them to the model, and let it answer from them.
When a RAG system answers badly, teams tend to reach for a bigger model or a longer prompt. In our experience the problem is usually upstream. The model was handed the wrong passages, or the right passages cut in the wrong places. RAG is a search problem first and a generation problem second.
Measure retrieval on its own
Before judging answers, judge what was retrieved. For a set of real questions, note which documents should come back, then check whether they do and how high they rank. If the right passage isn't in the top results, no prompt will save the answer.
Chunk for meaning, not for length
Splitting documents every few hundred tokens is the default, and it's often wrong. A policy split mid-sentence, or a table separated from its heading, loses its meaning. Split on the document's own structure, such as sections, headings, and list items, and keep the title and context with each chunk so it still makes sense on its own.
Combine keyword and semantic search
Vector search is good at meaning and bad at exact terms. Ask about a product code, an error number, or a person's name and embeddings can miss it entirely. Hybrid search, which combines keyword matching with vector similarity and then re-ranks the merged results, is more robust than either alone.
Keep the index fresh and permissioned
- Freshness: stale documents produce confidently outdated answers. Re-index when sources change, and store the source date with every chunk.
- Permissions: retrieval must respect who's asking. If a user can't open a document, the assistant shouldn't quote it to them. Filter at search time, not after generation.
Plan for questions the documents can't answer
Some questions have no answer in your knowledge base, and a RAG system should say so rather than improvise. Set a relevance threshold below which the assistant admits it doesn't know, and offer a next step: a link to search, a contact, or a way to flag the gap. Log those questions too. They're a ready-made list of the documentation your organization is missing.
Show your sources
Cite the passages behind every answer and link to them. Citations let users verify, they make errors easy to report, and they make an invented answer much easier to spot.
Get search right and a modest model will give good answers. Get it wrong and the best model available will give wrong ones, fluently.
About the author
Saji Nammari
Founder & Chief Executive Officer
Saji serves as the Founder and Chief Executive Officer, having successfully pivoted from five years as a highly effective freelance software and data professional. His leadership is rooted in the pragmatic, results-driven mentality required to deliver all types of projects, driving a culture of agility and technical rigor. Outside of work, Saji enjoys the creative disciplines of literature, music, and the satisfaction of pure problem solving.
