Making AI answer from your own documents
Retrieval-augmented generation, explained for people who have to decide whether to fund it: what it is, where it breaks, and what good looks like.
Ask a general AI model about your refund policy and it will answer confidently, and probably wrongly. It has never seen your policy. Retrieval-augmented generation, usually shortened to RAG, fixes that by giving the model the right pages of your own documents before it answers.
How it works, in one paragraph
Your documents are split into passages and indexed so they can be searched by meaning, not just keywords. When someone asks a question, the system finds the handful of passages most likely to contain the answer, hands them to the model with the question, and asks it to answer from those passages only, citing them.
Where it breaks
- Retrieval misses: the answer exists, but the search didn't find it. The model then fills the gap with something plausible.
- Stale sources: the policy changed, the index didn't. The answer is confidently out of date.
- Messy documents: scanned PDFs, tables split across pages and duplicated versions confuse both search and model.
“When a RAG system gets something wrong, look at what it retrieved before you blame the model.”
What good looks like
A good system shows which passage each answer came from, so people can check it. It says "I don't have that" rather than guessing when nothing relevant is found. It refreshes its index whenever the source documents change. And it is tested against a list of real questions your team actually gets, with known good answers, every time anything changes.
Start with one well-kept set of documents, such as a product manual or an HR policy, rather than everything at once. Quality of sources matters more than quantity.
Written by the team at Bytelights, a product and AI studio in India.