Retrieval-augmented generation (RAG) looks simple: split your documents into chunks, store them in a vector index, find the closest chunks for each question and ask a model to answer from them. A prototype built that way often impresses in a demo and then disappoints with real users. The model is rarely the main problem. Most failures come from what is retrieved, how current it is, who is allowed to see it, and whether anyone can tell when an answer is wrong. This post walks through what breaks in RAG in production and how to fix each part.
First, find out which half failed
A bad RAG answer has two possible causes: the right information was not retrieved, or it was retrieved and the model did not use it well. The fixes are completely different, so every investigation should start by looking at the retrieved context for the failing question. If the answer is not in the context, work on retrieval. If it is, work on the prompt, the model or the way context is presented. Logging the retrieved chunks with every answer makes this a two-minute check instead of a guess.
What breaks, and how to fix it
| Symptom | Likely cause | What to do |
|---|---|---|
| Answer misses information that is in the documents | Poor chunking or retrieval that misses exact terms | Structure-aware chunking, hybrid search, reranking |
| Confident answer from an outdated document | Stale index, old versions still searchable | Incremental re-indexing, remove superseded versions, date metadata |
| User sees content they should not | Permissions not applied at retrieval time | Store access rules with each chunk and filter in the query |
| Answer is plausible but not in any source | Model fills gaps from its own training | Instruct it to refuse when context is insufficient, check groundedness |
| Citation points to the wrong place | Citations generated as text, not tied to chunks | Cite chunk IDs and map them back to source links |
| Good on tables or PDFs in testing, bad in production | Parsing loses structure | Fix document parsing before tuning anything else |
| Slow or expensive answers | Too many or too large chunks in the prompt | Retrieve fewer, better chunks; measure tokens per answer |
Parsing and chunking
Retrieval cannot find what parsing destroyed. PDFs with columns, scanned pages, tables, and slides often come out of a basic text extractor as scrambled text. Check a sample of parsed documents by eye before building anything on top of them.
Chunking then decides what a single search result can contain. Fixed-size chunks are easy but cut through tables, lists and procedures, so the retrieved half lacks the context that makes it useful. Better defaults:
- Split on the document’s own structure: headings, sections, list items, table rows grouped with their header.
- Keep the section title and document title with each chunk, so a fragment like “the limit is 30 days” still says what the limit applies to.
- Use modest overlap between chunks for running prose.
- Store metadata you will filter on later: source, owner, date, version, product, region and access rules.
There is no universally right chunk size. Try two or three settings against your own test questions and keep the one that retrieves best.
Retrieval quality
Vector search is good at meaning and weak at exact identifiers: product codes, error numbers, names and acronyms. Real users search with exactly those. Hybrid search, combining keyword search (such as BM25) with vector search, covers both. A reranking step, where a cross-encoder or a model reorders the top candidates for the specific question, often improves the final few chunks more than any change to the embedding model.
Two more fixes that pay off:
- Query rewriting. In a conversation, “what about the second one?” means nothing to a search index. Rewrite follow-up questions into standalone queries before retrieving.
- Metadata filters. If the user is asking about one product or region, filter to it rather than hoping the similarity score does.
Measure retrieval on its own with a labelled set of questions and the chunks that should be found. Recall at k (was a right chunk in the top k?) and mean reciprocal rank (how high was the first right chunk?) are simple and enough to compare changes.
Freshness
An index built once is out of date the next day. Plan the ingestion pipeline as a production system, not a one-off script:
- Re-index on change, using source events or modified timestamps, rather than rebuilding everything on a schedule.
- Delete or demote superseded versions. Two versions of a policy in the index will eventually produce an answer from the wrong one.
- Track index lag (how old is the newest change not yet searchable?) and alert when ingestion fails.
- Show the document date with the answer, so users can judge for themselves.
Permissions
A RAG system that answers from internal documents must respect who is allowed to read them. Filtering after generation is too late, because the model has already seen the restricted text. Apply access control at retrieval time: store access rules with each chunk, pass the user’s identity and groups into the query, and filter before ranking. Keep permissions in sync with the source system, test with users of different roles, and include “should not be able to see this” cases in the test set. Treat retrieved content as untrusted input too: a document can contain instructions aimed at the model, so it should not be able to trigger actions or override the system prompt.
Citations and refusals
Citations let users check answers and let you debug them. Give each chunk an ID in the prompt, ask the model to cite IDs for each claim, and map those IDs back to the source link and section in your own code. Then verify that cited chunks were actually retrieved, and drop or flag citations that were not. Just as important, tell the model what to do when the context does not contain the answer, and test that it does: a clear “I could not find this in the documents” is far better than a fluent guess.
Evaluating a RAG system
Evaluate retrieval and generation separately, then end to end. Useful measures are retrieval recall and ranking on labelled questions, faithfulness (is every claim supported by the retrieved context?), answer relevance, and correct refusal on unanswerable questions. Open-source tools such as Ragas provide metrics for these, typically using a model as the judge, so the same care applies as with any LLM-as-judge setup: narrow rubrics, calibration against human labels and a pinned judge version. The wider practice of test sets, CI gates and production monitoring is covered in how to evaluate LLM applications in production, and the basics of checking model output in QA in evaluating LLM output.
RAG production readiness checklist
- Parsed output checked by eye for each document type, including tables and scanned files.
- Structure-aware chunks carrying titles, dates, versions and access metadata.
- Hybrid search and a reranking step, compared against plain vector search on your own questions.
- Follow-up questions rewritten into standalone queries.
- Incremental ingestion with deletion of superseded versions, and an alert on index lag.
- Access control applied at retrieval time and tested per role.
- Citations tied to chunk IDs and verified in code.
- Defined refusal behaviour, tested with unanswerable questions.
- Retrieved context logged with every answer.
- Separate retrieval and generation metrics, run on every change.
- Tokens, latency and cost per answer tracked by feature.
Conclusion
Most RAG problems in production are data and retrieval problems that look like model problems. Log what was retrieved, measure retrieval separately, keep the index current and permission-aware, and make answers traceable to their sources. Do that and the choice of model becomes one of the easier decisions.
Knowledge and retrieval systems are a core part of how we build AI products, and ChatData in our AI Studio is a chatbot that answers from the documents, URLs and application data you connect. If you need the answers tested before users rely on them, our quality engineering team evaluates AI output as part of delivery.