A RAG chatbot answers questions from your own documents and data instead of from whatever a language model happened to learn in training. RAG stands for retrieval-augmented generation: for each question, the system first retrieves the most relevant passages from your content, then asks the model to answer using only those passages, with links back to the sources. For a business, that means a chatbot that can answer “what is our refund policy for annual plans?” correctly, show where the answer came from, and say so when it does not know. This guide covers how to build one that people can rely on: choosing the first use case, preparing the content, the build steps, testing, security and cost.
What a RAG chatbot is, in plain terms
Every RAG chatbot has the same parts, whatever tools it is built with:
- Sources. The documents, web pages, help-desk articles, wikis or database records it is allowed to answer from.
- Ingestion. A pipeline that reads those sources, extracts the text, splits it into passages (chunks) and keeps them up to date.
- Index. A searchable store of the chunks, usually a vector index for meaning plus keyword search for exact terms, with metadata such as source, date and who may see it.
- Retrieval. For each question, finding the handful of chunks most likely to contain the answer.
- Generation. A language model writes the answer from the retrieved chunks and cites them.
- Interface. Where people ask: a website widget, Slack or Microsoft Teams, a help-desk tool or your own application.
RAG is different from training or fine-tuning a model on your documents. Nothing about the model changes; the knowledge stays in your index, so you can update it the same day a policy changes, remove a document, and restrict who sees what. For answering from business content, RAG is almost always the right starting point.
Good first use cases
The best first project has a clear audience, a body of content that already exists, and questions people ask again and again.
| Use case | Typical sources | Watch out for |
|---|---|---|
| Internal help desk for HR, IT or finance policies | Policy documents, intranet pages, IT runbooks | Old versions of policies; country-specific rules |
| Customer support on your website | Help-center articles, product docs, FAQs | Questions about a customer’s own account, which need a hand-off or a system lookup |
| Sales and product knowledge | Product sheets, pricing rules, past proposals | Confidential pricing reaching the wrong people |
| Operations and procedures | SOPs, manuals, maintenance guides, scanned PDFs | Tables and diagrams that parse badly |
| Onboarding new staff | Handbooks, team wikis, recorded process notes | Content nobody owns or keeps current |
Start with one audience. A chatbot for employees and a chatbot for customers have different sources, different permissions and a different cost of a wrong answer, and building both at once doubles the testing.
When RAG is not the right tool
- The answer needs a calculation over live data. “What were sales in the north region last month?” is a database query, not a document search. Give the chatbot a tool that runs a safe, predefined query instead.
- The content is small. If everything fits in a few pages, putting it straight into the prompt can be simpler and just as good.
- The user wants something done, not answered. Raising a ticket, changing a booking or updating a record is the job of an AI agent with permissions and approvals. RAG can be one of its tools.
- The documents contradict each other and nobody owns them. The chatbot will repeat the confusion faster. Fix ownership first.
How to build a RAG chatbot, step by step
- Collect real questions. Gather 50 to 100 questions people actually ask, from support tickets, chat history or the team that answers them today, and write down the correct answer and its source for each. This becomes your test set, and it is the most useful thing you will build.
- Choose and clean the sources. List every source the chatbot may use, who owns it, and how often it changes. Remove duplicates and superseded versions before indexing, not after users find them.
- Parse and chunk the content. Extract text with its structure intact: headings, lists and tables with their headers. Split on the document’s own sections rather than every fixed number of words, and keep the document and section title with each chunk. Check a sample of parsed output by eye, especially PDFs and scans.
- Index with metadata. Store each chunk with its source link, date, version, product or region, and access rules. You will filter on these later, and adding them afterwards means re-indexing everything.
- Set up retrieval. Combine vector search with keyword search so product codes, names and acronyms are found, then rerank the top results for the specific question. Rewrite follow-up questions such as “and for contractors?” into standalone queries before searching.
- Write the answer instructions. Tell the model to answer only from the retrieved passages, cite them, keep to the length and tone you want, and say plainly when the answer is not in the sources. Tie citations to chunk IDs and turn them into links in your own code, so they cannot be invented.
- Put it where people already ask. A chatbot in the tool people use every day gets used; a new portal often does not. Add a way to reach a person, and pass the conversation along so nobody repeats themselves.
- Test before launch. Run the question set, check whether the right passages were retrieved and whether each answer is supported by them, and include questions it should refuse and content some users should not see.
- Launch small and watch it. Start with one team or a share of website visitors. Log every question with the retrieved passages and the answer, collect thumbs-up and thumbs-down, and review the failures each week.
Steps 2 to 5 are where most of the effort goes, and where most chatbots that disappoint went wrong. Our post on what breaks in RAG in production goes deeper into chunking, retrieval, freshness and permissions.
Choosing the components
There is no single right stack. These are the decisions to make, and what to consider for each:
| Component | Common options | What decides it |
|---|---|---|
| Document parsing | Open-source parsers, cloud document AI services, OCR for scans | How many PDFs, tables and scanned pages you have |
| Index and search | Postgres with pgvector, OpenSearch or Elasticsearch, a managed vector database | What you already run, data volume, and whether you need keyword and vector search together |
| Embedding model | Hosted embedding APIs or open models you run yourself | Languages, data residency, cost of re-indexing |
| Language model | Hosted models from the major providers, or open models in your own cloud | Answer quality on your test set, cost per answer, where data may go |
| Orchestration | A framework such as LangChain or LlamaIndex, or plain code | Team familiarity; plain code is often easier to debug |
| Interface | Website widget, Slack or Teams app, help-desk integration, your own app | Where the audience already works |
| Monitoring and evaluation | Tracing tools, evaluation libraries such as Ragas, your own dashboards | Whether you can see what was retrieved for every answer |
Pick the language model last, by running your question set against two or three candidates. With good retrieval, the differences are usually smaller than people expect, and the cheaper model often wins on cost per answer.
Security and privacy with business data
- Apply permissions at retrieval. Store who may read each chunk and filter the search by the user’s identity and groups. Hiding content after the model has seen it is too late.
- Know where the data goes. Check your model provider’s terms on data retention and training, and the region it processes data in. Some businesses need everything inside their own cloud account.
- Keep personal data out unless it is needed. Redact or exclude it at ingestion, and set a retention period for chat logs.
- Treat documents as untrusted input. A web page or uploaded file can contain text written to manipulate the model. Retrieved content should never be able to trigger actions or override the chatbot’s instructions.
- Log for audit. Record the question, the user, the retrieved sources and the answer, so you can explain any response later.
What a RAG chatbot costs
As with any AI system, there is a build cost and a running cost. The build cost depends mostly on the content and the integrations: how many sources, how messy the documents are, whether permissions have to be synced from other systems, and how many channels the chatbot appears in. The model is a small part of it.
The running cost is mainly model usage per answer, plus hosting the index and re-embedding content when it changes. A rough estimate, with illustrative prices: 20,000 questions a month, each sending about 4,000 tokens of question and retrieved passages and getting about 400 tokens back, at $3 per million input tokens and $15 per million output tokens, comes to 80 million input tokens ($240) and 8 million output tokens ($120), about $360 a month. Retrieving fewer, better passages lowers this directly. Our post on AI development costs walks through the full calculation and the costs that catch people out.
Build, buy or adapt
Chatbot builders that let you upload documents and embed a widget are the fastest way to try the idea, and fine for a small public FAQ. They fall short when you need permissions from your own systems, many source types, data kept in your cloud, or control over how answers are tested. A custom build gives you all of that at a higher up-front cost. In between is adapting an existing solution: our ChatData chatbot already answers from documents, URLs and application data, and we adapt it to a client’s sources, permissions and channels rather than starting from nothing.
Launch checklist
- A test set of real questions with known answers and sources, including ones it should refuse.
- Every source has an owner, and superseded versions are removed from the index.
- Parsed output checked by eye for each document type.
- Keyword and vector search combined, with reranking, measured on the test set.
- Answers cite sources as links, and say clearly when the answer is not available.
- Permissions applied at retrieval and tested with users of different roles.
- A route to a person, with the conversation passed along.
- Questions, retrieved passages and answers logged, with a weekly review of failures.
- Index updates automated, with an alert when ingestion fails.
- Cost per answer and response time tracked from the first day.
Conclusion
A useful RAG chatbot for a business is mostly a content and retrieval project with a language model at the end. Start with one audience and their real questions, clean and structure the sources, make retrieval good and measurable, require citations, and test before anyone relies on the answers. The model choice matters less than any of that, and it is the easiest part to change later.
We build knowledge chatbots as part of our generative AI development work, and our quality engineering team tests AI answers before they reach users. If you want to try the idea with real users first, see MVP development.