Blog

How to build a RAG chatbot for your business

What a RAG chatbot is, where it fits in a business, and the steps to build one that answers from your own documents and shows its sources.

A RAG chatbot answers questions from your own documents and data instead of from whatever a language model happened to learn in training. RAG stands for retrieval-augmented generation: for each question, the system first retrieves the most relevant passages from your content, then asks the model to answer using only those passages, with links back to the sources. For a business, that means a chatbot that can answer “what is our refund policy for annual plans?” correctly, show where the answer came from, and say so when it does not know. This guide covers how to build one that people can rely on: choosing the first use case, preparing the content, the build steps, testing, security and cost.

What a RAG chatbot is, in plain terms

Every RAG chatbot has the same parts, whatever tools it is built with:

  • Sources. The documents, web pages, help-desk articles, wikis or database records it is allowed to answer from.
  • Ingestion. A pipeline that reads those sources, extracts the text, splits it into passages (chunks) and keeps them up to date.
  • Index. A searchable store of the chunks, usually a vector index for meaning plus keyword search for exact terms, with metadata such as source, date and who may see it.
  • Retrieval. For each question, finding the handful of chunks most likely to contain the answer.
  • Generation. A language model writes the answer from the retrieved chunks and cites them.
  • Interface. Where people ask: a website widget, Slack or Microsoft Teams, a help-desk tool or your own application.

RAG is different from training or fine-tuning a model on your documents. Nothing about the model changes; the knowledge stays in your index, so you can update it the same day a policy changes, remove a document, and restrict who sees what. For answering from business content, RAG is almost always the right starting point.

Good first use cases

The best first project has a clear audience, a body of content that already exists, and questions people ask again and again.

Use caseTypical sourcesWatch out for
Internal help desk for HR, IT or finance policiesPolicy documents, intranet pages, IT runbooksOld versions of policies; country-specific rules
Customer support on your websiteHelp-center articles, product docs, FAQsQuestions about a customer’s own account, which need a hand-off or a system lookup
Sales and product knowledgeProduct sheets, pricing rules, past proposalsConfidential pricing reaching the wrong people
Operations and proceduresSOPs, manuals, maintenance guides, scanned PDFsTables and diagrams that parse badly
Onboarding new staffHandbooks, team wikis, recorded process notesContent nobody owns or keeps current

Start with one audience. A chatbot for employees and a chatbot for customers have different sources, different permissions and a different cost of a wrong answer, and building both at once doubles the testing.

When RAG is not the right tool

  • The answer needs a calculation over live data. “What were sales in the north region last month?” is a database query, not a document search. Give the chatbot a tool that runs a safe, predefined query instead.
  • The content is small. If everything fits in a few pages, putting it straight into the prompt can be simpler and just as good.
  • The user wants something done, not answered. Raising a ticket, changing a booking or updating a record is the job of an AI agent with permissions and approvals. RAG can be one of its tools.
  • The documents contradict each other and nobody owns them. The chatbot will repeat the confusion faster. Fix ownership first.

How to build a RAG chatbot, step by step

  1. Collect real questions. Gather 50 to 100 questions people actually ask, from support tickets, chat history or the team that answers them today, and write down the correct answer and its source for each. This becomes your test set, and it is the most useful thing you will build.
  2. Choose and clean the sources. List every source the chatbot may use, who owns it, and how often it changes. Remove duplicates and superseded versions before indexing, not after users find them.
  3. Parse and chunk the content. Extract text with its structure intact: headings, lists and tables with their headers. Split on the document’s own sections rather than every fixed number of words, and keep the document and section title with each chunk. Check a sample of parsed output by eye, especially PDFs and scans.
  4. Index with metadata. Store each chunk with its source link, date, version, product or region, and access rules. You will filter on these later, and adding them afterwards means re-indexing everything.
  5. Set up retrieval. Combine vector search with keyword search so product codes, names and acronyms are found, then rerank the top results for the specific question. Rewrite follow-up questions such as “and for contractors?” into standalone queries before searching.
  6. Write the answer instructions. Tell the model to answer only from the retrieved passages, cite them, keep to the length and tone you want, and say plainly when the answer is not in the sources. Tie citations to chunk IDs and turn them into links in your own code, so they cannot be invented.
  7. Put it where people already ask. A chatbot in the tool people use every day gets used; a new portal often does not. Add a way to reach a person, and pass the conversation along so nobody repeats themselves.
  8. Test before launch. Run the question set, check whether the right passages were retrieved and whether each answer is supported by them, and include questions it should refuse and content some users should not see.
  9. Launch small and watch it. Start with one team or a share of website visitors. Log every question with the retrieved passages and the answer, collect thumbs-up and thumbs-down, and review the failures each week.

Steps 2 to 5 are where most of the effort goes, and where most chatbots that disappoint went wrong. Our post on what breaks in RAG in production goes deeper into chunking, retrieval, freshness and permissions.

Choosing the components

There is no single right stack. These are the decisions to make, and what to consider for each:

ComponentCommon optionsWhat decides it
Document parsingOpen-source parsers, cloud document AI services, OCR for scansHow many PDFs, tables and scanned pages you have
Index and searchPostgres with pgvector, OpenSearch or Elasticsearch, a managed vector databaseWhat you already run, data volume, and whether you need keyword and vector search together
Embedding modelHosted embedding APIs or open models you run yourselfLanguages, data residency, cost of re-indexing
Language modelHosted models from the major providers, or open models in your own cloudAnswer quality on your test set, cost per answer, where data may go
OrchestrationA framework such as LangChain or LlamaIndex, or plain codeTeam familiarity; plain code is often easier to debug
InterfaceWebsite widget, Slack or Teams app, help-desk integration, your own appWhere the audience already works
Monitoring and evaluationTracing tools, evaluation libraries such as Ragas, your own dashboardsWhether you can see what was retrieved for every answer

Pick the language model last, by running your question set against two or three candidates. With good retrieval, the differences are usually smaller than people expect, and the cheaper model often wins on cost per answer.

Security and privacy with business data

  • Apply permissions at retrieval. Store who may read each chunk and filter the search by the user’s identity and groups. Hiding content after the model has seen it is too late.
  • Know where the data goes. Check your model provider’s terms on data retention and training, and the region it processes data in. Some businesses need everything inside their own cloud account.
  • Keep personal data out unless it is needed. Redact or exclude it at ingestion, and set a retention period for chat logs.
  • Treat documents as untrusted input. A web page or uploaded file can contain text written to manipulate the model. Retrieved content should never be able to trigger actions or override the chatbot’s instructions.
  • Log for audit. Record the question, the user, the retrieved sources and the answer, so you can explain any response later.

What a RAG chatbot costs

As with any AI system, there is a build cost and a running cost. The build cost depends mostly on the content and the integrations: how many sources, how messy the documents are, whether permissions have to be synced from other systems, and how many channels the chatbot appears in. The model is a small part of it.

The running cost is mainly model usage per answer, plus hosting the index and re-embedding content when it changes. A rough estimate, with illustrative prices: 20,000 questions a month, each sending about 4,000 tokens of question and retrieved passages and getting about 400 tokens back, at $3 per million input tokens and $15 per million output tokens, comes to 80 million input tokens ($240) and 8 million output tokens ($120), about $360 a month. Retrieving fewer, better passages lowers this directly. Our post on AI development costs walks through the full calculation and the costs that catch people out.

Build, buy or adapt

Chatbot builders that let you upload documents and embed a widget are the fastest way to try the idea, and fine for a small public FAQ. They fall short when you need permissions from your own systems, many source types, data kept in your cloud, or control over how answers are tested. A custom build gives you all of that at a higher up-front cost. In between is adapting an existing solution: our ChatData chatbot already answers from documents, URLs and application data, and we adapt it to a client’s sources, permissions and channels rather than starting from nothing.

Launch checklist

  • A test set of real questions with known answers and sources, including ones it should refuse.
  • Every source has an owner, and superseded versions are removed from the index.
  • Parsed output checked by eye for each document type.
  • Keyword and vector search combined, with reranking, measured on the test set.
  • Answers cite sources as links, and say clearly when the answer is not available.
  • Permissions applied at retrieval and tested with users of different roles.
  • A route to a person, with the conversation passed along.
  • Questions, retrieved passages and answers logged, with a weekly review of failures.
  • Index updates automated, with an alert when ingestion fails.
  • Cost per answer and response time tracked from the first day.

Conclusion

A useful RAG chatbot for a business is mostly a content and retrieval project with a language model at the end. Start with one audience and their real questions, clean and structure the sources, make retrieval good and measurable, require citations, and test before anyone relies on the answers. The model choice matters less than any of that, and it is the easiest part to change later.

We build knowledge chatbots as part of our generative AI development work, and our quality engineering team tests AI answers before they reach users. If you want to try the idea with real users first, see MVP development.

Where to go from here

Planning a chatbot on your own documents?

Tell us who will use it and what it should answer from, and we will tell you what we would look at first.