In short. A RAG chatbot does not “know” anything: for every question it fetches the right passages from your documents, then copies them into the model prompt. Everything happens in retrieval, not in the model. The economic consequence nobody spells out: indexing 500 pages costs less than one cent, but sending the retrieved passages back on every question costs 0.08 to 0.77 cents per answer depending on the model. And legally, the French regulator is blunt: the knowledge base you plug in is yours to answer for.
RAG has been everywhere for two years, and most pages explaining it stop at the three-box diagram. You come away knowing it stands for Retrieval-Augmented Generation, without knowing what it costs, where it breaks, or what you are allowed to feed it.
This article takes the problem from the other end. It walks the full pipeline, prices every line item with public rates recorded on 5 August 2026, cites a primary regulatory source on the subject, and embeds a chunking tester that shows you, live, the mistake that produces the most wrong answers in production.
What a RAG chatbot actually is
The CNIL, France’s data protection authority and a GDPR supervisory authority, gives an official wording in its questions and answers on generative AI dated 18 July 2024: “Retrieval Augmented Generation (RAG) consists in integrating an information retrieval mechanism into a vectorised database (or embedding). It produces answers enriched by external data, potentially more specific and easier to update than the model itself.” (our translation from the French original.)
In plain terms: the language model does not memorise your prices, your procedures or your returns policy. On every question, a search engine picks the three or five closest passages from your documents and pastes them into the prompt just before the question. The model answers from what it has just read, not from what it learned during training.
The term comes from a Meta AI research paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and eleven co-authors, submitted to arXiv on 22 May 2020 (reference 2005.11401) and presented at NeurIPS that same year. Six years on, it is the default way every serious business chatbot works.
A RAG chatbot is an internal search engine with a writer behind it. If the search misses, the writer produces a beautifully phrased wrong answer. That is why 90 % of the work sits in retrieval, not in the choice of model.
The six pipeline stages, and what you decide at each one
Every stage carries a setting that someone has to choose. On a no-code platform those settings are taken for you by default. If you build the pipeline yourself, they are all yours, including the ones you did not know existed.
| Stage | What happens | The setting that matters | What breaks if you get it wrong |
|---|---|---|---|
| 1. Ingestion | Your pages, PDFs, files and FAQs are collected and converted to text | What you feed in, and how often you refresh it | The bot quotes a procedure that was withdrawn six months ago |
| 2. Chunking | The text is cut into fixed-size pieces (the chunks) | Chunk size and overlap between two chunks | The answer is cut in half and no chunk contains it |
| 3. Embedding | Each chunk becomes a list of numbers (the embedding) | The embedding model, and the language it works in | An English-only model misses synonyms in your language |
| 4. Retrieval | The question is embedded too, and the closest chunks are searched | How many chunks come back, and the similarity threshold | Too few: incomplete answer. Too many: the model drowns |
| 5. Generation | Chunks and question go to the model with instructions | The instructions, and which source wins in a conflict | The model fills the gaps with training memories |
| 6. Guardrails | The answer is checked before it is shown | The threshold below which the bot says it does not know | The bot answers anyway, confidently, and wrongly |
Remember the order: a mistake at stage 2 is never recovered at stages 4, 5 or 6. If the information was cut in half during chunking, no downstream setting will piece it back together. That is what the tester below makes visible.
Chunking: where most bad answers are decided
Chunking is the most mechanical stage of the pipeline, and by far the most underestimated. The base principle is brutal: cut the text every N characters, optionally with an overlap so that nothing sitting on a boundary is lost. That is what the most widespread splitters do, with two settings named chunk size and chunk overlap.
Two forces pull against each other. Short chunks are precise and cheap: you send little text back to the model on each question. But they cut answers in half. Long chunks cut nothing, but bill you four times more on every question, and dilute the useful line in noise.
The tester below applies exactly that algorithm to a real commercial policy document. Change the size, the overlap and the question asked, then watch whether the answer survives the split.
Chunking tester: does your answer survive the split?
The document below stands in for a knowledge base. Pick a setting and a question, then look at the chunks produced.
Chunk size
Overlap between two chunks
The question your customer asks
The answer sits WHOLE inside chunk 2: the bot can quote it as is.
This setting produces 3 chunks. On a real knowledge base, with 3 chunks retrieved per question, you send 720 characters back to the model on every answer, that is roughly 180 tokens.
Three lessons come out of this small tool, and they hold for any knowledge base.
- Overlap guarantees nothing. It moves the cuts, it does not remove them. Across the 27 combinations of this tester, 3 lose the answer, and the return question gets lost at 120 characters with 30 % overlap while it survived with no overlap at all.
- Large chunks never lose the answer, but they cost. Going from 120 to 480 characters multiplies by four the text sent back to the model on every question, and therefore the bill.
- Fixed-size chunking cuts in the middle of words. Serious splitters try separators first (paragraph, sentence, punctuation) and only fall back on the brutal cut as a last resort. If your tool does not, do the work yourself by structuring your documents into short, self-contained blocks.
The best chunking optimisation is not a setting, it is a rewrite. An FAQ where each answer fits in a self-contained paragraph of 300 to 600 characters splits itself, whatever the tool. We cover that preparation in our guide on how to train a chatbot on your own data.
What it really costs: indexing is free, the question is what you pay for
This is the most counter-intuitive part of the subject, and the most absent from the pages competing on this query. Everyone assumes the cost of a RAG chatbot sits in building the base. It does not, and by an order of magnitude.
Here is the calculation with public rates recorded on 5 August 2026 on the OpenAI pricing page. Take a base of 500 pages of 600 words, roughly 400,000 tokens to index.
| Line item | Volume | Public unit price | What you pay |
|---|---|---|---|
| Indexing 500 pages | 400,000 tokens | $0.02 / million (text-embedding-3-small) | $0.008, less than one cent |
| Reindexing one edited page | 800 tokens | $0.02 / million | $0.000016, effectively nothing |
| Embedding the question | 30 tokens | $0.02 / million | Negligible |
| Storing the vectors | A few thousand vectors | Qdrant Cloud free tier: 0.5 vCPU, 1 GB RAM, 4 GB disk | Nothing for a prototype, usage-based after that |
| Generating one answer | 2,330 input tokens, 250 output | $0.20 / $1.20 per million (gpt-5.6-luna) | 0.08 cents per answer |
| The same, on a stronger model | Same | $2 / $12 per million (gpt-5.6-terra) | 0.77 cents per answer |
Put differently: the base costs nothing to build, it costs every time it is read. The spending line of a RAG chatbot is the 2,330 input tokens sent back on every question: the system instructions, the retrieved chunks, the conversation history. That is exactly what the tester above lets you measure when you change the chunk size.
Two levers cut that bill without degrading answers. The first is prompt caching, billed far less on reads by most providers, which we detail in our article on the cost of a Claude chatbot. The second is returning three well-chosen chunks rather than eight average ones.
These figures are raw API costs, recorded on 5 August 2026 and liable to change without notice. They cover neither hosting, nor a production vector database, nor connectors to your channels, nor the human time spent preparing documents, which is usually the heaviest line of all. A RAG chatbot that costs “a few euros a month” on paper costs several person-days to get started.
Three ways to build a RAG chatbot
The question is not which one is best, but which one matches what you already have: a technical team, a budget, or neither. We laid out that reasoning for chatbots in general in our guide on the three routes to developing a chatbot; here is how it plays out for RAG.
Code everything
You have developers
You assemble the splitter, the embedding model, the vector database, the retrieval engine and the guardrails yourself. It is the route that gives the most control, especially if the base has to live on your own servers. It is also the one where the six settings from the table above are entirely yours, maintenance included.
Assemble building blocks
You already wire up automations
You plug an automation tool into a managed vector database and a model API. You write almost no code, but you stay responsible for the chain: you decide the chunking, the number of chunks retrieved and the behaviour when nothing is found. This route is fast for a prototype and holds up poorly as volume grows.
Use a platform that does the RAG for you
You want a result, not a pipeline
You upload your pages, your documents and your FAQ, the platform handles chunking, embedding, retrieval and guardrails, and you connect the bot to your channels. The six settings still exist, but they come pre-set and battle-tested across thousands of bots.
Botnation publishes a no-code chatbot building platform and builds custom chatbots for its clients, through its Enterprise offer and its chatbot creation experts. Either way the RAG part is handled: you supply the content, not the pipeline.
The For free plan at 0 € lets you test the whole chain on a small corpus before committing anything. The Basic plan at 39 € and the Pro plan at 59 € per month include 500 and 1,000 users respectively, plus 500 and 1,000 free AI credits granted once. The Enterprise offer, on demand, explicitly includes chatbot creation management.
What you should not pour into the base
This is the chapter that almost every RAG chatbot guide skips, and it is the one that costs the most when things go wrong. The CNIL is explicit in its questions and answers on generative AI:
“The deployer who chooses to connect the system to its own knowledge base (RAG) will also be responsible for that processing when it contains personal data.” (our translation from the French original.)
The practical consequence is simple: pouring a customer export, a ticket history or an internal directory into a vector database is processing of personal data, with everything that follows (legal basis, information of the individuals, retention period, access and erasure rights). The model provider does not carry that responsibility for you.
What goes into the base without trouble
- Your public pages, product sheets and help articles
- Your terms, returns and delivery policies
- Your anonymised internal procedures
- Your public prices and option grids
- Your de-identified training material
What requires an explicit decision
- A CRM export or named support tickets
- A staff directory with contact details
- Medical, HR or legal case files
- Non-public strategic documents
- Anything under a confidentiality agreement with a third party
On sensitive content, the CNIL invites you to look closely at where the base is hosted, and considers that an on-premise deployment is often the safer route. In practice that means asking your provider the question before you upload anything, not after. Our ten most frequent chatbot creation mistakes come back to this point.
This section describes the framework applicable on 5 August 2026 and does not replace advice from your data protection officer or from legal counsel. Regulatory positions on AI keep moving, and further recommendations have been announced.
Since 2 August 2026, saying it is an AI is no longer optional
Article 50 of the European Artificial Intelligence Act (Regulation EU 2024/1689) has applied since 2 August 2026. Its first paragraph is direct:
“Providers shall ensure that AI systems intended to interact directly with natural persons are designed and developed in such a way that the natural persons concerned are informed that they are interacting with an AI system, unless this is obvious from the point of view of a natural person who is reasonably well-informed, observant and circumspect.”
A RAG chatbot falls squarely inside that scope: it interacts directly with natural persons. The obligation is light (a clear notice that the interlocutor is an AI), but it is not optional, and Article 99(4) of the same regulation provides for fines of up to 15 million euros or 3 % of worldwide annual turnover. The risk diagram published by the CNIL explicitly files chatbots under “transparency obligations”.
Two sentences are enough, and they usually improve the conversation: say in the first message that the visitor is talking to an automated assistant, and explain how to reach a human. A RAG bot that is honest about its nature gets better questions, and therefore gives better answers.
Seven common symptoms, and their real cause in the pipeline
When a RAG chatbot gives a bad answer, the reflex is to change model. It is almost always the wrong reflex: in six cases out of seven the failure sits upstream of generation.
| What you observe | The stage at fault | What to check first |
|---|---|---|
| The bot invents a plausible but wrong answer | Retrieval and guardrails | Nothing was retrieved and the model filled the void. Force an “I do not know” below a similarity threshold |
| The bot answers halfway | Chunking | The answer is cut between two chunks. Test it with the tool above |
| The bot quotes outdated information | Ingestion | The old version of the document is still in the index. An update does not delete the old vector by itself |
| The bot misses on very short questions | Embedding | A three-word question carries little signal. Rewrite the question before searching |
| The bot ignores information that is present | Retrieval | The right chunk exists but ranks eighth. Retrieve more chunks or add a reranking step |
| The bot mixes up two products or two offers | Chunking and ingestion | Two similar sheets look alike. Prepend the product name to every chunk |
| The bot is slow | Generation | Too many chunks retrieved, so an over-long prompt. This is the only symptom where the model is genuinely at fault |
One last point, often forgotten: you have to read the questions asked, not only the answers given. It is the questions without a satisfying answer that tell you which documents are missing from the base. That is the same principle described in our article on where chatbots get their information.
Frequently asked questions
What is the difference between RAG and fine-tuning?
Fine-tuning changes the model itself, to teach it a style, a vocabulary or an answer format. RAG does not touch the model: it hands it the right documents at answer time. For knowledge that moves (prices, stock, procedures) RAG wins by a mile, because updating a document costs a fraction of a cent while retraining costs a project. Fine-tuning keeps its value for tone and format. The two combine.
Do you need to code to build a RAG chatbot?
No. No-code platforms run the whole pipeline for you: you upload your pages and documents, they handle chunking, embedding, retrieval and guardrails. Coding becomes necessary when you have a constraint the platform does not cover: hosting imposed on your own servers, an exotic data source, or a need for very fine retrieval tuning.
How many documents do you need for it to work?
Far fewer than people think. Twenty well-written pages covering the questions actually asked beat a thousand pages of badly structured documentation. The right starting point is the list of questions your support team receives most often, not your entire website. Adding useless documents even degrades retrieval, by drowning the good chunks among plausible neighbours.
Does RAG remove hallucinations?
It reduces them sharply, it does not remove them. As long as the model is allowed to answer when the search returned nothing relevant, it will fill the void. The real protection is not RAG itself, it is the guardrail that forces an “I do not know” below a similarity threshold, and showing the source used so the reader can check.
What chunk size should you choose?
There is no universal value, and that is precisely the problem: it depends on how long your answers are. The reliable method is to take your ten most frequent questions, find the sentence that answers each of them in your documents, and check that it fits whole inside one chunk. That is exactly what the tester above does, in miniature. If your answers fit in one sentence, short chunks are enough and cost you less.
Can you run RAG without sending your documents to a foreign provider?
Yes, provided you decide it before you start. The vector database can be hosted in Europe or on your own servers, and there are embedding and generation models that run on European infrastructure. The CNIL considers that for sensitive or strategic documentation, an on-premise deployment is often the safer route. Put the hosting question in your specification, not in your acceptance test.
Must a RAG chatbot announce that it is an AI?
Yes. Article 50 of the European AI Act, applicable since 2 August 2026, requires that the person knows they are talking to an AI system, unless it is obvious. A notice at the start of the conversation is enough. Fines under Article 99 of the same regulation can reach 15 million euros or 3 % of worldwide annual turnover.
What to remember
A RAG chatbot is not a clever model, it is a chain of six links whose weakest one is chunking. Its bill does not come from indexing, which is nearly free, but from the passages sent back to the model on every question. And the base you plug in engages your responsibility, not the model provider’s.
If you are starting from scratch, the most honest shortcut is this one: write twenty short, self-contained answers to the twenty questions you receive most, then upload them into a platform that runs the pipeline. You will know within a day whether the subject deserves a project, and you will have spent nothing to find out. Our guide to the generative AI chatbot and our overview of AI chatbot agencies cover the rest of the road.
Plug a chatbot into your own content
Your FAQs, your documentation and your help articles already exist. Botnation turns them into answers for your customers on your website, WhatsApp, Messenger and Instagram, without you building any pipeline.
Sources: CNIL, questions and answers on the use of a generative AI system (18 July 2024, published in French); Regulation (EU) 2024/1689 on artificial intelligence, Articles 50 and 99; Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv 2005.11401 (22 May 2020); public pricing grids of OpenAI, Qdrant and Botnation recorded on 5 August 2026.
