What is RAG (retrieval) in a chatbot?
What it is, in plain words
Think of a new employee on the help desk who is clever but has never worked at your company. You can ask them to answer customers from what they generally know about the world. Or you can hand them the company binder and say: before you answer, find the right page and answer from that.
The second approach is RAG, short for retrieval-augmented generation. It has two steps:
- Retrieval. When a question comes in, the system searches your material (help articles, PDFs, product sheets, policies) and pulls out the few passages that look most relevant.
- Generation. It hands those passages to the AI model together with the question, and the model writes an answer based on them.
The model itself is not changed. Your documents are not “inside” it. They sit in a searchable store, and the bot looks things up every time. Terms you will hear around it:
- Knowledge base: the documents the bot may answer from.
- Chunks: the small pieces your documents are cut into so they can be searched.
- Vector database or embeddings: a way of searching by meaning rather than exact words, so “Can I get my money back?” finds the page titled “Refund policy”.
A concrete example
Say a small online shop wants a chatbot that answers delivery and returns questions. The shop has a returns page, a delivery page and a spreadsheet of product care instructions.
A customer types: “I bought the linen shirt two weeks ago, can I still return it?”
- The retrieval step searches the store and finds the returns paragraph (“30 days from delivery, unworn, tags attached”) and the linen care entry.
- The model receives the question plus those two passages and answers: yes, within 30 days of delivery, if unworn with tags.
Now the shop changes its policy to 14 days and edits the returns page. If the bot re-reads the updated page, the next answer changes. Nobody retrains anything.
Where it goes wrong is almost always step one. If the returns page was never added to the store, or was cut into pieces so that “30 days” ended up in a different chunk from “linen”, the model gets the wrong passage or none at all, and may still answer confidently.
I have seen the same failure shape outside a chatbot. In September, while fact-checking a piece of my own writing, I searched my files for the record behind one sentence (an overnight job that hit a monthly usage limit), found nothing, and removed the sentence as unsupported. It was true. The record sat in a different project’s notes, and the search had missed it. The judgement was sound for what the search returned; the search just did not return the page that held the answer. RAG systems fail the same way: the answer is only as good as what the search brings back.
Why it matters when you hire someone
You will not build the retrieval yourself, but four things about it decide whether the bot is trustworthy, and a buyer who does not ask about them usually finds out from a customer.
1. Most wrong answers are search problems. When a RAG bot gets something wrong, the model is often doing exactly what it was told with the wrong page. A freelancer who can show you which passages the bot used for an answer can fix this quickly. One who cannot is guessing.
2. Someone has to keep the documents current. The bot is exactly as up to date as the material in its store. Ask who adds new documents, how often the store is refreshed, and whether you can do it yourself. The chatbot brief has a field for exactly this.
3. “I don’t know” has to be built in. Retrieval does not stop the model from filling a gap. The bot needs an instruction to say it does not know when nothing relevant was found, and a route to a person.
4. You need a test, not a demo. A demo uses questions the documents answer well. A real test includes questions they cannot answer, and counts the made-up answers. The guide to testing a chatbot before you accept it covers how to build that test set.
What to ask the freelancer
Paste these into your first message, or into the “Knowledge” field of the chatbot brief:
A good answer is specific: the formats it reads, the refresh step, a way to see the source passages, a defined “I don’t know” behaviour, and a store on your account. If the answer is “we train it on your data”, ask what that means in practice, because for most small business bots it should mean retrieval.
Questions people ask
Is RAG the same as training a chatbot on my data?
No. Training changes the model itself and is rarely what a small business needs. RAG leaves the model as it is and hands it the relevant pages of your documents at the moment a question comes in. That is why updating a document can change the answer the same day.
Does RAG stop a chatbot from making things up?
It reduces it, but does not remove it. If the search step finds the wrong passage, or finds nothing, the model can still produce a confident answer. The bot also needs an instruction to say it does not know, and a test with questions your documents cannot answer.
Do I need RAG for my chatbot?
If the bot must answer from information that is specific to your business, such as prices, policies, opening hours or product details, then yes, some form of retrieval is the usual way to do it. A bot that only greets people and routes them to a person may not need it.