How to stop a chatbot you are paying for from making things up

The short answer

A chatbot makes things up when nothing in the build tells it not to, and nothing checks whether it did. The model is built to produce a fluent answer. If your documents do not contain the answer, or the bot’s search misses the right passage, a fluent guess is what you get, delivered in the same confident tone as a correct one.

You cannot fix that with one line saying “do not hallucinate”. You fix it with rules the developer can build and you can test: answer only from your material, say “I don’t know” in words you chose, never state a price, date or policy that is not in a source, hand over when unsure, and keep a log of what each answer was based on. Put them in the brief before the quote. The brief on the chatbot job page has the fields; this guide is about what to write in them so the bot does not invent.

Why chatbots fill gaps

Three situations produce almost every made-up answer:

• The answer is not in what you handed over. The customer asks about weekend opening hours, and your documents do not mention weekends. The bot answers anyway, from what opening hours usually look like.
• The answer is there, but the search missed it. Most bots that answer from your documents first look up the passages that seem relevant, then write an answer from them. If the lookup brings back the wrong passage, the answer is built on the wrong material.
• The question asks for something to be worked out. “So how much would three of these cost with delivery?” The bot calculates, or estimates, instead of quoting.

In all three, the bot is not lying. It is doing what a fluent text generator does when it has no instruction for the gap. That is why the gap belongs in the brief.

What running an AI writer taught me

I do not run a customer chatbot. I do run an unattended AI job in my content pipeline that writes drafts from a topic queue, and one of its rules is the one you want for a chatbot: nothing about my own experience may appear unless it can be traced to my records. Three things from running it apply directly.

A rule is not a check. In an early test, a draft contained one sentence about something that had happened in my own automations, with no record pointing to it. I searched, found nothing and deleted it, and then tightened the rule. I also added a separate, read-only review that compares every number and every first-person claim in a draft against the source files before anything is scheduled. Run on that same draft, it traced four figures and three stories back to their records. The writer was told the rule. The review is what enforces it.

“Not found” is a fact about the search. That deleted sentence later turned out to be true. The event was recorded, in another project’s notes, and my search had missed it. A strict system will sometimes refuse a true answer because its lookup did not reach the right document. For a chatbot that is the right failure: “I don’t have that information” is recoverable, an invented policy is not. But it means the documents you hand over have to be complete, and the developer should tell you what the bot can and cannot search.

When unsure, hold. When the drafting job cannot confirm something only I know, it leaves the claim out and writes a question for me. A draft that carries an open question is never published automatically, even when it passes every other check. It waits until I answer. When I answer, anything I say I am unsure about is marked uncertain and stays out of the page. For a chatbot, the equivalent is the handoff: when the answer cannot be confirmed from a source, a person answers instead.

Seven rules to put in the brief

Rule	What to write in the brief	How you test it
1. Answer only from the material	“Answer only from the documents I provide. Do not use general knowledge about businesses like mine.”	Ask five questions whose answers are true in general but not in your documents
2. Exact words for “I don’t know”	“When the answer is not in the documents, say: [your wording], and offer [handoff].”	Ask ten questions it cannot answer; count the guesses
3. No unsourced prices, dates or policies	“Never state a price, date, deadline or policy unless it appears in the documents. Never calculate totals.”	Ask for a total, a delivery date and a refund rule
4. Hand over when unsure	“If the question is ambiguous or the documents conflict, hand over to [person, channel, time].”	Ask a question your documents answer two different ways
5. Never-do topics	“Refuse and hand over on: [refunds, legal or medical advice, other customers, …].”	Ask each one, phrased three ways
6. A log of what it used	“Log each conversation with the document passages each answer was based on, on my account.”	Pick three answers and check the passages in the log
7. Out-of-date material removed	“Only the documents I mark as current. I will remove old price lists before handover.”	Ask about something that changed recently

Rules 2 and 6 matter most. Rule 2 turns “I don’t know” from a failure into a tested, correct answer. Rule 6 is your version of my read-only review: without it, you cannot tell whether a wrong answer came from a missing document, a bad search or an invention, and each needs a different fix.

A brief section you can copy

Accuracy rules.

1. Answer only from the documents listed in this brief. No general knowledge.
2. When the answer is not in them, reply: “I don’t have that information. Would you like me to pass this to [team]?”
3. Never state a price, date, deadline or policy unless it appears in the documents. Never calculate totals.
4. If the documents conflict or the question is unclear, hand over to [person] on [channel] within [time].
5. Refuse and hand over on: [never-do list].
6. Log every answer with the passages it was based on, stored on my account for [period].

Acceptance. I will run my own test set: [30] real questions, [10] it cannot answer, [5] never-do topics and [3] conflicting ones. Pass mark: no invented prices, dates or policies, and at most [N] wrong answers on the real questions.

The pass mark is yours to set, but set it before the demo. How to build and score the test set is in how to test an AI chatbot before you accept it.

What to ask the developer before you order

• “How do you measure wrong answers?” A good answer names a test set and a number. “The model is very accurate” is not an answer.
• “What happens when the search finds nothing relevant?” You want a specific behaviour, ideally rule 2, not “it will still try to help”.
• “Can I see which passages each answer used?” If not in the chat window, then in the log.
• “Who updates the documents after launch, and how fast does the bot use the new version?” Most wrong answers in month three come from stale material, not the model.

Sellers whose gigs mention documentation, testing and human handoff, such as Bensbahou, are already talking about these things; ask them anyway, and get the answers in writing in the order thread.

Where it goes wrong anyway

• Your documents disagree with each other. The bot picks one. Find conflicts before handover; rule 4 covers the rest.
• The test set is written by the developer. It tests what the bot already does well. Write it yourself, before the demo.
• Success is measured by how many questions it answers. A bot tuned to answer more will guess more. Count wrong answers and handoffs separately, as the outcome-first brief suggests.
• Nobody reads the log. Pick ten conversations a week for the first month and check the passages behind the answers.

Wrap-up

A chatbot that never says “I don’t know” is not a better chatbot; it is one whose guesses you cannot see. Write the rules into the brief: answer from your material only, exact wording for the gap, no unsourced prices or policies, a handoff when unsure, and a log of what each answer was based on. Then test the gaps on purpose, before you pay. What you hand over in the first place is covered in what to give a chatbot developer before they start.