How to test an AI chatbot before you accept it

The short answer

A demo shows you the questions the builder picked, on the day it went well. Acceptance should show you the questions your customers ask, including the ones the bot must not answer, in the channel they will use.

So before the final payment, run a test sheet you wrote before you saw the demo. It has five parts: about 30 real questions with the answers you expect, a handful it should not be able to answer, five never-do topics, one real handoff, and a few running basics (accounts, logs, cost, updating an answer). Count wrong and made-up answers, not good ones, and compare with a pass mark you set in advance. The acceptance list on the chatbot job page is the short version; this guide explains how to run it.

Why a good demo proves little

A builder demos the bot the way anyone shows off their work: with questions it answers well. That is fine. It just is not a test.

Three things make a chatbot fail after acceptance, and none of them show up in a demo:

• Questions from outside the builder’s head. Customers ask in their own words, with typos, half-sentences and two questions at once.
• Questions with no answer in your documents. A chatbot that has not been told how to say “I don’t know” will fill the gap with something plausible.
• Paths that only run when something goes wrong. The handoff to a person, the refusal, the retry. Nobody demos those.

I do not run a customer chatbot, but I run automations that answer people and AI jobs that write from instructions, and my records point the same way. The failures that cost me most were never in the part I had watched working.

The five-part test sheet

Write this before the demo, and keep it to yourself until you test. If the builder sees the questions first, you are testing whether they read your sheet.

Parts 1 to 3 come straight from the brief. If you have not written one yet, what to give a chatbot developer before they start covers the 30 questions, the never-do list and escalation rules.

How to score it

Decide the pass mark before you test, and write it into the brief so the builder quotes for it. Then score like this:

Result	Counts as
Correct answer, in your material	Pass
“I don’t know” or a handoff where the answer is not in your material	Pass
Correct, but vague or wordy	Pass, note it for tuning
Wrong answer	Fail
Confident answer that is not in your material, even if it sounds right	Fail
Answers a never-do topic	Fail, and it should block acceptance
Handoff promised, nobody receives it	Fail, and it should block acceptance

Count failures, not successes. A bot that answers almost everything well and invents a refund policy once is not an almost-perfect bot. It is a bot that will invent a refund policy for a customer.

Test from outside the system

The most useful lesson in my own records is about where you test from.

I run an automation that answers people who comment on my posts. When I paid to promote one post, the platform created a separate copy of it for the ad. The list of posts my automation read did not include that copy, and the comment count did not include its comments either. Four people who commented on the promoted copy got no answer for three days, while every comment on the normal post was answered.

My first check said everything was fine. I had compared the automation’s list of comments with the platform’s comment count, and both came through the same filter, so the system was checking itself. The comparison that found the gap was outside it: the app screen, where both copies’ comments showed together.

For a chatbot, that means:

• Test in the real channel your customers use (the site widget, WhatsApp, the phone), not the builder’s test console.
• Read the conversation logs yourself after the test, and check that every test conversation is there.
• Don’t accept the bot’s own report of how it did. A summary made by the same system that answered is not a second opinion.

Who should run the test

You should, with questions the builder did not choose. That is the whole point.

For the drafts in my content pipeline I went a step further: every draft an AI writer produces is checked by a second, separate session that can only read files, never edit them. It traces every number and every first-person claim back to a source file. If it cannot, the draft stays unpublished and the reason is written into the topic queue. On its first test it traced four kinds of numbers and three first-person stories to their sources. The writer and the checker are separate on purpose, so the checker has no stake in the draft being right.

You do not need a second AI to test a chatbot. You need the same separation: the person who tests is not the person who built it, and the questions are not the builder’s.

After launch: what to re-test

A chatbot changes when its documents, prompts or AI model change. Keep the test sheet and re-run it:

• after any change to the documents or instructions
• when the provider changes the model the bot uses
• once a month, with a few new real questions added from the logs

Agree who does that, at what rate, in the same conversation as the build. Putting maintenance in the contract has six lines you can copy. And if you are still choosing a builder, what 252 Fiverr reviews say about hiring a chatbot developer lays out what buyers paid, how long builds took and how often they came back.

Wrap-up

Write the test before the demo. Include what the bot must not answer and the handoff it must make, test in the real channel, and count failures against a pass mark you set in advance. Then acceptance is a result you can point to, not a feeling from a good demo.