How should I split a chatbot project into milestones?

By what you can test, not by what the developer builds. A common split is: answers from your documents, limits and handoff to a person, the real channel and any actions, then a short live pilot. Each milestone ends with a test you run yourself and a pass mark agreed in advance.

Should I release a milestone payment if a few things are still broken?

Not unless the open items are written down with a date and do not affect the next milestone. Later work is built on top of earlier work, so an accepted problem tends to become harder to fix, and harder to call a fix rather than a change.

Who should run the milestone tests, me or the developer?

Both. The developer should run them before telling you a milestone is ready, and you should run them again yourself, on the real channel, from an account that is not theirs. Their test proves it can work; yours proves it works for you.

Chatbot Milestones: What to Check Before Paying

The short answer

A milestone payment is the only point in a longer chatbot project where you have leverage. Release it when that milestone’s test passes, not when the demo looks good. Split the project by what you can test, write the pass mark for each milestone before the work starts, and run every test yourself from outside the system, the way a customer would meet the bot.

Four milestones cover most support bots: answers, limits and handoff, channel and actions, and a short live pilot. The table below gives the test and the pass mark for each.

Why milestones need their own tests

Chatbot work runs longer than most automation work. In the 252 chatbot developer reviews I read, about one order in five took 15 days or more, and about a quarter cost over $1,000. Those are the projects that get paid in parts.

The risk with parts is simple: each milestone is built on the one before. If the bot’s answers are weak at milestone one and you accept them anyway, milestone two adds the handoff on top of weak answers, and milestone three connects weak answers to your website. By the end, a problem that was one fix early is spread across everything, and it is no longer clear whether fixing it is part of the order or new work.

A chatbot also makes “done” fuzzier than a workflow does. A workflow either put the row in the sheet or it did not. A bot can answer every question in the demo and still get your customers’ questions wrong. That is why each milestone needs a test you wrote, not one the builder picked.

I have paid for work both ways. With a fixed price, payments went out at set points in time; with milestones, I released each one after checking what that stage had delivered. That check is the only thing a milestone gives you that a date does not.

A four-milestone plan

Checklist · Copy this into the order

Milestone	What is delivered	Your test	Pass mark (agree before starting)
1. Answers	Bot answers from your documents and FAQs, in a test link	Your 30 real customer questions, typed as customers write them	No made-up answers; at most an agreed number of weak ones
2. Limits and handoff	Never-do list, “I don’t know” path, handoff to a person	Five never-do questions, five it cannot answer, one handoff request	Refuses or hands over every time; a person receives the handoff with the history
3. Channel and actions	Bot on the real channel; any actions (ticket, booking, lookup) connected	Repeat a sample of tests 1 and 2 on the real channel; trigger each action once	Same results as on the test link; each action appears in the real system
4. Live pilot	Bot live for real customers for an agreed period	Read the conversation logs; check costs; update one answer yourself	No new never-do failure; cost per 100 conversations known; your edit shows up
:::

The 30 questions and the never-do list come from your brief. If you have not written them yet, what to give a chatbot developer before they start walks through both. The final check at milestone four matches the acceptance list on the chatbot page, so the last payment is tied to the same test you would have used without milestones.

For the details of scoring, what counts as “weak” and how to write the questions, see how to test an AI chatbot before you accept it.

Test from outside, at every milestone

The most useful rule I have for any bot that talks to people unattended: do not check it with its own reports.

I run a bot that replies to comments on my social posts. It is not an AI chatbot, but it answers people with nobody watching, and it taught me this the hard way. When I promoted one post, the platform created a separate copy of it for the promotion, and the comments on that copy never appeared in the list my bot reads. Five comments on the normal post were answered. Four on the promoted copy got no reply for three days.

My first check that morning said everything was fine. I had compared the bot’s list of comments with the platform’s comment count, and they matched. But the count came through the same filter as the list, so I was checking the bot against itself. The real comparison was the app screen, which showed both sets of comments together. Once I looked there, the gap was obvious.

For a chatbot milestone, “from outside” means:

Test on the real channel, from a phone and an account that are not the developer’s.
Count answers yourself from the conversation, not from a dashboard the developer built.
For actions, look in the real system (the ticket queue, the booking calendar), not at the bot saying “done”.

Ask what happens between milestones

Two questions are easy to miss because they only matter once real people use the bot.

What does it do when it cannot deliver? My comment bot once failed to send a message to one person, most likely because of that person’s own message settings (the same bot had reached someone else 40 minutes earlier), and tried again every 20 minutes, six times. Nothing was wrong with the bot, and nothing was going to change. It now gives up after two failures and posts a short public reply asking the person to message first. Ask your developer: when the bot cannot reach someone or cannot complete an action, how many times does it retry, and who hears about it?

Who is told when it goes quiet? A bot that stops answering produces no error, just silence. Ask for an alert to you when there have been no conversations for longer than usual, or when handoffs stop arriving.

The message to send before milestone one

Paste this into your messages with the developer before ordering:

Before we start, can we agree how each milestone is accepted?

Each milestone ends with a test link or live setup I can use myself, on a phone and account that are not yours.

I will run the tests in the table I attached and reply within [two working days].

A milestone is paid when its test passes at the agreed mark. Anything open is written down with a date before payment.

Accounts, AI provider key and conversation logs are on my accounts from milestone one.

Changes to what the bot should do are new work; fixes to what the milestone promised are not.

Does that work for you, or what would you change?

Line 2 matters as much as the others. Your testing time is part of the project length. If you take a week to test each milestone, a four-milestone project gains a month.

Where it goes wrong anyway

The pass mark was never written down. Then every milestone becomes a discussion. Agree it before starting, with a number.
You change the questions mid-project. New questions are fine, but they are a change. Add them at the next milestone and agree what it costs.
The pilot is too short to see anything. A few days may bring only a handful of real conversations. Agree the pilot length by number of conversations, not days.
Nobody owns the bot after the last payment. Knowledge goes out of date. Agree who updates answers after launch, at what rate, as the maintenance guide describes.

Wrap-up

Milestones protect you only if each one is tested. Split the project by what you can test, write the pass mark first, run the tests yourself from outside the bot’s own reports, and write anything open down before you pay. Then the final milestone is a formality, not the first time you find out how the bot behaves.

What to check at each chatbot milestone before you pay

The short answer

Why milestones need their own tests

A four-milestone plan

Questions people ask

If you'd rather hand it off

The short answer

Why milestones need their own tests

A four-milestone plan

Questions people ask

If you'd rather hand it off

More in Acceptance and handover