Test, Launch & Maintain
Testing is the step that gets cut when a deadline gets close. It’s also the only thing standing between a chatbot and a confident wrong answer in front of a real user.
Test
Only you can judge whether an answer is right. We can tell you whether the chatbot found the content and used it sensibly. Whether the content is correct is a subject-matter question, and it belongs to the people who know the subject.
Build a question set. Aim for 50–100 questions for a managed chatbot, 20 for a Gem. Start with the twenty from your planning guide.
- Use your users’ actual words, including the vague and badly spelled ones
- Include several phrasings of your most common questions
- Write down what a correct answer contains, so scoring isn’t a matter of opinion
- Mark the ones where a wrong answer would cause real harm
Include edge cases — aim for at least 15–20% of your set:
- Questions clearly outside its scope
- Questions that are close to scope but not quite
- Ambiguous questions
- Hostile or abusive messages
- Wellbeing or crisis-adjacent messages
- Questions about the chatbot itself
- Requests for someone’s personal records
Score simply
Four ratings are enough.
| Rating | Means |
|---|---|
| Correct | Accurate and complete. You’d be happy for a user to see it. |
| Thin | Accurate but incomplete or awkward. Acceptable. |
| Wrong | Inaccurate, misleading, or invented. |
| Harmful | Wrong on something critical, failed to route a safety concern, or revealed something it shouldn’t have. |
Any harmful rating blocks launch, whatever the overall numbers say.
Decide your bar before you test. Write down what “good enough to launch” means while you still don’t know the results. Decided afterwards, it drifts to match whatever you got — every time.
Record why each failure happened. Content missing, content wrong, didn’t find the content, or bad instructions. This is the most useful column in the Chatbot Workbook, because it tells you whose problem each failure is. Most of them will be content, and content is yours.
Test again after you fix things. Changes have a way of breaking answers that used to work.
Launch
Soft launch first. Give it to a small group for a week or two. They’ll ask things you never thought of.
Work through the pre-launch checklist in the Chatbot Planning Guide.
Brief the staff whose work this touches. Including, directly, whether it changes their job. People will assume the worst if you don’t tell them, and front-line staff are the ones who’ll hear about it first when it gets something wrong.
Set expectations in the interface. What it knows, what it doesn’t, and how to reach a person.
Make sure feedback goes somewhere a human reads. A feedback button nobody monitors is worse than none, because it looks like you’re listening.
Managed chatbots: watch usage in the first week. Real traffic is rarely what anyone projected.
Maintain
Chatbots decay. Content drifts. Deadlines pass. Links break. The person who knew how it worked takes another job. Left alone, a chatbot doesn’t stop working — it keeps answering, just increasingly wrongly.
A cadence that works
| How often | What to do |
|---|---|
| Weekly, first month | Read the conversation logs |
| Monthly | Read the logs; note content gaps |
| Quarterly | Review content; check links and dates |
| After content changes | Re-run your question set |
| Annually | Full re-test; decide whether to keep it |
Read the logs for what people asked and didn’t get. That’s where your next content update comes from. The questions people ask a chatbot are often not the ones you’d have guessed.
Plan for the handover. Write down how it works and who to call before the content owner moves on, not after. This is the single most common way a working chatbot quietly becomes an unmaintained one.
Retire it deliberately when the time comes
Any of these is a reason to:
- Nobody owns the content any more
- Reviews have stopped happening
- Usage doesn’t justify keeping it
- The program it supported has ended
- Funding has ended
- There’s a quality problem you can’t resolve
Retiring a chatbot on purpose is a normal outcome. Give people notice, redirect the entry points somewhere useful, and take the links down.
When to come back to us
A new use case, a significant change of scope, quality problems you can’t solve, a surprise on cost, or a change of owner.