Quick answer
An AI chatbot should solve a defined customer problem, such as answering routine service questions or finding information in a product catalogue. Start with actual enquiries and identify which can be answered from approved material. Keep a path to a person when the answer is uncertain or the request needs judgement.
Practical scope
Decide what documents the assistant may use, who updates them and whether it may access customer-specific records. If it collects personal information, define consent, retention and staff access. Test it with real questions, ambiguous wording and cases where the correct answer is “I don't know.”
Measure useful outcomes: resolved questions, handovers, incorrect answers and time saved. Ask developers to explain ongoing model, hosting and monitoring costs. A pilot with a narrow knowledge base is often the clearest way to learn whether the tool helps customers.
What to define before requesting proposals
- Specific customer problem and permitted knowledge sources
- Questions that require human judgement or escalation
- Personal data, permissions and retention
- Accuracy tests, monitoring, cost and success measures
Questions to ask service providers
- How does the chatbot cite or identify its source?
- What happens when information is missing or uncertain?
- Who reviews incorrect answers and updates knowledge?
How to assess the answers
Begin with a narrow, approved knowledge base and real test questions. Measure correct resolution, handover and harmful errors instead of message volume alone.
Separate answering from taking action
A chatbot that explains a policy has a different scope from one that changes a booking or retrieves a private order. List allowed actions, required approval and the handover path. Use fictional or suitably authorised test material; do not put confidential customer requests into a public demonstration.
Define a bounded pilot
- Choose approved source documents and assign an update owner.
- Prepare answerable, ambiguous and out-of-scope questions.
- Record incorrect answers and successful human handovers.
- Ask what access, logging and ongoing monitoring are included.
Should the assistant answer every question?
No. Define when it should decline, ask for clarification or pass the conversation to a person. An explicit limit is more useful than a confident answer unsupported by approved material.
AI chatbot pilot evaluation sheet
Copy these items into your brief, adapt the requirements and assign an owner and status. Examples illustrate planning decisions, not customer case studies. Fill cost fields with HKD quotations you obtain and state the billing period.
| Decision / item | Information to provide | Acceptance / comparison check |
|---|---|---|
| Answerable question | Approved source and expected answer | Answer is supported by the approved material |
| Ambiguous/out-of-scope | Missing information and escalation rules | Assistant clarifies, declines or hands over appropriately |
| Private action | Allowed operation, permission and approval | Unauthorized data/actions remain inaccessible |
Download the editable worksheet (CSV)