AI product questions · Hard
How would you design an AI assistant for customer support at an online retailer?
Short answer
Start with the support problems worth automating, usually frequent, low-risk questions such as order status and returns. Ground the assistant in order data and policy documents, define clear limits on what it can do, and design easy hand-off to humans. Measure resolution without escalation, customer satisfaction and accuracy, with guardrails on wrong answers and refunds.
Detailed answer
Framework
- Problem selection: analyse ticket categories by volume and risk.
- Scope and limits: what the assistant can answer, what it can do (actions), and what it must never do.
- Grounding: connect to reliable sources: order systems, policy documents, help articles.
- Experience design: tone, transparency that it is an AI, easy escalation.
- Evaluation: offline tests before launch, then live metrics.
- Risks and safeguards.
Worked example
Start narrow: order status, return eligibility and delivery changes. These are frequent, have clear answers and can be checked against data.
Grounding: the assistant looks up the customer's actual order rather than guessing. Policy answers come from current policy documents, with the source shown.
Actions with limits: it can start a return within policy; anything outside policy, or above a set refund value, goes to a human.
Hand-off: the customer can ask for a person at any time, and the conversation history passes along so they do not repeat themselves.
Evaluation:
- Before launch, build a test set of real (anonymised) questions with correct answers and review outputs.
- After launch: share of conversations resolved without escalation, customer satisfaction, repeat contact within a few days, and an audited sample for accuracy.
Guardrails: incorrect policy statements, inappropriate refunds, and complaints about being unable to reach a human.
Common pitfalls
- Trying to automate everything on day one.
- Letting the model answer from general knowledge instead of company data.
- Hiding the human option to reduce cost.
- Measuring only deflection, which rewards frustrating customers into giving up.