Skip to content

AI product questions · Hard

How would you design an AI assistant for customer support at an online retailer?

Short answer

Start with the support problems worth automating, usually frequent, low-risk questions such as order status and returns. Ground the assistant in order data and policy documents, define clear limits on what it can do, and design easy hand-off to humans. Measure resolution without escalation, customer satisfaction and accuracy, with guardrails on wrong answers and refunds.

Detailed answer

Framework

  1. Problem selection: analyse ticket categories by volume and risk.
  2. Scope and limits: what the assistant can answer, what it can do (actions), and what it must never do.
  3. Grounding: connect to reliable sources: order systems, policy documents, help articles.
  4. Experience design: tone, transparency that it is an AI, easy escalation.
  5. Evaluation: offline tests before launch, then live metrics.
  6. Risks and safeguards.

Worked example

Start narrow: order status, return eligibility and delivery changes. These are frequent, have clear answers and can be checked against data.

Grounding: the assistant looks up the customer's actual order rather than guessing. Policy answers come from current policy documents, with the source shown.

Actions with limits: it can start a return within policy; anything outside policy, or above a set refund value, goes to a human.

Hand-off: the customer can ask for a person at any time, and the conversation history passes along so they do not repeat themselves.

Evaluation:

  • Before launch, build a test set of real (anonymised) questions with correct answers and review outputs.
  • After launch: share of conversations resolved without escalation, customer satisfaction, repeat contact within a few days, and an audited sample for accuracy.

Guardrails: incorrect policy statements, inappropriate refunds, and complaints about being unable to reach a human.

Common pitfalls

  • Trying to automate everything on day one.
  • Letting the model answer from general knowledge instead of company data.
  • Hiding the human option to reduce cost.
  • Measuring only deflection, which rewards frustrating customers into giving up.