AI product questions · Medium
How would you measure the quality of a feature powered by a large language model?
Short answer
Combine three layers: offline evaluation against a representative test set with clear criteria, human review of samples for nuanced quality, and online product metrics showing whether users actually benefit. Define what 'good' means for your use case first, such as accuracy, helpfulness, tone and safety, and track failure categories so you can improve systematically.
Detailed answer
Framework
- Define quality for the specific task. A summary feature cares about faithfulness and coverage; a writing assistant cares about tone and usefulness.
- Offline evaluation: a test set of realistic inputs with expected outputs or grading rubrics. Run it on every prompt or model change.
- Human review: experts grade samples on a rubric; check that graders agree with each other.
- Automated checks: format validation, banned content, and model-based grading calibrated against human judgement.
- Online metrics: what users do with the output.
- Failure taxonomy: categorise errors to direct work.
Worked example
Feature: AI-generated summaries of long customer reviews on product pages.
Quality criteria: faithful (no claims not in reviews), balanced (positives and negatives), concise.
Offline: a test set of products with varied review profiles; reviewers check every summary for unsupported claims.
Online:
- Engagement with the summary (expanding, scrolling past).
- Effect on conversion and returns in an A/B test.
- User feedback ("Was this helpful?") and reports of inaccuracies.
Failure taxonomy: invented details, ignoring negative reviews, outdated information after new reviews arrive. Track the share of each over time.
Common pitfalls
- Relying on a few impressive demos.
- Using only automated model grading without checking it against people.
- Measuring engagement without checking accuracy.
- Not re-running evaluations after prompt or model changes.