Applied AIArticle
Evaluating LLM features before you ship them
Demos are easy; reliable AI features are not. How to build a small evaluation set, choose criteria, and decide whether a feature is ready.
The demo trap
A language-model feature that works in a demo can still fail for a meaningful share of real users. Demos show the inputs you thought of; production brings the ones you did not. Evaluation is how you find out which you have before your users do.
Step 1 — Write down what "good" means
Before testing anything, define success for this feature in plain language:
- Correctness: is the answer factually right, given the source material?
- Groundedness: does it only use information it was given?
- Format: does it follow the required structure?
- Tone and safety: is it appropriate for your users and brand?
- Usefulness: would a user act on it?
Pick the two or three that matter most. Trying to optimise everything at once produces vague results.
Step 2 — Build a small, honest test set
Start with 50 to 100 real or realistic inputs. Include:
| Slice | Why |
|---|---|
| Typical requests | The bread and butter |
| Edge cases | Very long, very short, ambiguous, multilingual inputs |
| Adversarial inputs | Attempts to make it ignore instructions |
| "Should refuse" cases | Requests the feature must not fulfil |
For each input, write what an acceptable output looks like — not a single perfect answer, but criteria a reviewer can check.
Step 3 — Grade consistently
You have three grading options, and most teams combine them:
- Code checks for anything deterministic: valid JSON, required fields present, length limits.
- Human review for nuanced quality, using a short rubric (for example 1–3 per criterion).
- Model-assisted grading to scale review, calibrated against human grades on a sample. If the model grader disagrees with humans often, do not trust it yet.
Step 4 — Change one thing at a time
Prompt edits, model changes and retrieval changes all interact. Re-run the full set after each change and compare scores per slice. An improvement on typical requests that breaks refusals is not an improvement.
run_id prompt_v model typical edge adversarial refusals
r12 v3 model-a 0.86 0.61 0.70 0.95
r13 v4 model-a 0.90 0.66 0.58 0.95 <- regression
Step 5 — Decide with a launch bar
Agree in advance what "ready" means — for example, "no critical failures on refusals, and at least 85 per cent acceptable on typical requests" — and the fallback when the model is unsure, such as handing off to a human or showing sources.
After launch
Keep evaluating. Log real inputs (with appropriate consent and privacy controls), sample them weekly, and add new failure cases to the test set. Your evaluation set should grow with your product.
Key takeaways
- Define "good" before you test.
- A small, varied test set beats a large, uniform one.
- Combine code checks, human review and calibrated model grading.
- Change one variable at a time and compare per slice.
- Set a launch bar in advance and keep evaluating after launch.
Written by
The ActuateMinds editorial team writes practical guides for people moving into product, design, data and applied-AI roles.
Get the next article in your inbox
One short email a week. No spam.
Related reading
Applied AI2 min read
Prompting is not a strategy: designing reliable AI features
Better prompts help, but reliable AI features come from product design: narrowing the task, grounding the model, and planning for failure.
Industry Insights2 min read
What changes when every product role uses AI
Observations on how AI tools are reshaping the day-to-day work of PMs, designers and analysts — and which human skills are becoming more valuable, not less.
Product Management5 min read
How to write a PRD engineers actually read
Most product requirement documents are skimmed once and forgotten. A good PRD is short, opinionated about the problem and honest about what is still unknown. Here is a structure that earns attention from engineering.