Metrics & analytics · Easy
What are guardrail metrics, and why do they matter?
Short answer
Guardrail metrics are measures you commit not to harm while optimising a primary metric. They protect user experience, trust and the wider business from side-effects. For example, a team pushing notifications to lift engagement should watch unsubscribes and uninstalls. Define guardrails and acceptable thresholds before launching a test, so the decision is not rationalised afterwards.
Detailed answer
Definition
A primary metric tells you whether your change achieved its goal. A guardrail metric tells you whether you caused damage somewhere else getting there.
Typical categories
- User experience: error rates, page load time, crash rate.
- Trust and wellbeing: complaints, unsubscribes, opt-outs, refund requests.
- Business health: revenue per user, cost per order, support contacts.
- Other teams' metrics: a change to search might affect recommendations.
Worked example
A team wants to increase repeat purchases by sending more promotional push notifications.
- Primary metric: purchases per user over a set period.
- Guardrails: notification opt-out rate, uninstall rate, and customer complaints mentioning spam.
- Threshold: agree in advance what increase in opt-outs would block the launch, even if purchases rise.
If purchases rise but opt-outs climb sharply, the team has borrowed against the future: users who switch off notifications can no longer be reached for anything, including important updates.
How to use guardrails well
- Pick three to five, not twenty.
- Set thresholds before the test starts.
- Make sure guardrails are measured with the same rigour as the primary metric.
- Review them at the same cadence; a guardrail that nobody checks is decoration.
Common pitfalls
- Choosing guardrails after seeing the results.
- Listing guardrails with no thresholds, so any harm can be explained away.
- Forgetting long-term effects such as churn, which may only appear weeks after the test ends.