Metrics & analytics · Hard
Your A/B test shows a small lift in conversion, but it is not statistically significant. What do you do?
Short answer
First check the test was valid: correct randomisation, enough sample size for the effect you care about and no tracking issues. If it was underpowered, decide whether a longer run is worth the cost. If it was properly powered, treat it as 'no detectable effect' and decide on other grounds: cost, risk, strategic fit and qualitative evidence. Never declare a win from noise.
Detailed answer
Step 1: Validate the test
- Sample ratio: are control and treatment groups the expected sizes? A mismatch suggests a bug.
- Instrumentation: are both variants logging events the same way?
- Novelty or timing effects: did a sale, holiday or outage overlap the test?
- Power: was the test sized to detect the minimum effect that would matter to the business? If the sample was too small, a non-significant result is uninformative, not negative.
Step 2: Decide what the result means
- Underpowered: estimate how long a properly powered run would take. If that is weeks of opportunity cost for a small, low-risk change, it may be better to make the decision on judgement.
- Well powered, no significant effect: the honest reading is that any effect is smaller than you can detect. That is a result.
Step 3: Make the decision
Consider:
- Cost to maintain: if the variant adds complexity, the default is to not ship.
- Secondary metrics and guardrails: did anything get worse?
- Segments: look for pre-registered segments only. Hunting through dozens of slices after the fact will eventually find a "significant" one by chance.
- Strategic or qualitative reasons: a change that makes the product more accessible, or a step towards a larger planned change, might be shipped even with a neutral result.
Worked phrasing
"Assume the test was sized to detect a lift of around one percentage point (an illustrative threshold). If it was properly powered and we saw no significant effect, I would not ship on the basis of a lift, and I would decide based on maintenance cost and guardrails."
Common pitfalls
- Extending a test until it turns significant ("peeking").
- Slicing into segments until something looks good.
- Treating "not significant" as proof of no effect when the test was underpowered.