Skip to content

Metrics & analytics · Hard

Your A/B test shows a small lift in conversion, but it is not statistically significant. What do you do?

Short answer

First check the test was valid: correct randomisation, enough sample size for the effect you care about and no tracking issues. If it was underpowered, decide whether a longer run is worth the cost. If it was properly powered, treat it as 'no detectable effect' and decide on other grounds: cost, risk, strategic fit and qualitative evidence. Never declare a win from noise.

Detailed answer

Step 1: Validate the test

  • Sample ratio: are control and treatment groups the expected sizes? A mismatch suggests a bug.
  • Instrumentation: are both variants logging events the same way?
  • Novelty or timing effects: did a sale, holiday or outage overlap the test?
  • Power: was the test sized to detect the minimum effect that would matter to the business? If the sample was too small, a non-significant result is uninformative, not negative.

Step 2: Decide what the result means

  • Underpowered: estimate how long a properly powered run would take. If that is weeks of opportunity cost for a small, low-risk change, it may be better to make the decision on judgement.
  • Well powered, no significant effect: the honest reading is that any effect is smaller than you can detect. That is a result.

Step 3: Make the decision

Consider:

  • Cost to maintain: if the variant adds complexity, the default is to not ship.
  • Secondary metrics and guardrails: did anything get worse?
  • Segments: look for pre-registered segments only. Hunting through dozens of slices after the fact will eventually find a "significant" one by chance.
  • Strategic or qualitative reasons: a change that makes the product more accessible, or a step towards a larger planned change, might be shipped even with a neutral result.

Worked phrasing

"Assume the test was sized to detect a lift of around one percentage point (an illustrative threshold). If it was properly powered and we saw no significant effect, I would not ship on the basis of a lift, and I would decide based on maintenance cost and guardrails."

Common pitfalls

  • Extending a test until it turns significant ("peeking").
  • Slicing into segments until something looks good.
  • Treating "not significant" as proof of no effect when the test was underpowered.