Understanding statistical significance

Understanding statistical significance

"Statistical significance" is the chance that the difference you see between variants is real, not just noise from random sampling. Netaj uses a 95% confidence threshold by default — meaning if a result is significant, there's less than a 5% chance it's a fluke.

What significance is not

Significance does not tell you the magnitude of the effect. A variant that converts 0.1% better can become "significant" given enough traffic. Always check the lift size, not just the p-value.

How long should you wait?

You need enough conversions for the test to settle. As a rule of thumb: at least 200 conversions per variant before you trust the number. Less than that, and the result will swing wildly week to week.

Don't peek and stop early

If you watch results in real time and stop the moment a variant looks like it's winning, you'll declare false winners constantly. Wait for the result to be significant and for your minimum sample to be reached.