Assume you have two website designs – A & B on your eCommerce website, and you end up with 45 conversions out of 100 visitors for design A and 50 conversions out of 100 visitors for design B.
What’s the chance that design B is better than design A?
10%? No, that’s wrong. Design B is actually 76% better than design A and to make the switch, this probability has to be > 90%. Part-2 above also provides a shortcut formula to make this calculation.
The below three part series provide very good English and Math explanation on how to evaluate results from split testing on two designs.