A/B testing is based on statistical methods. You don’t need to know all the maths behind, but a little brush up in statistics won’t hurt and certainly improve the chances of your success.
There are 2 main statistical methods behind A/B testing solutions. There aren’t one better than the other, they just have different use. Here is how we handle it with Kameleoon’s statistical engine.
Frequentist approach
Allows a simple read on result reliability thanks to a confidence level: with a level of 95% or more, you have a 95% chance of obtaining the same result should you reproduce the experiment in the same conditions. But this method has a downside: it has a “fixed horizon”, meaning the confidence level has no value up until the end of the experiment.
Bayesian approach
Provides a result probability as soon as the experiment starts. No need to wait until the end of the experiment to spot a trend and interpret the data. But this method also has prerequisites: you need to know how to read the confidence interval given to the estimations during the experiment. With every additional conversion, the trust in the probability of a reliable winning variant improves.
