What does the Bayesian method promise you?
The Bayesian approach is a different model of thought where instead of considering the parameters we wish to estimate as unknown constants, we model each parameters as a random variable. It takes its name from the Bayes’ rule and it allows us to compute quantities which are not available in the frequentist framework.
The Bayes’ rule combine the experiments results with any prior information we might have, although in the case of A/B testing we use non informative priors. By choosing carefully the a priori distribution we can have a convenient posterior one. We can then leverage the posterior distribution to compute, for example, the probability for the variant to be better than the original.
Access your Bayesian results
When you click on “Results” from the dashboard, you are taken by default to the classic results page.
To access the results generated by Bayesian statistics, click on the “Actions” menu at the top right of the page > “Enable bayesian”.
It is not possible to access the Bayesian report page in the following cases: 100% of your traffic is diverted to the original; the number of visitors to your experiment is 0.
The Bayesian results page
Structure of the Bayesian reporting page is quite similar to the classic results page.
However, some elements are different:
- New indicators appear such as the probability of beating the original, the reliability of the results according to Bayes.
- Several graphs do not appear on the page, and only the conversion rate is displayed.
A few definitions
Probability to beat the original
This is the probability that a variation will beat the original page with a higher conversion rate for a given goal.
In the case where the traffic allocated to the original is 0%, the variations, sharing 100% of the traffic, do not compete with the original. We will then talk about the “Probability to be the winning variation”.
Reliability of results according to Bayes
This corresponds to the confidence rate attributed to the results. This rate is calculated on a 3-level scale, easy to interpret thanks to the legend that appears automatically in the “Reliability” column of the results tables. The results are totally reliable when the 3 boxes are full: this means that the reliability rate has stabilized over time. Be careful, to avoid any reversal of trend, it is advised not to exploit your results before having reached a sufficient reliability rate.
My results are very different, is it normal?
Both statistical methods lead to equivalent results, but they do not guarantee a perfect similarity between the two. It is therefore normal that you may observe differences between certain rates.
In some cases, two different variations can be announced winners on a same experiment.
Make sure that the confidence levels are at their maximum in both methods before comparing the two data. If it is and there is still doubt, then we recommend that you use the results of the classic method.
Further reading
If you would like more details on how Kameleoon’s statistic engine works, you can read our Statistical paper.
