What does the Bayesian method promise you?
The Bayesian approach is a different model of thought where instead of considering the parameters we wish to estimate as unknown constants, we model each parameters as a random variable. It takes its name from the Bayes’ rule and it allows us to compute quantities which are not available in the frequentist framework.
The Bayes’ rule combine the experiments results with any prior information we might have, although in the case of A/B testing we use non informative priors. By choosing carefully the a priori distribution we can have a convenient posterior one. We can then leverage the posterior distribution to compute, for example, the probability for the variant to be better than the original.
Access your Bayesian results
When you click on Results from the dashboard, you are taken by default to the classic results page.
To access the results generated by Bayesian statistics, click on the Actions menu at the top right of the page > Enable bayesian.
Note: You can’t access the Bayesian results page in the following cases: 100% of your traffic is diverted to the original; the number of visitors to your experiment is 0.
The Bayesian results page
Structure of the Bayesian reporting page is quite similar to the classic results page.
However, some elements are different:
- New indicators appear such as the probability of beating the original, the reliability of the results according to Bayes.
- Several graphs do not appear on the page, and only the conversion rate is displayed.
Note: If you switch to “All conversions” then the number displayed in the Improvement rate column will not be the observed uplift but the estimated mean of the posterior distribution. This value is the observed uplift adjusted toward our prior.
A few definitions
Probability to beat the original
This is the probability that a variation will beat the original page with a higher conversion rate for a given goal.
In the case where the traffic allocated to the original is 0%, the variations, sharing 100% of the traffic, do not compete with the original. We will then talk about the “Probability to be the winning variation”.
Reliability of results according to Bayes
This corresponds to the confidence rate attributed to the results. This rate is calculated on a 3-level scale, easy to interpret thanks to the legend that appears automatically in the Reliability column of the results tables. The results are totally reliable when the 3 boxes are full: this means that the reliability rate has stabilized over time. Be careful, to avoid any reversal of trend, it is advised not to exploit your results before having reached a sufficient reliability rate.
Bayesian continuous metrics
When you hover over non-binomial metrics, such as Revenue per visit/visitor or Average cart value, an overlay appears.
- Improvement rate: This value is the estimated average lift. It is the mean of the posterior distribution, not the observed uplift and it indicates the rate of improvement for the metric. It will be displayed in green if positive and red if negative, alongside a credible interval.
- Probability to win over reference: This value shows the probability that the variation will outperform the control. It will be displayed in green if it is higher than the reliability threshold set in the configuration, otherwise, it will be red.
- Credible Interval Table: The min and max values are the bounds of the credible interval around the improvement rate. There is 95% chance for the improvement rate to be between those values. The values are displayed in green if the are positives, and in red when they are negatives.
If you would like more details on how Kameleoon’s statistic engine works, you can read our Statistical paper.
My results are very different, is it normal?
Both statistical methods lead to equivalent results, but they do not guarantee a perfect similarity between the two. It is therefore normal that you may observe differences between certain rates.
In some cases, two different variations can be announced winners on a same experiment.
Make sure that the confidence levels are at their maximum in both methods before comparing the two data. If it is and there is still doubt, then we recommend that you use the results of the classic method.
Further reading
If you would like more details on how Kameleoon’s statistic engine works, you can read our Statistical paper.
