What does the Bayesian method promise you?
By using the bayesian method of reporting, you are trusting a deductive solution of data analysis. Results are generated faster than with the classical method, and are just as safe, which is ideal for users with low traffic or urgent experiments to come up with solutions to their A/B comparisons.
This method combines the actual data generated by the experiment with the a priori knowledge coming from prior studies or experts opinion. It provides an a posteriori information such as the estimation of future values for the conversion rate and the improvement rate. This anticipation of tendencies is called « Forecast » on the reporting page.
Access your Bayesian results
When you click on « View results » from the Dashboard, you access to the classical reporting page by default.
The Bayesian button is placed at the top of your results page.
To access to the results generated by the bayesian statistics, just switch to “ON”.
Warning ! It is not possible to access to the Bayesian reporting page in the following cases :
- 100% of your traffic is deviated on the original
- The number of visitors of your experiment is equal to 0
The Bayesian results page
Structure of the Bayesian reporting page is quite similar to the classical reporting page.
However, some elements differ from it.
New indicators appear such as the probability to beat the orginal, the reliability of results according to Bayes, and the idea of forecast.
Several graphs will disappear on this page, and the conversion rate alone will be displayed.
A few definitions
Probability to beat the original
This is the probability that a variation has to beat the original page with a higher conversion rate on a given goal.
In the case where the traffic allocated to the original is 0%, the variations, thus sharing 100% of the traffic, do not compete with the original anymore. We will then talk about the « Probability to be the winning variation ».
Reliability of results according to Bayes
It corresponds to the confidence rate attributed to the results. The rate is calculated on a 3-level scale, which is easy to interpret thanks to the legend that automatically appears in the “Reliability” column of the result tables. The results are completely reliable once the 3 squares are full: this means that the reliability rate has stabilized over time. Be careful, to guard against any reversal of trend, it is advisable not to exploit your results before reaching a sufficient reliability rate.
Forecast is the anticipation of a value to which will converge the results. This data is only available for both conversion rate and improvement rate.
Forecasting a future value requires to collect a certain amount of data, so this indicator may not appear the first days after the launching of the experiment.
The forecasts appear directly on the table board, after the actual values of the conversion rate and improvement rate.
The value forecast also appears in the conversion rate graph, it is located in the grey area on the right. It is symbolized by a circle, and takes the color of the variation it represents.
Use the dotted horizontal lines to compare the actual evolution of your conversion rate with its future value.
My results are very different, is it normal?
Both statistical methods lead to equivalent results, but they do not insure a perfect similarity between the two. It is thus normal that you happen to observe differences between some rates.
In some cases, two different variations can be announced winners on a same experiment.
Make sure that the trust rate is at its maximum in both methods before comparing the two data. If they are and that the doubt remains, then we advise you to use the results of the classical method.