Results according to the Bayesian method

3

min

quiz

What the Bayesian method promises you

By using the bayesian method of reporting, you are trusting a deductive solution of data analysis. Results are generated faster than with the classical method, and are just as safe, which is ideal for users with low traffic or urgent tests to come up with solutions to their A/B comparisons.

This method combines the actual data generated by the test with the a priori knowledge coming from prior studies or experts opinion. It provides an a posteriori information such as the estimation of future values for the conversion rate and the improvement rate. This anticipation of tendencies is called « Forecast » on the reporting page.

Access to your bayesian results

When you click on « View results » from the Dashboard, you access to the classical reporting page by default.

The Bayesian button is placed at the top of your results page.

To access to the results generated by the bayesian statistics, just switch to “ON”.

Warning ! It is not possible to access to the Bayesian reporting page in the following cases :

  • 100% of your traffic is deviated on the original
  • The number of visitors of your test is equal to 0

Functioning of the reporting page

Structure of the Bayesian reporting page is quite similar to the classical reporting page.

However, some elements differ from it.

New indicators appear such as the probability to beat the orginal, the reliability of results according to Bayes, and the idea of forecast.

Several graphs will disappear on this page, and the conversion rate alone will be displayed.

A few definitions …

Probability to beat the original

This is the probability that a variation has to beat the original page with a higher conversion rate on a given objective.

In the case where the traffic allocated to the original is 0%, the variations, thus sharing 100% of the traffic, do not compete with the original anymore. We will then talk about the « Probability to be the winning variation ».

Reliability of results according to Bayes

It corresponds to the confidence rate attributed to the results. The rate is calculated on a 3-level scale, which is easy to interpret thanks to the legend that automatically appears in the “Reliability” column of the result tables. The results are completely reliable once the 3 squares are full: this means that the reliability rate has stabilized over time. Be careful, to guard against any reversal of trend, it is advisable not to exploit your results before reaching a sufficient reliability rate.

Capture_d_e_cran_2018-10-03_a__15.19.34.png

 

Forecast

Forecast is the anticipation of a value to which will converge the results. This data is only available for both conversion rate and improvement rate.

Forecasting a future value requires to collect a certain amount of data, so this indicator may not appear the first days after the launching of the test.

The forecasts appear directly on the table board, after the actual values of the conversion rate and improvement rate.

The value forecast also appears in the conversion rate graph, it is located in the grey area on the right. It is symbolized by a circle, and takes the color of the variation it represents.

Use the dotted horizontal lines to compare the actual evolution of your conversion rate with its future value.

My results are very different, is it normal?

 

Both statistical methods lead to equivalent results, but they do not insure a perfect similarity between the two. It is thus normal that you happen to observe differences between some rates.

In some cases, two different variations can be announced winners on a same test.

Make sure that the trust rate is at its maximum in both methods before comparing the two data. If they are and that the doubt remains, then we advise you to use the results of the classical method.