Results according to the Bayesian method

portrait de l'auteur Julie Trenque

Written by Julie Trenque

Updated on 22/09/2020

2 min

Intermediate

Was this content useful?

What does the Bayesian method promise you?

By using Bayesian reporting, you are relying on a deductive data analysis solution. Results are generated faster than with the classic method, and are just as reliable, which is ideal for users with low traffic or experiments that need to be launched urgently.

This method combines the actual data generated by the experiment with the a priori knowledge coming from prior studies or experts opinion. It provides an a posteriori information such as the estimation of future values for the conversion rate and the improvement rate. This anticipation of tendencies is called « Forecast » on the reporting page.

Access your Bayesian results

When you click on “Results” from the dashboard, you are taken by default to the classic results page.

To access the results generated by Bayesian statistics, click on the “Actions” menu at the top right of the page > “Enable bayesian”.

It is not possible to access the Bayesian report page in the following cases: 100% of your traffic is diverted to the original; the number of visitors to your experiment is 0.

The Bayesian results page

Structure of the Bayesian reporting page is quite similar to the classic results page.

However, some elements are different:

  • New indicators appear such as the probability of beating the original, the reliability of the results according to Bayes, and the forecast.
  • Several graphs do not appear on the page, and only the conversion rate is displayed.

A few definitions

Probability to beat the original

This is the probability that a variation will beat the original page with a higher conversion rate for a given goal.

In the case where the traffic allocated to the original is 0%, the variations, sharing 100% of the traffic, do not compete with the original. We will then talk about the “Probability to be the winning variation”.

Reliability of results according to Bayes

This corresponds to the confidence rate attributed to the results. This rate is calculated on a 3-level scale, easy to interpret thanks to the legend that appears automatically in the “Reliability” column of the results tables. The results are totally reliable when the 3 boxes are full: this means that the reliability rate has stabilized over time. Be careful, to avoid any reversal of trend, it is advised not to exploit your results before having reached a sufficient reliability rate.

Forecast

The forecast is the anticipation of a value towards which the results will converge. This data is only available for the conversion rate and the improvement rate.

The prediction of a future value requires the collection of a certain amount of data. This indicator may therefore not appear the first few days after the experiment is launched.

The forecasts appear directly on the table, after the actual values of the conversion rate and the improvement rate.

The forecast value also appears in the conversion rate graph, it is located in the grey area on the right. It is symbolized by a circle, and takes the color of the variation it represents.

Use the horizontal dotted lines to compare the actual evolution of your conversion rate with its future value.

My results are very different, is it normal?

Both statistical methods lead to equivalent results, but they do not guarantee a perfect similarity between the two. It is therefore normal that you may observe differences between certain rates.

In some cases, two different variations can be announced winners on a same experiment.

Make sure that the confidence levels are at their maximum in both methods before comparing the two data. If it is and there is still doubt, then we recommend that you use the results of the classic method.

  • In this article :