The duration of an A/B experiment depends on the kind of experiment running and on the traffic.
Let’s imagine an experiment with such a positive modification that the first visitors of the variation make a conversion. You will see very quickly how efficient the experiment is. But this kind of result is very rare.
Most of the time, the conversion rate does not change a lot right after the launching, but if you keep your experiment running long enough, the variation will be seen by hundreds of thousands or even millions of visitors, and the results will be relevant. We recommend to use the z-score, a mathematical tool calculating the statistical significance of your result. As you use it, you will learn to detect how many chances a variation has to perform better than another.
Note: If you are using Google Analytics or Kameleoon internal reporting tool, Kameleoon will calculate automatically the significance of your experiment according to the goals you previously set up in your back-office. For further information, you can read our article about Statistical significance.
Estimate the duration of an A/B experiment with Kameleoon
You can estimate the duration of your A/B experiment from Kameleoon editor. To do this, click on the Finalize button, located on the top right of the Kameleoon Graphic editor.
The finalization page opens. Click on the 3-dot menu next to the Launch button.
Then, click on Estimate the duration.
To have an estimation of the duration, you need to fill in some information.
- Average number of daily visitors: traffic in number of daily visitors on the tested page(s).
- Current conversion rate (Original): conversion rate measured on your website for the goal used as a reference.
- Desired reliability: required confidence to consider the experiment as significant. For example, a significance level at 95% means that if the variation is winning, there is a 5% chance that it is not actually improving the conversion rate.
- Desired improvement rate: the improvement expected between the conversion rate of the reference page and the conversion rate of the variation.
The estimator automatically takes into account the traffic allocation and the number of variations.
Once you filled in these information, click on Calculate and the estimate duration of your A/B experiment will display. You can then close the pop-in to go back to the edition of your experiment.
