How long does an A/B test should last?




The duration of an A/B test depends on the kind of test running and on the traffic.

Let’s imagine a test with such a positive modification that the first visitors of the variation make a conversion. You will see very quickly how efficient the test is. But this kind of result is very rare.

Most of the time, the conversion rate does not change a lot right after the launching, but if you keep your test running long enough, the variation will be seen by hundreds of thousands or even millions of visitors, and the results will be relevant. We recommend to use the z-score, a mathematical tool calculating the statistical significance of your result. As you use it, you will learn to detect how many chances a variation has to perform better than another.

Note: if you are using Google Analytics or Kameleoon internal reporting tool, Kameleoon will calculate automatically the significance of your test according to the goals you previously set up in your back-office. For further information, you can read our article about Statistical significance.

Estimating the duration of an A/B test with Kameleoon

You can estimate the duration of your A/B test from Kameleoon editor. To do this, click on the “Finalize” button, located on the top right of the Kameleoon editor.

Then, click on the clock icon to estimate the duration of the test.

To have an estimation of the duration, you need to fill in the following information:

  • Daily traffic: traffic in number of daily visitors on the tested page(s).
  • Conversion rate for measured goal: conversion rate measured on your website for the goal used as a reference.
  • Necessary reliability index: required confidence to consider the test as significant. For example, a significance level at 95% means your variation has 95% chances to beat the original.
  • Necessary improvement rate: the improvement expected between the conversion rate of the reference page and the conversion rate of the variation.

Once you filled in these information, click on “Calculate” and the estimate duration of your A/B test will display. You can then close the pop-in to go back to the edition of your test.