Skip to main content

Finalizing an experiment

Watch this video in our academy for more information about the differences between A/B and multivariate tests.

Access the finalization page​

Once you create your variations, you are ready to launch your experiment.

In the header's right side, you will find the Finalize button. Clicking the button opens the finalization page where you can complete these steps before launching your experiment:

On the finalization page, you can also:

  • Estimate an experiment's duration
  • Schedule an experiment

Define targeting​

The first step in finalizing your A/B experiment is defining targeting.

note

For more information on defining segments and triggers, refer to this article.

Create a new segment​

Read our documentation on segments

Associate a segment to your experiment​

  1. On the Finalization page, click Targeting > Target a segment.
  2. Select your segment from the list.
  3. Click Next to validate.

Target by webpage​

You can also define targeting based on your site’s pages.

A specific page​

You can target visitors based on the exact URL of the page they’re visiting. Specify the page’s URL in the text box. When you select this option, the targeting will apply not only to the specified URL, but also to any version of the URL that includes parameters (for example, query strings or hash fragments). So, if a visitor is on the specified URL with any additional parameters, they will still be targeted. For example, targeting www.example.com/product will also match www.example.com/product?ref=homepage.

The URLs containing a specific fragment​

You can target visitors who are on a page whose URL contains a specific fragment. Specify the fragment in the text field. This option is useful when you want to include multiple pages or variations of a URL without listing each one individually. For example, targeting the fragment /product will match URLs like www.example.com/product/123, www.example.com/category/product?, or www.example.com/product-review.

The URLs of all modified pages​

You can target the pages that have been modified as part of your experiment or personalization. When you select this option, the campaign will apply to the URLs of all pages where changes have been made, based on the modifications configured in the editor.

The entire site​

If you want your campaign to run across all of your site’s pages, select The entire site. When you select this option, every page within the project scope will be included, regardless of the URL structure or parameter.

note

Ensure you use this option only when the campaign is relevant to users’ experience across your entire site.

Distribute traffic​

The second step of finalizing your A/B experiment is traffic allocation. By default, traffic is evenly distributed among your variations; however, you can change this setting.

In the example above, 33.33% of visitors will see variation 1; 33.33% will see variation 2; and 33.33% of visitors will see the original variation.

To change the traffic allocation:

  • Click and drag the slider next to a variation.

OR

  • Click the number to the right of a slider and enter your desired percentage.

Click Next to validate this step.

Excluded traffic​

The traffic that you don't assign to any of your variations will be automatically attributed to Excluded traffic. These visitors will see your page's original versions.

Equal allocation per variation​

Specify the percentage of traffic to divert to experiment variations.

For example, with three variations, a 75% diversion percentage allocates 75% of traffic to the variations and 25% to the original page. Kameleoon will then display each variation equally (25% of the time).

Different allocation per variation​

To allocate different traffic percentages to each variation, use the sliders to adjust the desired percentage for each.

You can also click the percentage and enter the value you want to apply to the variation.

At any time, you can return to an equal distribution between the variations by clicking the Allocate equally, which is just below the list of variations.

Automatically optimize traffic allocation: multi-armed bandit​

You can also let Kameleoon automatically manage your traffic allocation in real time, based on the varaitions' performance. Kameloon also features a multi-armed bandit algorithm, based on the epsilon-decreasing approach. This feature allows you to limit the opportunity cost of an experiment made up of "lost" conversions on the least effective version.

If you check the Automatically optimize traffic distribution option, then, based on the first results observed during the experiment period, Kameleoon will disable your experiment's worst performing variations and divert traffic from these variations to the best performing variation(s).

note

Dynamic traffic allocation optimizes variation performance but doesn't consider individual visitor preferences. It prioritizes high-performing variations, potentially redirecting visitors away from a successful element within a lower-performing variation.

Contextual bandits​

Contextual bandits dynamically optimize traffic allocation in experiments using machine learning. They adapt in real-time to redistribute traffic based on variation performance and user context to maximize effectiveness.

Key differences exist between multi-armed bandits and contextual bandits. Understanding these differences is critical for choosing the best method for your experiments:

  • Multi-armed bandits:
    • Multi-armed bandits optimize traffic distribution among multiple variations (arms) to maximize a defined goal, such as click rates or conversions.
    • Multi-armed bandits treat all users equally; no distinction is made based on user attributes.
    • Ideal for scenarios where user-specific data is unavailable or unnecessary, and the focus is on finding the best-performing variation for the overall audience.
  • Contextual bandits:
    • Contextual bandits incorporate additional user-specific data, like device type, location, or behavior, into decision-making.
    • Contextual bandits allow more personalized decisions, tailoring variations to specific users for improved outcomes.
    • The variability introduced by user attributes allows contextual bandits to optimize decisions in dynamic environments.

So, multi-armed bandits optimize traffic allocation uniformly across users, while contextual bandits leverage contextual data to make personalized decisions.

If you would like more details on how Kameleoon’s contextual bandits work, read our Statistical paper.

Configuring contextual bandits​

To enable contextual bandits:

  1. Navigate to the Finalization panel.
  2. Click Traffic allocation.
  3. Click the dropdown menu beneath Select the allocation method.
  4. Click Contextual bandit.

By default, the algorithm will begin optimizing traffic based on predefined user attributes available in Kameleoon.

However, to fully leverage the power of contextual bandits—especially with the Contextual Bandit Premium feature included in the AI Predictive Targeting add-on (paid or trial)—you can provide custom data as additional input to the machine-learning model. Custom data allows the algorithm to make even more accurate predictions by incorporating business-specific attributes (for example, CRM segments, purchase history, in-app behavior).

To activate the use of custom data in your experiment:

  • Go to your custom data configuration panel.
  • Enable Use this custom data as input for AI Predictive Targeting.

Once enabled, Kameleoon’s algorithm will use these attributes as part of the decision-making process to deliver the most relevant variation to each visitor.

To learn more about AI Predictive Targeting, read our article on the subject here.

Advanced reallocation​

The Advanced reallocation feature allows you to redistribute traffic among variations in your experiment. When applied, the traffic allocation is reset, and visitors who had previously seen a specific variation will be treated as new visitors. This can be particularly useful when you want to focus on a subset of variations or exclude certain variations from receiving further traffic.

Click on the Advanced reallocation option located at the top-right of the traffic distribution step. In the panel that appears, you can choose which variations will be part of the reallocated traffic.

This reallocation will be effective once you'll have clicked on the Reallocate button and then on the Save button in the top-right of the page.

Define goals​

This step is mandatory unless you configured an integration (reporting tool).

Select one or several goals to activate Kameleoon as a reporting tool.

Available goals​

To use Kameleoon as a reporting tool, you must define a conversion goal. A goal is what you want to improve with your A/B experiment.

Several goals are available:

  • Engagement: This goal is achieved if the visitor visits other pages after the landing page.
  • Click tracking: This goal is achieved if the visitor clicks on a specific element you defined.
  • Scroll tracking: This goal is achieved if the visitor scrolls beyond a specific part of your page.
  • Access to a page: This goal is achieved if the visitor reaches a page of your choice.
  • Number of pages viewed: This goal is achieved if the visitor visits a certain number of pages.
  • Time elapsed: This goal is achieved if the visitor spends a predefined amount of time on your website.
  • Custom goal: For more complex goals, you can create custom goals via a Kameleoon API call.

Create a new goal​

To learn how to add a new goal, read this article.

Associate a goal to your experiment​

Once you have created a goal, you need to associate it with your experiment.

  1. Click Goals in the Finalization page.
  2. Find your goal and click it.
  3. Click Next to validate this step.

Set up reporting tools​

This step is mandatory unless you configured a goal.

Add a new integration​

To learn how to add a new integration, read this article.

Activate an integration on an experiment​

Once you've added a reporting tool to your list of integrations on the Integrations page, you can associate it with a campaign.

To do this:

  1. Click Integrations in the Finalization page.
  2. Find your desired tool and click it.
  3. Click Next to validate this step.

Simulate​

Simulation mode allows you to check if:

  • Your variations or personalizations are displayed correctly.
  • Your campaign's targeting is configured correctly, and if not, understand why.
  • The goals you have defined convert.
  • Your different visitors see the right content at the right time.

For more information about simulation, read this article.

Estimate an experiment's duration​

In the finalization panel of the editor, it is possible to estimate an experiment's duration.

To do this, you must provide certain information:

  • Average number of visitors per day visiting the tested pages - This is the amount of visitors your test will target daily across all the test's variations.
  • Current conversion rate of the goal (which will be used as a reference) - This is an estimation of the current conversion rate of the main goal of the experiment you are trying to improve.
  • Minimum Detectable Effect (MDE) - This is the smallest change in the goal metric you aim to identify. It's calculated relative to the control variation's mean. For example, with a control conversion rate of 1%, a 5% MDE means you can detect if the rate falls below 0.95% or rises above 1.05%.
  • Desired reliability rate (by default, it's 95%, but you can change its value) - This setting lets you balance the risk of detecting an improvement which is not real - a false positive result. A common value is 95%. Increasing this parameter will lower your risk of getting a false positive result at the cost of increasing the required number of visitors needed to detect the same change.
note

This is an estimation; once your experiment is launched, the reliability index will inform you if your results are reliable. For more information, you can consult our documentation on the Results page.

The estimator automatically accounts for the traffic allocation and the number of variations.

note

You can also use our free A/B testing duration calculator to get a more precise estimate based on your traffic and conversion goals.

Launch​

Launch immediately​

When you complete all mandatory finalization steps, a green check icon appears.

We strongly recommend simulating your experiment, but it's not mandatory. Simulating checks your variations' display, your experiment's targeting, and whether the defined goals lead to conversions.

When you are satisfied with your variations and experiment, click Launch.

A Configuration summary panel allows you to check your experiment settings.

To modify settings, click the settings icon > Edit.

When you are satisfied with your settings, click Launch in the bottom-right.

Your experiment is now online.

note

There may be a short latency time (up to 10 minutes) between the launch of an A/B experiment and its visibility on your website. Don't worry if your experiment does not appear immediately.

Schedule​

You can schedule your experiment by defining a starting date, an ending date or both.

To do so, you can either click the three-dots menu > Schedule:

Or click Schedule at the bottom of the Configuration summary.

A panel will open allowing you to schedule your experiment.

Advanced schedule allows you to set the experiment time zone and/or automate the experiment's conclusion. Automatic stops can be triggered when the reliability rate reaches and stabilizes at the configured value, or when the experiment reaches a defined traffic threshold.

We recommend you avoid setting an end date for A/B experiments before launch. The confidence rate is the primary indicator of whether an experiment can be stopped or should continue before conclusive results are obtained. However, defining an end date can be beneficial for experiments tied to specific events or timeframes. Regardless of whether an end date is set, always review the confidence rate before analyzing experiment results.