In this article:
– How to define the goals and metrics for your experiment
– How to create variations of your website pages using Kameleoon’s Graphic editor or Code editor
– How to define the experiment settings, such as targeting options and traffic allocation
– How to launch and monitor your experiment
– How to analyze and interpret the results of your experiment using Kameleoon’s reporting tools (and best practices for ensuring the accuracy and reliability of your results)
Access the finalization panel
Once your variations are created, you are ready for the last step: launching your graphic A/B experiment!
On the right of the header, you will find the “Finalize” button.
It opens the finalization menu where you can complete the three necessary steps (+ one optional) before launching your experiment:
- Allocate traffic to variations
- Set targeting
- Set up reporting tools and goals
- Simulate your experiment (optional)
In the finalization panel, you can also:
Allocate traffic
The first phase of finalizing your A/B experiment is traffic allocation, i.e. the percentage of your visitors who will see your variation(s). By default, the traffic is evenly distributed among the variations; however, you can change this setting.
Two types of views are available: list and graph.
The “List view” shows sliders. In the example above, 33.33% of visitors will see variation 1; 33.33% will see variation 2; and the same goes for the reference version (or original).
To change the traffic allocation, click and hold the mouse down to drag the slider of a variation; alternatively, enter the percentage directly by clicking on the number displayed to the right of the slider. The traffic that you don’t assign to any of your variations will be automatically attributed to “Traffic excluded”.
The “Graph view” represents the allocation of traffic between your variations as a graph. As in the list view, you can change this distribution, either by opening the drop-down menu and selecting or unselecting one or more variation(s); or by clicking on a zone and dragging the slider. You can also enter a percentage directly.
In each of these views, two ON/OFF buttons on the bottom left give access to additional options:
- You can ask Kameleoon to automatically optimize the traffic distribution based on experiment results. This means that the more conversions a variation generates, the more visitors are directed to it.
- You can also perform a multivariate test, or MVT. This enables you to combine several changes in one variation and analyze which combination generates the most conversions for your goal. For example, an MVT with three variations of an image and five variations of a button will test the 15 (3 x 5) possible combinations. Find out more about this kind of experiment
Equal allocation per variation
When you run an experiment on your production website, you have the possibility of specifying the percentage of your traffic to divert to the variations (this is step 2 in the launch pop-in for your experiment).
For example, if you have 3 variations in your experiment, setting a diversion percentage of 75% will tell Kameleoon to divert 75% of your traffic to your variations and 25% to your original page. By default, Kameleoon will then display Variation 1 for 25% of the time, Variation 2 for 25% of the time and Variation 3 for 25% of the time.
Different allocation per variation
You also have the possibility to allocate different parts of your traffic to each variation. To do this, use the sliders to modulate the required percentage for each variation.
You can also click on the percentage and directly enter the value you want to apply to the variation.
At any time, you can return to an equal distribution between the variations by clicking on the “Allocate equally” button, which is just below the list of variations.
Graph view
Click on “Graph view” to switch to the second type of view.
When you hover over the graph, the portion of traffic allocated to each variation will appear. Click to access the percentage and modify it if needed, in the same way as in the list view.
Advanced options
Advanced diversion options are available at the bottom of the pop-in.
Automatically optimize traffic allocation: multi-armed bandit
You can also let Kameleoon automatically manage your traffic allocation, in real time, based on the performance of the variations. Kameloon also features a multi-armed bandit algorithm, based on the epsilon-decreasing approach. This allows you to limit the opportunity cost of an experiment made up of “lost” conversions on the least effective version. This alternative to the traditional A/B testing approach uses adaptive learning to select the best variation among several options. This type of traffic distribution is recommended if you are short of time.
If you check the “Automatically optimize traffic distribution” option, then, based on the first results observed during the experiment period, Kameleoon will disable the worst-performing variations of your experiment and divert traffic from these variations to the best-performing ones.
Note : Dynamic traffic allocation has its limits. It doesn’t take the preferences of each visitor into account; instead, it seeks to maximize the performance of certain variations. If an element has proven successful with a segment of visitors, but that element is part of a low-performing variation, then these visitors will be redirected to a variation that is of interest to the greatest number of visitors.
Read more about dynamic traffic allocation
Traffic reallocation
With the Reallocation option, you can change the distribution of traffic and reset the previous associations of variations seen by visitors. So, returning visitors will go through the variation assignment process again (this is sometimes called bucketing), meaning that a visitor will potentially see several variations during their visits. This could be risky, so use this tactic with caution!
- If you check a variation, the visitors previously associated with this variation will be re-bucketed. This means they will be likely to see other variations than the one they already visited.
- If you change the traffic allocated to a variation (without checking it), the changes in distribution will only affect new visitors. Returning visitors for this unchecked variation will still see the old variation.
Once you’ve made the changes, validate to reallocate the traffic.
Define the targeting
In the finalization phase of an experiment, after allocating traffic to each variation, you must define the targeting.
Read our documentation on the subject!
Set up reporting tools
To analyze and track your A/B experiment, several systems are available. Kameleoon integrates all major Web Analytics solutions (Google Analytics, AT Internet, Webtrends, Adobe Analytics, Eurelian, etc.).
Available tools
To access the Integrations page, log in to your Kameleoon App.
Click “Administrate” > “Integrations” in the left menu.
On this page, you will find the list of all solutions you can integrate with Kameleoon.
Kameleoon’s reporting tool is already configured by default – so you will not find it in this list!
For each tool, the status on the right indicates you if the solution has been set up or not:
- If an “Install” icon is displayed, the tool has not been set up; you will not be able to select it as reporting tool;
- If “OFF” is displayed, the tool is set up but disabled; you will not be able to select it as reporting tool;
- If “ON” is displayed, the tool is set up and enabled; you will be able to select it for the selected website(s);
- If a padlock is displayed, your package does not give you access to this tool.
Add a new reporting tool
Click on the tool of your choice to set it up and add it to your integrations.
In the menu that unfolds click on the “Install the tool” button.
On the left, select the website(s) on which you want to set up the tool.
On the right, enter the settings specific to the selected tool (if needed).
Finally, click on the “Validate” button to add the tool.
Enable/Disable a tool
To disable a tool, simply unselect all websites on which the tool is configured.
On the contrary, to enable a tool, select one or several websites on which you want to configure it.
Set up conversion goals
What is a goal?
To use Kameleoon reporting, you must define a conversion goal. This goal is what you want to improve with your A/B experiment.
Several goals are available:
- Engagement: this goal is achieved if the visitor visits other pages after the landing page;
- Click tracking: this goal is achieved if the visitor clicks on a specific element you defined;
- Scroll tracking: this goal is achieved if the visitor scroll beyond a specific part of your page;
- Access to a page: this goal is achieved if the visitor reaches a page of your choice;
- Number of page viewed: this goal is achieved if the visitor visits the number of pages of your choice;
- Time elapsed: this goal is achieved if the visitor spends a predefined amount of time on your website;
- Custom goal: for more complex goals, you can create custom goals via a Kameleoon API call.
For further information about Kameleoon goals, please read this article.
Create a new goal
To create a new goal, go to “Configure” > “Goals” in the left menu.
Click on the “New goal” button to set up your new goal.
The following pop-in will open:
Select the website for which you want to create a goal, then the type of goal you want to create. Click on the “Next” button to go to the next step.
Enter the name of your goal and all the parameters related to the type of goal selected. For example, if you chose the “Access to a page” goal, the following pop-in will display:
For all types of goals, advanced settings are available.
For more information about goal creation, please read this documentation.
Don’t forget to click on “Create the goal”!
Simulate
Simulation mode allows you to check if:
- Your variations or personalizations are displayed correctly;
- The targeting of your campaign is configured correctly, and if not, understand why;
- The goals you have defined convert or not;
- Your different visitors see the right content for them at the right time.
For more information about simulation, please read this documentation.
Simulate an experiment and preview it are two different actions. To learn more about the preview, you can read our article on the subject.
Estimate the duration
In the finalization panel of the editor, it is possible to estimate the duration of an experiment thanks to the button 
- Daily traffic on your website (average number of visitors per day on the tested pages);
- Conversion rate of the measured goal (the current conversion rate of the goal, which will be used as a reference);
- Necessary reliability index (by default, it is 95%, but you can change its value);
- Necessary improvement rate.
Note: This is an estimation; once your experiment is launched, the reliability index will let you know if your results are reliable or not.For more information, you can consult our documentation on the results page.
[If some of these terms leave you speechless, do not hesitate to consult our lexicon to learn more about the data processed by Kameleoon.]
Then, click on “Close” to go back.
The estimator automatically takes into account the traffic allocation and the number of variations.
Launch
Launch immediately
When all of the finalization steps are completed, a green “check” icon appears as well as a summary of the chosen parameters. To be able to run the experiment, the first three steps must be validated. The simulation step is strongly recommended but not mandatory. It helps check your variation’s display, your experiments’ targeting and whether the defined goals lead to conversion.
When your are satisfied with your variations and your experiment is set, you can click to launch it.
A pop-in “Experiment Summary”allows you to check if all of you experiment settings are okay.
If you want to change some settings, you can edit them by clicking on the pen icon.
If everything seems fine to you, click on the “Launch” button on the bottom right.
Congratulations, your experiment is now online ! A pop-in displays to confirm it to you.
Note: There may be a short latency time (up to 10 minutes) between the launch of an A/B experiment and its visibility on the website. Don’t worry if your experiment does not appear immediately!
Schedule
In the Kameleoon editor, you can schedule your experiment by defining a starting date, an ending date or both.
To do so, once your variations are created, follow the steps on the finalization panel, then on the summary pop-in click on “Schedule” instead of “Launch”.
A new page will open allowing you to schedule your experiment.
“Advanced schedule” allows you to set the time zone and/or set an automatic experiment stop. The automatic stop can take place when the reliability rate has reached the configured value and has stabilized; or when the traffic has reached a certain threshold, which you must define.
We recommend you not to define an end date before launching your A/B experiment, as only the trust rate will tell you if your experiment can be stop of if it should still run before having reliable results. However it can be very useful if you want to run your experiment on a specific period or event for example. In all cases, we recommend you to check the trust rate before reading your results.
Update and Configuration
The finalization panel adapts during the creation and configuration of your experiments. The black button changes depending on the situation.
- “Finalize” when your experiment has not yet started;
- “Update” when your experiment is already online or paused;
- “Configuration” when your experiment is stopped.
Finalize a draft or scheduled experiment
As long as you have not yet launched your experiment, the available button is “Finalize”.
When editing a scheduled experiment, it is necessary to save your changes with the “Save” button in the right panel.
A pop-in reminds you of the need to save any changes in order to see it applied.
Update an online experiment
If your experiment is online or paused, the button becomes “Update”.
Several options are available:
Pause
Stop
Schedule the experiment on a particular date range
Access the results
As soon as you make a change, in the panel the check on the right of the item turns gray (instead of green), and the “Save” button is clickable. Click to save the change(s).
At the top of the menu, just above the previously detailed icons, a sentence confirms that the update is taken into account.
If you leave the window without saving your changes, a pop-in alert asks you to confirm your choice. There is still time to click on the button!
Set up a stopped experiment
If your experiment is stopped, the button becomes “Configuration” and only the simulation and the consultation of the results are available.
Click on the “Summary” button to display the summary pop-up of your experiment. No saving is available because of the status of the experiment.
