In this article:
– The definition of a MVT
– How to create a MVT using Kameleoon’s Graphic editor
– How to define the goals and metrics for your MVT
– How to configure the MVT settings, such as targeting options and traffic allocation
– How to launch and monitor your MVT
– How to analyze and interpret the results of your MVT using Kameleoon’s reporting tools (and best practices for ensuring the accuracy and reliability of your MVT results)
All our A/B testing plans include the creation of MVT experiments.
What is a multivariate test or MVT?
While a classical A/B experiment consists of a reference version and a variation, MVT experiments enable you to change several elements on the same page and to analyze which combination generates more conversions. Let’s look at the example of a simple MVT that compares the performance of six CTAs:
In the example, there are 3 colors (blue, red, orange) and 2 wordings: Shop and Add to cart.
With an MVT the 6 possible combinations can be compared.
MVT Glossary
Multivariate test: A multivariate test offers to test the conversion of one or several combination(s) of several elements.
MVT: MVT stands for MultiVariate Test.
Section: A section groups several variations of an element.
Variation: The variation is a suggested change for a given element. It differs from the reference version. A MVT can include several variations that are gathered in sections.
Combination: Combination means that several variations from different sections are combined and displayed to exposed visitors.
Where to create a new MVT experiment?
Via the App
From the Experiment dashboard click on New experiment.
In the screen that appears, choose Multivariate experiment (MVT).
Via the Graphic editor
Once logged in, you can choose Create an MVT experiment on the home screen.
A window opens where you indicate the experiment name. You can also add a description and associated tags:
After validation, you need to set up sections and variations for your MVT.
Set up your MVT sections and variations
Several variations make up a section.
Remember, we are testing the color and wording of a CTA. There are 3 colors (blue, red, orange) and 2 wordings: Shop and Add to cart.
You can create a first section (Section A, you can name it “COLOR”) for colors that includes your reference version as well as the variations red and orange.
At the same time you want to test the CTA’s wording and compare Shop with Add to cart now. You can create a second section (Section B, you can name it “WORDING”) for the wording which will include your reference version as well as the variations Add to cart now.
Click on Create: the MVT version of the graphic editor opens.
The MVT editor
As is the case for classical A/B experiments, you create your variations’ content via the Graphic editor. There is only one slight change: the editor’s header is different from the one you are used to :
In the dropdown menu on the left, you find the sections you have created. Next to it you see the different variations of the selected section.
Sections
Open the section’s dropdown menu. Upon hover over a section’s name, a symbol made of 3 points appears. With a click on the symbol, several actions are suggested.
- Edit the section : return to the page presented above;
- Duplicate the section : duplicates the section and all associated variations;
- Delete the section : deletes the section and all associated variations.
You can also re-name the section by double-clicking on it.
Variations
The menus regarding the variations associated to the selected section are situated next to the section menu. The original version is always displayed and cannot be edited. For the variations, the same options as for classical A/B experiments are available (add an element, URL redirection, etc.)
If you delete all variations associated with a section, you will delete the section itself.
Note: We recommend you introduce changes to no more than 2 elements in your page (i.e. to create a maximum of 2 sections) with 3 to 4 variations of the element.
Why? For an MVT, the variations of each section are associated to find the best combination. The more sections you created the more traffic you need to get usable, reliable results.
The number of possible combinations on a given page can easily get too important. In that case you need a great number of unique visitors to obtain results with statistical significance. To learn more about this subject, please refer to our article on statistical significance.
Kameleoon automatically creates all possible combinations from each variation you design for a given experiment using Full Fractional factorial design, ensuring that all possible combinations are tested simultaneously.
Traffic allocation
Once you have created your variations, you need to finalize your experiment before you can put it online. You can do it exactly the same way as with classic A/B experiments. The only exception is your experiment’s traffic distribution.
Traffic allocation for the experiment
On top of the window, you can choose the percentage of visitors to be exposed in the MVT,
- either by moving the blue cursor
- or by double-clicking on the percentage and editing its value.
By default, 100% of the visitors are exposed to the experiment.
Traffic allocation for exposed visitors
You can choose to distribute your traffic according to:
- sections;
- combinations.
Traffic distribution by sections
Stay on the Sections tab to distribute the traffic according to the sections and the variations they contain.
When you select the Sections tab, it displays a list of blocks. Each block represents a section, all associated variations are listed below.
If you check a section, 100% of exposed visitors will see one of its variations. In other words, visitors exposed to the experiment see one of the checked variations, according to the traffic distribution among variations you defined.
Example:
In the illustration below, 30% of exposed visitors will see a red CTA, 40% will see an orange one.
Edit the traffic allocated to a variation
By default, traffic is equally distributed among the selected variations of a given section.
There are two ways to edit the distribution :
- Type the percentage in the text field at the end of the line dedicated to the variation, using the keyboard;
- Use the pointers to edit the percentage.
Block traffic on a variation
Click on the lock to edit the percentage allocated to a given variation.
Allocate equally
When you click on Allocate equally, it undoes all edits and distributes traffic equally among the selected variations that haven’t been blocked.
Traffic distribution by combination
You can also define the traffic distribution according to the combinations you created. Let’s say you want to test a CTA’s color and the wording which gives you a MVT with the sections A and B:
| Section A: COLOR | Section B: WORDING |
| Variation A0: Original | Variation B0: Original |
| Variation A1: Red | Variation B1: Add to cart |
| Variation A2: Orange |
The generated combinations have the following names:
- A0B0 Original / Original
- A0B1 Original / Add to cart
- A1B0 Red / Original
- A1B1 Red / Add to cart
- A2B0 Orange / Original
- A2B1 Orange / Add to cart
By selecting the Combinations tab, you can define the traffic for each combination.
On top of the page, you see the number of combinations selected for the experiment.
The definition of traffic distribution by combination is more precise than by section, but it is also more time-consuming as the number of combinations can be quite high.
When using the Combinations tab for traffic distribution you can exclude certain combinations. In an MVT with changed colors of background and text for instance, all combinations with the same color for both can be excluded right away.
Edit the traffic allocated to a combination
You must distribute the traffic among all combinations to total 100%.
You can only edit the traffic for a given combination if you select it. By default, all combinations are selected. Just click in the selection box to uncheck the combinations you want to de-select.
Block traffic on a combination
Click on the lock to block the percentage of traffic allocated to a given combination.
Equal traffic distribution
Next to the number of selected combinations, you see the button for equal traffic distribution. Click on it to equally distribute traffic among all selected variations.
The same percentage of traffic is allocated to all selected, unlocked combinations. If combinations are added or deleted, the distribution is automatically adjusted.
Note: By default, the Sections tab is selected. The tabs are independent, i.e. editing the distribution for one of them doesn’t impact the other. The traffic distribution that will be taken into account is the one of the tab (either Sections or Combination) that is selected when you hit the Validate button.
Simulation, launch, analysis: business as usual!
You created your variations and defined the traffic distribution. Well done! The rest is the same as for classical A/B experiments.
Simulate your MVT
Before going live, we recommend you launch a simulation to check that everything meets your expectations.
The simulation panel generates your experiment’s combinations. Click on Currently displayed in simulation to check the display of the combination of your choice.
More about simulation in our dedicated articles.
Launch the MVT
Everything is displayed correctly? The targeting has been set up? Your goals and reporting tools have been defined?
You are ready to publish your MVT experiment! To do so, just hit the Launch button!
Results
If you selected Kameleoon as your reporting tool, your MVT’s results are transferred to the A/B experiment results page you already know. Each combination is considered as a variation (as defined for classical A/B experiments).
Note: You might experience a short latency time (up to 10 minutes) from the time you launch an A/B experiment to its visibility on the website. Don’t worry if your experiment does not appear immediately!
