The creation of MVT tests is included in all our A/B testing plans.
What is a multivariate test or MVT?
While a classical A/B test consists of a reference version and a variation, MVT tests enable you to change several elements on the same page and to analyze which combination generates more conversions. Let’s look at the example of a simple MVT that compares the performance of six CTAs:
In the example, there are 3 color variations (red, green, blue) and 2 wordings: “Download Firefox” and “Download now”.
With an MVT the six possible combinations can be compared.
Multivariate test: A multivariate test offers to test the conversion of one or several combination(s) of several elements.
MVT: MVT stands for MultiVariate Test.
Section: A section groups several variations of an element.
Variation: The variation is a suggested change for a given element. It differs from the reference version. A MVT can include several variations that are gathered in sections.
Combination: Combination means that several variations from different sections are combined and displayed to exposed visitors.
How to create a new MVT test?
Via the back office
From the A/B test tab, access the overview page.
Click on “New test”.
You can also click on “New Test” on top of the back office’s left hand menu.
In the pop-in that opens, choose “Multivariate test”.
Via the graphic editor
Once logged in, you can choose “Create a multivariate” test on the home screen.
A window opens where you indicate the test name. You can also add a description and associated tags:
After validation, you need to set up sections and variations for your MVT.
Setup of your test’s sections and variations
A section consists of several variations.
Let’s say you want to test the color of a CTA and want to know which color among blue (reference version), yellow and green generates more conversions.
You can create a first section (Section A) for colors that includes your reference version as well as the variations “yellow” and “green”.
At the same time you want to test the CTA’s wording and compare “Subscription” with “Subscribe now” and “I want to subscribe”. You can create a second section (Section B) for the wording which will include your reference version as well as the variations “Subscribe now” and “I want to subscribe”.
Each section is automatically attributed letter of the alphabet to which you must add a unique name.
The MVT editor
As is the case for classical A/B tests, your variations’ content is created via the graphic editor. There is only one slight change: the editor’s header is different from the one you are used to :
In the dropdown menu on the left, you find the sections you have created. Next to it you see the different variations of the selected section.
Open the section’s dropdown menu. Upon hover over a section’s name, a symbol made of 3 points appears. With a click on the symbol, several actions are suggested.
- Edit the section : return to the page presented above;
- Duplicate the section : duplicates the section and all associated variations;
- Delete the section : deletes the section and all associated variations.
You can also re-name the section by double-clicking on it.
The menus regarding the variations associated to the selected section are situated next to the section menu. The original version is always displayed and cannot be edited. For the variations, the same options as for classical A/B tests are available (add an element, URL redirection, etc.)
If you delete all variations associated to a section, the section itself will be deleted.
Note: We recommend you introduce changes to no more than 2 elements in your page (i.e. to create a maximum of 2 sections) with 3 to 4 variations of the element.
Why? For an MVT, the variations of each section are associated to find the best combination. The more sections you created the more traffic you need to get usable, reliable results.
The number of possible combinations on a given page can easily get too important. In that case you need a great number of unique visitors to obtain results with statistical significance. To learn more about this subject, please refer to our article on statistical significance.
Once you created your variations, you need to finalize your test before publishing it online. No problem, that’s done exactly the same way as with classical A/B tests. The only exception is your test’s traffic distribution.
Traffic allocation for the experiment
On top of the window, you can choose the percentage of visitors to be exposed in the MVT,
- either by moving the blue cursor
- or by double-clicking on the percentage and editing its value.
By default, 100% of the visitors are exposed to the test.
Traffic allocation for exposed visitors
You can choose to distribute your traffic according to:
Traffic distribution by sections
Open the Sections tab to distribute the traffic according to the sections and the variations they contain.
Upon selecting the Sections tab, a list of blocks is displayed. Each block represents a section, all associated variations are listed below.
If a section is checked, 100% of exposed visitors see one of its variations. In other words, visitors exposed to the test see one of the checked variations, according to the traffic distribution among variations you defined.
In the illustration below, 30% of exposed visitors will see a pink CTA, 40% will see a yellow one.
Edit the traffic allocated to a variation
By default, traffic is equally distributed among the selected variations of a given section.
There are two ways to edit the distribution :
- Type the percentage in the text field at the end of the line dedicated to the variation, using the keyboard;
- Use the pointers to edit the percentage.
Block trafic on a variation
Click on the lock to fix the percentage allocated to a given variation.
Equal traffic distribution
With a clock on “Equal traffic distribution” all edits are undone, and traffic is equally distributed among the selected variations that haven’t been blocked.
Traffic distribution by combination
You can also define the traffic distribution according to the combinations you created. Let’s say you want to test a CTA’s color and the wording which gives you a MVT with the sections A and B:
|Section A: Color||Section B: Text|
|Variation A0: Original||Variation B0: Original|
|Variation A1: Green||Variation B1: Shop|
|Variation A2: Red||Variation B2: Buy|
The generated combinations have the following names:
A0B0 Original / Original
A0B1 Original / Shop
A0B2 Original / Buy
A1B0 Green / Original
A1B1 Green / Shop
A1B2 Green / Buy
A2B0 Red / Original
A2B1 Red / Shop
12B2 Red / Buy
By selecting the Combinations tab, you can define the traffic for each combination.
On top of the page, you see the number of combinations selected for the test.
The definition of traffic distribution by combination is more precise than by section, but it is also more time-consuming as the number of combinations can be quite high.
When using the Combinations tab for traffic distribution you can exclude certain combinations. In an MVT with changed colors of background and text for instance, all combinations with the same color for both can be excluded right away.
Edit the traffic allocated to a combination
The traffic to be distributed among all combinations must total 100%.
You can edit the traffic for a given combination only if it is selected. By default, all combinations are selected. Just click in the selection box to uncheck the combinations you want to de-select.
Block traffic on a combination
Click on the lock to fix the percentage of traffic allocated to a given combination.
Equal traffic distribution
Next to the number of selected combinations, you see the button for equal traffic distribution. By clicking on it, traffic is equally distributed among all selected variations.
The same percentage of traffic is allocated to all selected, unlocked combinations. If combinations are added or deleted, the distribution is automatically adjusted.
Note: By default, the Sections tab is selected. The tabs are independent, i.e. editing the distribution for one of them doesn’t impact the other. The traffic distribution that will be taken into account is the one of the tab (either Sections, or Combination) that is selected when you hit the Validate button.
Simulation, launch, analysis: business as usual!
You created your variations and defined the traffic distribution. Well done! The rest is the same as for classical A/B tests.
Simulate your test
Before going live, we recommend you launch a simulation to check that everything meets your expectations.
Your test’s different combinations are generated in the simulation panel. Click on “Currently displayed in simulation mode” to check the display of the combination of your choice.
More about simulation in our dedicated articles.
Launch the test
Everything is displayed correctly? The targeting has been set up? Your goals and reporting tools have been defined?
Your test is ready to be published! To do so, just hit the “Launch” button!
If you selected Kameleoon as your reporting tool, your MVT’s results are transferred to the A/B test result page you already know. Each combination is considered as a variation (as defined for classical A/B tests).
Note: There may be a short latency time (up to 10 minutes) between the launch of an A/B test and its visibility on the website. Don’t worry if your test does not appear immediately!
What about my existing MVTs?
You may wonder what happens to MVTs you already created. Don’t worry, all changes are automatically applied to your older tests.
You are now officially ready for combinations and multivariations!