Launch of a mutivariate test

3

min

quiz

The creation of MVT tests is included in all our A/B testing plans.

What is a multivariate test or MVT?

While a classical A/B test consists of a reference version and a variation, MVT tests enable you to change several elements on the same page and to analyze which combination generates more conversions. Let’s look at the example of a simple MVT that compares the performance of six CTAs:

In the example, there are 3 color variations (red, green, blue) and 2 wordings: “Download Firefox” and “Download now”.

With an MVT the six possible combinations can be compared.

MVT Glossary

Multivariate test: A multivariate test offers to test the conversion of one or several combination(s) of several elements.

MVT: MVT stands for MultiVariate Test.

Section: A section groups several variations of an element.

Variation: The variation is a suggested change for a given element. It differs from the reference version. A MVT can include several variations that are gathered in sections.

Combination: Combination means that several variations from different sections are combined and displayed to exposed visitors.

How to create a new MVT test?

Via the back office

From the A/B test tab, access the overview page.

Click on “New test”.

You can also click on “New Test” on top of the back office’s left hand menu.

In the pop-in that opens, choose “Multivariate test”.

Via the graphic editor

Once logged in, you can choose “Create a multivariate” test on the home screen.

A window opens where you indicate the test name. You can also add a description and associated tags:

After validation, you need to set up sections and variations for your MVT.

Setup of your test’s sections and variations

A section consists of several variations.

Let’s say you want to test the color of a CTA and want to know which color among blue (reference version), yellow and green generates more conversions.

You can create a first section (Section A) for colors that includes your reference version as well as the variations “yellow” and “green”.

At the same time you want to test the CTA’s wording and compare “Subscription” with “Subscribe now” and “I want to subscribe”. You can create a second section (Section B) for the wording which will include your reference version as well as the variations “Subscribe now” and “I want to subscribe”.

Each section is automatically attributed letter of the alphabet to which you must add a unique name.

The MVT editor

As is the case for classical A/B tests, your variations’ content is created via the graphic editor. There is only one slight change: the editor’s header is different from the one you are used to :

In the dropdown menu on the left, you find the sections you have created. Next to it you see the different variations of the selected section.

Sections

Open the section’s dropdown menu. Upon hover over a section’s name, a symbol made of 3 points appears. With a click on the symbol, several actions are suggested.

  • Edit the section : return to the page presented above;
  • Duplicate the section : duplicates the section and all associated variations;
  • Delete the section : deletes the section and all associated variations.

You can also re-name the section by double-clicking on it.

Variations

The menus regarding the variations associated to the selected section are situated next to the section menu. The original version is always displayed and cannot be edited. For the variations, the same options as for classical A/B tests are available (add an element, URL redirection, etc.)

If you delete all variations associated to a section, the section itself will be deleted.

Note: We recommend you introduce changes to no more than 2 elements in your page (i.e. to create a maximum of 2 sections) with 3 to 4 variations of the element.

Why? For an MVT, the variations of each section are associated to find the best combination. The more sections you created the more traffic you need to get usable, reliable results.
The number of possible combinations on a given page can easily get too important. In that case you need a great number of unique visitors to obtain results with statistical significance. To learn more about this subject, please refer to our article on statistical significance.

Traffic allocation

Once you created your variations, you need to finalize your test before publishing it online.  No problem, that’s done exactly the same way as with classical A/B tests. The only exception is your test’s traffic distribution.

Traffic allocation for the experiment

On top of the window, you can choose the percentage of visitors to be exposed in the MVT,

  • either by moving the blue cursor

  • or by double-clicking on the percentage and editing its value.

By default, 100% of the visitors are exposed to the test.

Traffic allocation for exposed visitors

You can choose to distribute your traffic according to:

  • sections;
  • combinations.

Traffic distribution by sections

Open the Sections tab to distribute the traffic according to the sections and the variations they contain.

Upon selecting the Sections tab, a list of blocks is displayed. Each block represents a section, all associated variations are listed below.

If a section is checked, 100% of exposed visitors see one of its variations. In other words, visitors exposed to the test see one of the checked variations, according to the traffic distribution among variations you defined.

Example:

In the illustration below, 30% of exposed visitors will see a pink CTA, 40% will see a yellow one.

Edit the traffic allocated to a variation 

By default, traffic is equally distributed among the selected variations of a given section.

There are two ways to edit the distribution :

  • Type the percentage in the text field at the end of the line dedicated to the variation, using the keyboard;
  • Use the pointers to edit the percentage.

Block trafic on a variation

Click on the lock to fix the percentage allocated to a given variation.

Equal traffic distribution

With a clock on “Equal traffic distribution” all edits are undone, and traffic is equally distributed among the selected variations that haven’t been blocked.

Traffic distribution by combination

You can also define the traffic distribution according to the combinations you created. Let’s say you want to test a CTA’s color and the wording which gives you a MVT with the sections A and B:

Section A: Color Section B: Text
Variation A0: Original  Variation B0: Original
Variation A1: Green  Variation B1: Shop
Variation A2: Red  Variation B2: Buy

The generated combinations have the following names:

A0B0 Original / Original

A0B1 Original / Shop

A0B2 Original / Buy

A1B0 Green / Original

A1B1 Green / Shop

A1B2 Green / Buy

A2B0 Red / Original

A2B1 Red / Shop

12B2 Red / Buy

By selecting the Combinations tab, you can define the traffic for each combination.

On top of the page, you see the number of combinations selected for the test.

The definition of traffic distribution by combination is more precise than by section, but it is also more time-consuming as the number of combinations can be quite high.

When using the Combinations tab for traffic distribution you can exclude certain combinations. In an MVT with changed colors of background and text for instance, all combinations with the same color for both can be excluded right away.

Edit the traffic allocated to a combination

The traffic to be distributed among all combinations must total 100%.

You can edit the traffic for a given combination only if it is selected. By default, all combinations are selected. Just click in the selection box to uncheck the combinations you want to de-select.

Block traffic on a combination

Click on the lock to fix the percentage of traffic allocated to a given combination.

Equal traffic distribution

Next to the number of selected combinations, you see the button for equal traffic distribution. By clicking on it, traffic is equally distributed among all selected variations.

The same percentage of traffic is allocated to all selected, unlocked combinations. If combinations are added or deleted, the distribution is automatically adjusted.

Note: By default, the Sections tab is selected. The tabs are independent, i.e. editing the distribution for one of them doesn’t impact the other. The traffic distribution that will be taken into account is the one of the tab (either Sections, or Combination) that is selected when you hit the Validate button.

Simulation, launch, analysis: business as usual!

You created your variations and defined the traffic distribution. Well done! The rest is the same as for classical A/B tests.

Simulate your test

Before going live, we recommend you launch a simulation to check that everything meets your expectations.

Your test’s different combinations are generated in the simulation panel. Click on “Currently displayed in simulation mode” to check the display of the combination of your choice.

More about simulation in our dedicated articles.

Launch the test

Everything is displayed correctly? The targeting has been set up? Your goals and reporting tools have been defined?
Your test is ready to be published! To do so, just hit the “Launch” button!

Analysis

If you selected Kameleoon as your reporting tool, your MVT’s results are transferred to the A/B test result page you already know. Each combination is considered as a variation (as defined for classical A/B tests).

To learn more about the results page

What about my existing MVTs?

You may wonder what happens to MVTs you already created. Don’t worry, all changes are automatically applied to your older tests.

You are now officially ready for combinations and multivariations!