Example: 10 steps for an A/B test




To help you set up your A/B tests, we offer you a little practical exercise.

We will see together, step by step, how to imagine a test, configure it and put it online.

Let’s go !

1. Imagine your test and identify your goals

The company SnapEvent, specialized in renting exceptional places for the organization of public and private events, seeks to expand its catalog. Indeed, its client base is evolving quickly, leading to a growing demand for new locations, especially in Paris. SnapEvent would like to highlight their option “Louer mon bien”(Rent my property”) to attract new owners from the homepage of the website, without degrading the experience of other visitors (and thus potentially reduce the number of reservations).

For now, the action “Louer mon bien” only appears in the footer. As part of this exercise, we want to set up an A/B test to test several possibilities:

  1. Keep the option only in the footer (current situation);
  2. Add the option in the menu;
  3. Add the option in the sticky header.

2. Launch the editor

Once on the SnapEvent homepage, we use the following keyboard shortcut to launch the Kameleoon editor:

3. Create a new test

We then click on “Create a new test”.

We type the title of our test and choose to associate a short description of the tested hypothesis.

By clicking on “Create test”, we validate these choices and access the interface of the graphic editor.

4. Create variations

By default, a first variation (Variation 1) already exists. To display the modification actions, we click on the variation in order to unroll the menu.

For greater clarity, we rename the first variation “MENU” because it will be associated with a modification of the menu.

In this variation of the home page, we are looking for the “Louer mon bien” option to appear in the yellow menu at the top of the page.

For that, we just have to click on the variation, then on “New element”.

We then choose the “Paragraph” option.

In the pop-in, we type the text “Louer mon bien”. It is possible to format this text. Here, the selected font is Rawline, the color is a shade of yellow that is part of the graphical charter of the website, the size is 13 pixels, in uppercase and bold.

Once validated, the element is added to the page. By default, it appears in the center. It must be moved to complete the menu.

To do that, we click on the toolbox which is on the left of the editor. The “Position” tab allows, thanks to the “Insert before” option, to insert our text between two pre-existing elements: the telephone number and the “Mon compte” heading.

Here is the result:

We then select the added element and associate it, thanks to the “Style” tab of the toolbox, a URL leading to the right page.

In case of error, it would have been possible to go back with the History tool, located on the right of the editor.

The modification we have just made is the first variation tested. In a second variation, the new element will be inserted into the white sticky header, just to the right of “Prestations événementielles”.

To add a new variation, we click on the + sign in the variation menu and rename it “STICKY HEADER”.

We repeat the same actions to adapt the element to the other ones of the home page.

We get this result:

We want to place trackers on our new elements: the “Louer mon bien” from the yellow menu and the white sticky header. For this, in the editor, we select in the first variation the created element and, in the “Tracking” tab of the toolbox, associate a clicks tracking with it.

It is then displayed in the lower part of the toolbox, under the heading “Associated with the element”: it has been taken into account.

We repeat the operation for the second variation. This second tracker will be named “click-louer-mon-bien-header”.

We will find the two trackers in step 8 of the exercise.

5. Allocate traffic

Once the test variations are created, we enter the finalization phase.

A click on the “Finalize” menu at the top right of the editor allows us to access the panel.

The first step of this finalization is to distribute the traffic on our variations.

When we click on “Traffic allocation”, the pop-in that opens gives us the possibility to adjust, with sliders, the percentage of visitors allocated to each variation.

We want to evenly distribute the traffic between the original version of the page and the 2 variations we have just created. Half of the visitors will see the original; the other half, one of the two variations.

To set up this distribution, we click on the padlock of the original to unlock the changes, then click on the percentage to the right and indicate 50. We enter the number 25 for each variation.

We then click on “Continue” to go to the next step.

6. Target

We need to define the targeting of the test. We are trying to attract Parisian owners as a priority: we will target this population.

A click on “Target a segment” allows us to access the option to create a new segment. The segment builder appears.

We drag and drop the “IP Geolocation” condition (“Visitor Characteristics” category) on the right side of the segment builder, and decide to include only visitors from Paris.

We refine this condition by excluding smartphone users. This means that only visitors meeting these criteria will be able to see our variations. All others will only have access to the original version of our homepage.

In order to quickly identify our segment later and reuse it in other experiences, we rename it “Parisiens” and validate its creation. We just have to click on “Continue” to go to the next step.

7. Select a tracking tool

In the pop-in selection of a tracking tool, we click on Kameleoon to activate it. Then you need to associate at least one goal so that it can track the performance of your variations and show you results.

8. Associate a goal

By default, only the “Engagement” goal is set (this goal is reached if the visitor has visited other pages after the landing page). But we created, in step 4, two click trackings, which are then proposed to us in the list of goals. We will select them by clicking on the corresponding line. To choose one of them as the main goal, we click on the star.

By continuing the finalization of the test, we access its summary. It’s time to check that everything is ok:

  • Traffic is allocated 50/50 to our two variations and the original one;
  • The “Parisiens” segment is targeted;
  • The Kameleoon reporting tool is selected;
  • The three goals we have chosen are associated with the test.

Each category can be scrolled to display the details.

9. Simulate

Everything is ok, it’s almost time to launch our test! But before, it is very important to simulate our experience to check the display and targeting of variations.

We close the summary pop-in and, in the finalization menu, click on the iconwhich leads us to the simulation of the test.

A new page opens with the simulation panel.

In our example, we do not match all targeting conditions: our status is “Untargeted”. By clicking on the “Targeting” tab, we display all the conditions and our status in relation to them. Here, we are on a laptop: the condition “Device type” that we have set in our segment is True. However, our IP is not geolocated in Paris: this condition is False.

These elements allow us to check the targeting of our test. In our case, everything works as expected. We would still want to check the display of the homepage in case all targeting conditions are true. To do this, we just have to click on “Force display” in the “Experiences” tab. The simulation page is reloaded and allows us to verify everything.

Last step: make sure our goals are properly set. In the “Goals” tab of the simulation panel, the Achieved/Not achieved status of each goal is indicated (like for the True/False targeting conditions):

We have not yet clicked on the new elements of our variations. It is therefore normal that associated click trackings have not been achieved. All we have to do is click on one of these elements and notice that the status changes and becomes “Achieved”: everything is fine!

We can leave the simulation by clicking on “Close” at the bottom right of the page.

10. Launch the test

Back to the graphic editor. Our test is ready to go online. Click on “Launch” in the finalization panel, then on “Put online”, and presto!

Depending on the daily traffic on our website, as well as our goals and the confidence rate that we want to achieve, we will have to wait more or less time before we can analyze the test results.

In our case, we will have waited 5 days. It would have been possible to estimate this duration thanks to the tool included in the finalization panel of the editor (just to the right of the “Simulate” button).

If you are looking for some more information, do not hesitate to consult our A/B testing documentation and use the search bar.

(NB: SnapEvent is not a Kameleoon customer, this A/B test is fictional.)