Example: 10 steps for an experiment

portrait de l'auteur Julie Trenque

Written by Julie Trenque

Updated on 22/09/2020

5 min

Beginner

Was this content useful?

To help you set up your experiments, we offer you a little practical exercise.

We will see together, step by step, how to imagine an A/B experiment, configure it and launch it.

Let’s go !

1. Imagine your experiment and identify your goals

We would like to rework the newsletter part of our blog homepage.

Our goal is to increase subscriptions to the newsletter. We will therefore test several versions of this element of the homepage.

A first hypothesis consists in revising the wording of the block. Another is to move it to another place on the page. So we will test three versions of the page:

  1. Keep the block as it is (current situation);
  2. Change the wording of the block;
  3. Move the block.

2. Launch the editor

Once on the SnapEvent homepage, we use the following keyboard shortcut to launch the Kameleoon editor:

3. Create a new experiment

We then click on “Create an experiment”.

We type the title of our experiment and choose to associate a short description.

By clicking on “Create”, we validate these choices and access the graphic editor.

4. Create variations

By default, a first variation (Variation 1) already exists. To display the modification actions, we click on the variation in order to unroll the menu.

For greater clarity, we rename the first variation “Wording” because it will be associated with a modification of the wording.

In this variation of the homepage, we are looking to replace “Sign up for our newsletter” with “Let’s stay in touch with our newsletter”

To do this, we select the element to be edited: the block title. The tool panel opens on the left of the editor.

We then click the “Edit content” option in the tool panel.

In the pop-up window, we type the new text. We could change its style with the options, but we keep the style of our graphic charter.

Once validated, the element is modified on the page.

In case of error, it would have been possible to go back with the History tool, located on the right of the editor.

The modification we have just made is the first variation. In a second variation, the element will be moved on the page.

To add a new variation, we click on the + sign in the variation menu and rename it “POSITION”.

This time we will select the entire block and click on the “Position” tab of the tool panel.

The move options allow us to position the element where we want.

We would like to place trackers on our element. For this, we select the “Subscribe” button in each variation and, in the “Tracking” tab of the tool panel, associate a click tracking to it.

It is then displayed in the lower part of the tool panel, under the heading “Associated to the element”: it has been taken into account.

There is no need to repeat the operation on the other variation: the tracking is associated with the element on the entire experiment.

We will use this tracking in step 8 of the exercise.

5. Allocate traffic

Once the experiment variations are created, we enter the finalization phase.

A click on the “Finalize” menu at the top right of the editor allows us to access the panel.

The first step of this finalization is to distribute the traffic on our variations.

When we click on “Traffic allocation”, the pop-in that opens gives us the possibility to adjust, with sliders, the percentage of visitors allocated to each variation.

We want to evenly distribute the traffic between the original version of the page and the 2 variations we have just created. Half of the visitors will see the original; the other half, one of the two variations.

To set up this distribution, we click on the padlock of the original to unlock the changes, then click on the percentage to the right and indicate 50. We enter the number 25 for each variation.

We then click on “Proceed” to go to the next step.

We need to define the targeting of the experiment. We are trying to attract returning visitors as a priority: we will target this population.

6. Target

A click on “Target a segment” allows us to access the option to create a new segment. The segment builder appears.

We drag and drop the “New or returning visitors” condition (“Visitor Characteristics” category) on the right side of the segment builder, and decide to include only returning ones.

We add another condition excluding smartphone users (“Device type” condition). This means that only visitors meeting these criteria will be able to see our variations. All others will only have access to the original version of our blog homepage.

In order to quickly identify our segment later and reuse it in other experiences, we rename it “returning-visitors” and validate its creation. We just have to click on “Proceed” to go to the next step.

7. Select a tracking tool

In the pop-in selection of a tracking tool and goals, we stay on the Kameleoon tab to activate reporting on it. Then you need to associate at least one goal so that it can track the performance of your variations and show you results.

8. Associate a goal

By default, only the “Engagement” goal is set (this goal is reached if the visitor has visited other pages after the landing page). But we created, in step 4, a click tracking, which are then proposed to us in the list of goals. We will select it by clicking on the corresponding line. It becomes the main goal of our experiment.

By continuing the finalization of the experiment, we access its summary. It’s time to check that everything is ok:

  • Traffic is allocated 50/50 to our two variations and the original one;
  • The “returning-visitors” segment is targeted;
  • The Kameleoon reporting tool is selected;
  • The click tracking is associated with the experiment.

Each category can be scrolled to display the details.

9. Simulate

Everything is ok, it’s almost time to launch our experiment! But before, it is very important to simulate our experience to check the display and targeting of variations.

We close the summary pop-in and, in the finalization menu, click on the fourth step which takes us to the experiment simulation.

A new page opens with the simulation panel.

In our example, we match all targeting conditions: our status is “Targeted”. By clicking on the “Targeting” tab, we display all the conditions and our status in relation to them. Here, we are on a laptop: the condition “Device type” that we have set in our segment is True. We are also a returning visitor: this condition is True.

These elements allow us to check the targeting of our experiment. In our case, everything works as expected.

Last step: make sure our goals are properly set. In the “Goals” tab of the simulation panel, the Achieved/Not achieved status of the goal is indicated (like for the True/False targeting conditions):

We have not yet clicked on the subscribe button of our variations. It is therefore normal that associated click tracking has not been achieved. All we have to do is click on one of the element and notice that the status changes and becomes “Achieved”: everything is fine!

We can exit the simulation by clicking on “Close” at the bottom right of the page. The step will turn green in the finalization panel. This confirmation is independent of the behavior in the simulation panel: it only indicates that the simulation mode has been used.

10. Launch the experiment

Back to the graphic editor. Our experiment is ready to go online. Click on “Launch” in the finalization panel, then on “Put online”, and presto!

Note: There may be a short latency time (up to 10 minutes) between the launch of an experiment and its visibility on the website. Don’t worry if your experiment does not appear immediately!

Depending on the daily traffic on our website, as well as our goals and the confidence rate that we want to achieve, we will have to wait more or less time before we can analyze the experiment results.

In our case, we will have waited 5 days. It would have been possible to estimate this duration thanks to the tool included in the finalization panel of the editor (just to the right of the “Simulate” button).

If you are looking for some more information, do not hesitate to consult our Experimentation documentation and use the search bar.

  • In this article :