To help you set up your experiments, here’s a little practical exercise.
Step by step, we’ll look at how to imagine an A/B experiment, configure it and launch it.
Let’s go!
1. Imagine your experiment and identify your goals
We’d like to rework the newsletter part of our blog’s home page.
Our goal is to increase newsletter subscriptions. So we will test several versions of this element of the home page.
A first hypothesis consists in revising the wording of the block. Another is to move it to another place on the page. So we will test three versions of the page:
- Keep the block as it is (current situation);
- Change the wording of the block;
- Move the block.
2. Launch the Graphic editor
Once on the SnapEvent home page, we use the following keyboard shortcut to launch the Kameleoon editor: Shift + F2.
3. Create a new experiment
We then click on “Create an experiment”.
We type in the name of our experiment and choose to give it a short description.
By clicking on “Create”, we validate these choices and access the Graphic editor.
4. Create variations
By default, a first variation (Variation 1) already exists. To display the modification actions, we click on the variation in order to unfold the menu.
For greater clarity, we rename the first variation “WORDING” because it will be associated with a modification of the wording.
In this variation of the home page, we are looking to replace “Sign up for our newsletter” with “Let’s stay in touch with our newsletter”
To do this, we select the element to be edited: the block title. The tool panel opens on the left of the editor.
We then click the “Edit content” option in the tool panel.
We type in the new text in the pop-up window. We could change its style with the options available, but we respect the style of our graphic charter.
Once validated, the element is modified on the page.
If a mistake occurs, it’s possible to go back with the History tool, located on the right of the editor.
The modification we have just made is the first variation. In the second variation, the element will be moved on the page.
To add a new variation, we click on the + sign in the variation menu and rename it “POSITION”.
This time we will select the entire block and click on the “POSITION” tab of the tool panel.
The move options allow us to position the element where we want.
We would like to place trackers on our element. For this, we select the “Subscribe” button in each variation and, in the “Tracking” tab of the tool panel, associate a click tracking to it.
It is then displayed in the lower part of the tool panel, under the heading “Associated to the element”: it has been taken into account.
There is no need to repeat the operation on the other variation: the tracking is associated with the element on the entire experiment.
We will use this tracking in step 8 of the exercise.
5. Allocate traffic
Once the experiment variations are created, we enter the finalization phase.
By clicking on the “Finalize” menu at the top right of the editor, we can access the panel.
The first step of this finalization is to distribute the traffic to our variations.
When we click on “Traffic allocation”, the pop-in that opens gives us the possibility to adjust the percentage of visitors allocated to each variation using sliders.
We want to distribute the traffic between the original version of the page and the 2 variations we have just created. Half of the visitors will see the original; the other half will see one of the two variations.
To set up this distribution, we click on the padlock of the original to unlock the changes, then click on the percentage to the right and indicate 50. We enter the number 25 for each variation.
We then click on “Proceed” to go to the next step.
We need to define the targeting of the experiment. We are trying to attract returning visitors as a priority: we will target this population.
6. Target
By clicking on “Target a segment”, we can access the option to create a new segment. The segment builder appears.
We drag and drop the “New or returning visitors” condition (“Visitor Characteristics” category) on the right-hand side of the segment builder, and decide to include only returning ones.
We add another condition excluding smartphone users (“Device type” condition). This means that only visitors meeting these criteria will be able to see our variations. All the others will only have access to the original version of our blog’s home page.
In order to quickly identify our segment later and re-use it in other campaigns, we rename it “returning-visitors” and validate its creation. We just have to click on “Proceed” to go to the next step.
7. Select a tracking tool
In the pop-in selection of a tracking tool and goals, we stay on the Kameleoon tab to activate reporting on it. Then we need to associate at least one goal so that it can track the performance of the variations and show results.
8. Associate a goal
By default, only the “Engagement” goal is set (this goal is reached if the visitor has visited other pages after the landing page). However, in step 4, we created a click tracking, which is then displayed as an option in the list of goals. We will select it by clicking on the corresponding line. It becomes the main goal of our experiment.
In continuing to finalize the experiment, we now access its summary. It’s now time to check that everything is ok:
- Traffic is allocated 50/50 between our two variations and the original one;
- The “returning-visitors” segment is targeted;
- The Kameleoon reporting tool is selected;
- Click tracking is associated with the experiment.
Each category can be scrolled to display the details.
9. Simulate
Everything is ok, so it’s almost time to launch our experiment! Before we do that though, it’s very important to simulate our campaign to check the display and targeting of variations.
We close the summary pop-in and click on the fourth step in the finalization menu, which takes us to the experiment simulation.
A new page opens with the simulation panel.
In our example, we match all the targeting conditions: our status is “Targeted”. By clicking on the “Targeting” tab, we display all the conditions and our status in relation to them. Here, we are using a laptop: so the “Device type” condition that we defined in our segment is True. We are also a returning visitor: so this condition is True.
These elements enable us to check the targeting of our experiment. In our case, everything is working as expected.
The final stage: let’s make sure our goals are properly set. In the “Goals” tab of the simulation panel, the Achieved/Not achieved status of the goal is indicated (like for the True/False targeting conditions):
We have yet to click on the subscribe button for our variations. So it’s normal that associated click tracking hasn’t been achieved. All we have to do now is click on one of the elements and note that the status changes and becomes “Converted”: everything is fine!
We can exit the simulation by clicking on “Close” at the bottom right of the page. The step will turn green in the finalization panel. This confirmation is independent of the behavior in the simulation panel: it only indicates that the simulation mode has been used.
10. Launch the experiment
Let’s go back to the Graphic editor. Our experiment is ready to go online. Click on “Launch” in the finalization panel, then on “Put online”, and presto!
Note: There may be a short latency time (up to 10 minutes) between the launch of an experiment and its visibility on the website. So don’t worry if your experiment doesn’t appear immediately!
Depending on the daily traffic on our website, as well as our goals and the confidence rate that we want to achieve, we will have to wait for a certain amount of time before we can analyze the experiment results.
In this case, we will have waited five days. It would have been possible to estimate this duration thanks to the tool included in the finalization panel of the editor (just to the right of the “Simulate” button).If you are looking for more information, don’t hesitate to consult our Experimentation documentation and use the search bar.
