In a multi-campaign environment, understanding the true impact of changes can be challenging, especially when experiments overlap.
Kameleoon’s Holdouts solve this by creating a control group that remains unaffected by any active campaigns. This allows you to establish a reliable baseline, so you can measure the aggregate impact of all changes against a group that experiences no modifications. In this guide, we’ll explore how holdouts work, why they’re valuable for meaningful analysis, and how to set them up effectively in Kameleoon.
This feature is currently available only for web experiments. While it does not yet support personalizations or feature experiments, feature experiment compatibility is coming soon.
What is a Holdout?
A holdout is a control group in Kameleoon that does not see any experiments or changes. It’s used to create a baseline against which you can measure the overall impact of all active experiments. Rather than comparing individual test groups, a holdout allows you to measure the cumulative effect of all changes on user engagement or other metrics.
Key benefits
In a typical A/B test, you compare users who see a single experiment against those who don’t. However, when multiple experiments are running, the results may overlap and impact each other, making it difficult to see the overall effect of the changes. A holdout solves this by:
- Providing a Baseline: The holdout group receives no experiments, creating a clear point of comparison.
- Measuring Aggregate Impact: With a holdout, you can see if the total impact of your changes is positive by comparing it to a group that sees none of them.
Example of a Holdout
Imagine an e-commerce site running three experiments: changing button colors, updating product images, and adding a “Top Picks” section. The holdout group would:
- See the original versions of each of these elements (no changes applied).
- Act as a baseline to understand if the cumulative effect of these changes actually improves conversion rates or engagement.
How to set up a Holdout in Kameleoon
- Create a new experiment: Set up an experiment in Kameleoon and tag it with the “HOLDOUT” label.
- Set traffic split: Split your traffic so that one group (Control) sees no experiments, and the other (Variation) can be assigned to any experiment (as long as they meet the targeting criteria).
- Add goals: Define the goals you want to track in the experiment. These goals will be essential for comparing the performance of the holdout group against visitors who are not in the holdout, allowing you to measure specific metrics like conversions, engagement, or other key actions.
- Launch your experiment: Once the holdout experiment is created and launched, no further setup is needed. Simply monitor its performance over time by checking the results page of the experiment tagged as holdout. This allows you to track the baseline metrics continuously and gain insights into the overall impact of active campaigns.
Your control group, which does not see any changes, becomes your baseline to measure the overall impact of the experiments displayed to the variation group.
Best practices
- Traffic allocation: It’s recommended to create a holdout with a maximum of 10% excluded traffic over a limited period, typically 3-6 months. This ensures a reliable baseline without heavily impacting overall traffic.
- Simple segmentation: You can set up multiple holdout groups for different segments, such as one for desktop and another for mobile. However, it’s best to avoid overly complex targeting rules for a holdout to maintain clarity and effectiveness.
