Holdouts help measure the long-term impact of product updates by excluding a group of users from changes, allowing for a clearer analysis of key metrics over time. This article explains how to set up and monitor holdouts in Kameleoon.
What are Holdouts?
Holdouts allow you to measure the long-term impact of product updates by setting aside a group of users who will not see the changes. By tracking key metrics over days, weeks, or months, you can identify delayed effects, unintended consequences, or opportunities for further improvements.
How to setup Holdouts
Note: This article explains holdouts for Feature Experimentation. To use holdouts for Web Experimentation refer to this help doc.
To get started, navigate to the Feature Experimentation dashboard and click the ⋮ button near the top right to open the Holdouts panel.
In the overlay, you will see a list of your Kameleoon projects with an overview of their holdout status. By default all projects will have their holdout as “Not set”.
To configure your holdout, click on any project. A typical setup requires you to:
- Toggle on your holdout for this project.
- Assign a traffic split for the holdout, typically between 1-10%. This is the percentage of users who will be excluded from all experiments and rollouts while the holdout is active.
- Select Goals which are important for your team in the long run, to ensure that the holdout measures this metric data completely isolated from any product updates the rest of your users will receive.
Once your holdout is configured, click Save to finalize it and return to dashboard. When you reopen the Holdouts panel, you’ll be able to see a summary of which ones are configured and have results available.
To access holdout results, simply click on a project in the above view and click Results at the bottom to open a dedicated reporting page where you can monitor metric trends, as well as collaborate with your data, product, and engineering teams to analyze and interpret the results.
Best Practices for using Holdouts
- Differentiate Testing Approaches:
A/B tests capture immediate reactions; holdout tests reveal the enduring effects of updates so it is advised to plan the longevity of holdouts accordingly. - Focus on Key Metrics:
Monitor long-term indicators like engagement, retention, and revenue rather than relying solely on short-term data. Optimize for long-term when using holdouts. - Encourage Collaboration:
Facilitate ongoing communication among product managers, data scientists, and engineers to ensure accurate setup, monitoring, and interpretation. - Duration of Holdouts:
Consider how long to keep users in a holdout to minimize dissatisfaction as they will not be receiving updates.
