Skip to main content

Create and manage holdouts

Holdouts allow you to measure the long-term impact of product updates by excluding a group of users from those changes, which makes it easier to analyze key metrics over time. This article explains how to set up and monitor holdouts in Kameleoon.

Setting up holdouts​

note

This article focuses on holdouts for Feature Experimentation. To use holdouts for Web Experimentation, refer to this article.

To get started, go to the Feature Experimentation dashboard and click the three dots menu in the top-right corner > Holdouts. The Holdouts panel will open.

In the panel, you'll see a list of your Kameleoon projects, each with a summary of its holdout status. By default, all projects will show Not set under holdout status.

To configure a holdout:

  1. Click a project.
  2. Enable a holdout by toggling it on.
  3. Set the traffic split—typically between 1-10%. The traffic split is the percentage of users who will be excluded from all experiments and rollouts while the holdout is active.
  4. Select the key goals you want to track. These should be metrics your team values long-term, as the holdout will measure them in a fully isolated environment, unaffected by any product updates.

Once you've finished configuration, click Save to apply your changes and return to the dashboard.

When you reopen the Holdouts panel, you'll see a summary showing which holdouts have been configured and whether results are available.

To view holdout results:

Click a project and select Results at the bottom of the panel. A dedicated reporting page opens where you can:

  • Monitor trends in your selected metrics.
  • Share insights with your product, data, and engineering teams.
  • Analyze the long-term impact of your changes.

Best practices​

  • Differentiate testing approaches: Use A/B tests to measure immediate user reactions and holdout tests to assess the long-term impact of product updates. Plan the duration of your holdouts with this distinction in mind.
  • Focus on key metrics: Prioritize long-term indicators such as engagement, retention, and revenue. Avoid relying solely on short-term metrics when evaluating holdout performance.
  • Encourage cross-team collaboration: Maintain active communication between product managers, data scientists, and engineers to ensure holdouts are properly configured, monitored, and analyzed.
  • Be mindful of holdout duration: Balance the need for long-term insights with user experience. Keeping users in a holdout too long—without updates—can lead to dissatisfaction.