Understanding visit count mismatches in capped experiments
When you use display limits (capping) in your feature flags or experiments, the total number of visits recorded for your variations often appears uneven compared to the Original version.
If you notice that the Original version has significantly more visits than the Variation, this behavior is expected and does not indicate a failure in your experiment setup, such as a Sample Ratio Mismatch (SRM).
Use this guide to understand exactly why capping creates this difference in visit counts and why you should focus on the unique visitor count instead.
The effect of display limits on experiment data
In advanced setups, a display limit is often achieved by dynamically adding a user to an exclusion segment after they have seen the variation once, which is commonly done by setting a custom data or local storage value for the visitor.
You can also configure display limits in the Code editor for experiments and set similar capping limits for Personalizations.
Example: Pop-up display limits and visit counts
Suppose you configure an experiment where the Variation displays a marketing pop-up, but the Original does not. To prevent annoying users, you use a display limit to show the pop-up only once per month.
This configuration often causes a discrepancy in the total number of visits (sessions) recorded, even though the number of unique visitors remains balanced.
Original: No display limit triggers
When Kameleoon allocates a visitor to the Original version:
- The pop-up never displays.
- Because the pop-up never displays, the once-per-month limit never activates.
- The system tracks this visitor on every subsequent page view or return visit.
Result: The Original records a high number of total visits because nothing restricts the tracking.
Variation: Display limit applies
When Kameleoon allocates a visitor to the Variation:
- On the first visit, the visitor sees the pop-up.
- The system records that the display limit (1/1) has been reached.
- On the next visit, the display limit prevents the pop-up from showing. Depending on your configuration, the system may stop counting these subsequent visits as part of the active experiment exposure.
The visit count disparity
This scenario creates a balanced split at the unique visitor level, but a significant imbalance at the total visit level.
- Original: One visitor might generate 10 visits.
- Variation: One visitor might generate only one tracked visit.
| Metric | Original (no pop-up) | Variation (pop-up capped at one) |
|---|---|---|
| Unique visitors | 500 | 500 |
| Total visits | High (targeted on every return visit) | Low (targeted only once) |
This is not a Sample Ratio Mismatch
It is common to suspect a setup error when you see a large difference in visit counts. However, if the unique visitor count is equal, this is not a Sample Ratio Mismatch (SRM).
- What is SRM? SRM is a statistical error where the initial allocation of unique visitors does not match your configured split (50/50, for example). SRM usually indicates a tracking or technical issue.
- What is happening here? The display limit is working as intended. It caps the exposure of the Variation, which naturally reduces the total visit count for that variation. The Original remains uncapped because it has no element to limit.
Always confirm your experiment split by looking at the number of unique visitors. If the unique visitor count is balanced according to your configuration, your experiment is running correctly.