Checking for potential SRM

portrait de l'auteur Julie Trenque

Written by Julie Trenque

Updated on 05/11/2022

1 min

Advanced

Was this content useful?

What is it ?

A Sample Ratio Mismatch (usually referred to as SRM) means that there is a significant difference between the expected deviations of the experiment variations and the observed ones. For example, if you launched a campaign with a traffic allocation of 50% toward the reference and 50% toward the variation, but in fact the data you observed is 50 000 visitors on the reference and 48 900 on the variation then this would come out positive to our sample ratio mismatch test. It is detected by running a “Chi-Square Goodness of Fit test” which helps to understand if a variable is likely to come from a specified distribution.

Why is it important ?

It is important because when we detect a potential SRM in a test, it means the observed distribution of visitors among the variations is very different from the expected distribution. This means that there is a bias which might influence the assignation mechanism and it makes the campaign result invalid because one of the chore hypothesis on which statistical computations for A/B test are based is violated.

Why can it happen ?

We have seen that the root causes can greatly change between customers but mainly we have seen that mismatch could occur more frequently due to telemetry issues. When, for example, the variation design might affect our tracking mechanism. In order to understand from where the mismatch stem, a good practice is to look at different breakdowns such as browser or operating system breakdown. Sometimes the assignation difference root cause becomes obvious when looking at broken down data. Once you have managed to identify the root cause you can fix it and simply relaunch your test and safely interpret your results.

For help diagnosing potential causes you can check out this great and extensive taxonomy of causes: https://www.lukasvermeer.nl/srm/docs/causes/.

  • In this article :