Kameleoon offers many out-of-the-box analytics integrations which you can use to measure the impact of your campaigns (A/B tests or Personalizations). However, you may see different results (visits, visitors or conversions) between our reporting system and your Analytics platform (Google Analytics, Adobe Omniture, AT Internet, Matomo..). Small variations are usually “normal” though, even if your integration is correctly set up. This is because different analytics platforms often define metrics differently or are configured differently. Some analytics platforms will not manage ITP issues with the Safari browser as we do, bots filtering rules will not be exactly the same, etc. However, you should start looking for a problem as soon as you have a data discrepancy of more than 7-10% between Kameleoon and your analytics platform.
If you’re seeing large discrepancies in your data, this article will guide you to make sure you’re following some best practices to identify the root cause of it.
This documentation also assumes you are using our native analytics integrations. So please always compare figures for visitors who have been tagged with Kameleoon experiment and variation information, with a custom dimension, custom variable, eVar, etc, depending on the analytics platform you use. Indeed, when a native integration is being used in one of your experiments, Kameleoon will flag your visits when visitors are targeted by the experiment. So your analytics platforms’ data should be filtered to only show data that contains experiment and variation information. Otherwise it will make the comparison unreliable.
Visits / Visitors discrepancies
How are visits and visitors counted
First, you should know that Kameleoon measures unique visitors based on cookies and Local Storage. A cookie is a file placed on a browser that contains an anonymous unique identifier, called the Kameleoon visitor code. It is stored on the user’s browser for 365 days.
Each time the same visitor lands on your website, a new visit is created by Kameleoon and ends as soon as no new activity event (page view, scroll, clicks, etc.) has been received by Kameleoon over the last 30 minutes. In other words, a new visit will be created by Kameleoon after 30 minutes of inactivity.
So, first advice: please make sure both the number of visitors and visits are counted the same way in your analytics platform. For instance, in Google Analytics, there are two methods by which a visit ends:
- Time-based expiration: after 30 minutes of inactivity or at midnight.
- Campaign change: if a same visitor arrives via one campaign, leaves after 2 minutes, and then comes back via a different campaign 2 minutes later, Google Analytics will count 2 visits whereas in Kameleoon, it will still be the same visit.
Please also note that the number of visitors (users) displayed in Google Analytics and Adobe Analytics reports is not an exact count of the number of users which have been browsing your website for any given time frame. Instead, what is shown as numbers of users in all reports is a probabilistic estimate of the true number of users based on the HyperLogLog++ algorithm. To give you more insights about how Google Analytics or Adobe Analytics count their users, you can check out this great article.
Bots filtering
Kameleoon offers bot filtering out-of-the-box which may also differ from your analytics platform. Kameleoon leverages the IAB/ABC International Spiders and Bots List in order to detect events which needs to be filtered. The list is a key resource to minimise non-human traffic being counted in our web analytics. In addition a Kameleoon visit can be automatically rejected from our statistics if detected as an outlier (bot, troll, tracker bug, etc.) this is the case if at least one of the following conditions is met during a visit:
- Duration of the visit > 120 minutes
- Number of events (conversions, clicks, targeting, product, page view, etc.) > 10K
Browser compatibility
Kameleoon does not run on Internet Explorer so any visits from Internet Explorer users will be automatically excluded from your experiment reports. Please also note that Kameleoon only maintains full compatibility with the last 3 versions of any browser, so some visits might not appear in our reports as well.
IP exclusion
In most analytics platforms, you can add IP filtering rules, which will allow you to exclude certain IP ranges from showing up in your campaign results. You should have exactly the same filters in Kameleoon. You can set up them from the configuration page of each project so that we are measuring the same thing.
Ad / Content blockers
Many visitors are using ad blockers such as Adblock, Ghostery, uBlock..to block trackers and advertisers. Some ad blockers can also block client-side trackers such as analytics events, including unfortunately Kameleoon client-side events as well. So if you have a significant portion of your visitors that have ad blockers extensions enabled in their browser, there is a high chance that the number of visits will vary between Kameleoon and your Analytics platform. Indeed, if your analytics platform is not ad blocked whereas your Kameleoon is, the number of visits will be different. Some analytics platforms offer “on premise” tracking request URLs that allow them to avoid being blocked by ad blockers. Please note that Kameleoon can also offer the same capabilities. Please reach out to your Customer Success Manager for assistance.
If you believe you have a large portion of your visitors using ad blockers and that you have more visits and visitors in your analytics platform than we have in Kameleoon, we recommend sending an event to your analytics platform when Kameleoon has loaded so that you can have a clear idea of the percentage of visitors using ad blockers that block Kameleoon. To do so, you can use our Activation API events to send an event to your analytics platform when Kameleoon has loaded. Here is an example of code you can use. Please make sure it runs before the Kameleoon installation snippet.
window.addEventListener('Kameleoon::Loaded', function (event) {
// Here you should implement the logic to send the data to your analytics platform.
// It can be something similar to:
myAnalyticsPlatform.sendEvent("Kameleoon has loaded");
});
//Place here the Kameleoon installation snippet
Loading timing / implementation differences
One of our best practices is to have our snippet implemented in the <head> section of your source code to prevent any flickering from appearing in your experiments. It also means that Kameleoon will usually be one of the first tags to run, so we will send activity events as soon as possible. However, best practices for analytics platforms are completely different. In fact, most of the Analytics platforms will usually be loaded in the <body> section, or after the page has fully loaded. This means that Kameleoon will be sending events before most analytics platforms and will be counting more visits and visitors. Indeed, visitors may bounce, lose internet connection or click back. These visitors will not be counted by your analytics platform, whereas it will be the case in Kameleoon.
Analytics platforms may be installed on more or less pages than Kameleoon
This is a common root cause, especially if you want to run an A/A experiment that is targeting “the entire site” like it is displayed in the screenshot below.
When that’s the case, Kameleoon will simply execute the code of your experiment on all pages where the Kameleoon’ snippet has been installed. In other words, technically, it is like having no targeting at all. So a difference in counts could be due to this reason. For instance, if the exact same Kameleoon snippet has also been installed on your staging environment, your experiment will also run there whereas in your analytics platform there is a high chance you won’t get the visits from this environment. A good way to identify if this is the cause of the discrepancy, is to break down your data by “Visited page URL”. It will show you all main URLs where the experiment has run so that you can identify URLs where Kameleoon should not have loaded (other environments, webviews on mobile apps, etc.).
Best practices for these types of discrepancies are to:
- Always run an A/A test on a small part of your website to make sure you fully master the scope of your experiment.
- If you need to make targeting adjustments to an A/A test, never do it on the same experiment. Stop that one and run another A/A test to verify that the issue was corrected by having fresh data on this experiment. Why? Because when visitors land on your site, they will download the Kameleoon snippet which will be automatically put in the browser cache, so that they don’t need to download our script again on the next pages, for performance reasons. The downside is that if you make new updates to a running experiment, not all your visitors will see your latest updates because some returning visitors may be running the older cached copy of our snippet. So there is a good chance that your experiment will contain data from returning visitors who still have an older cached version of our snippet.
Events (click, transaction..) discrepancies
First, please note that if you have discrepancies in the number of visits or visitors, there is a good chance you will also have the same discrepancies in the number of conversions for a given event (click tracking, scroll tracking, transaction or total revenue). So you should first be looking at why you have visits or visitors discrepancies.
However, if you have similar visitor or visit counts but different conversion counts, below are some guidelines to read to ensure that your events conversions can be compared to Kameleoon results.
Event tracking configuration
The first thing is to check how the event is configured and tracked with your developers and our support team. For instance in Kameleoon a click tracking goal will listen both for mouse down events (for a desktop computer) and for touchdown events (in mobile phones and tablets), which is very different from an onclick event only.
Also, if you have been using the graphic editor to configure your click tracking, please make sure the CSS selector attached to it is the right one. When there is no ID on the selected element, Kameleoon selects the element according to its (unique) hierarchical position on the page, by creating a hierarchical path from one of the parent blocks containing an HTML ID. Thus, a data issue may come from a default CSS selector which is not measuring the right event conversions. Please read this documentation to know more about it.
If you have been using our Activation API to track custom conversions, please make sure that the API function is called after Kameleoon has loaded on your page. We strongly recommend you use Kameleoon Command Queue, which allows for delayed command execution. The principle is simple: instead of directly calling the Activation API via the Kameleoon.API JavaScript object, you pass commands and functions to another JavaScript object, kameleoonQueue. If the Kameleoon engine is already loaded, the commands / functions will be instantly executed, otherwise they will be queued and executed once the engine is ready. This makes sure that all conversion events will be received by Kameleoon.
In a nutshell, always make sure your events track the same things the same way, otherwise the data won’t match at all between Kameleoon and your analytics platform.
Event conversions count
In Kameleoon, for the same event, you can look at converted visits OR all conversions. The default view on our reporting page is “converted visits”, which means that even if a visitor clicks 3 times on an element, we will only display 1 converted visits. So if you want to compare the total number of conversions, you will need to switch to the “All conversions” view in our reports.
Event conversions attribution
In order to give you as much control over your experiment as we can, Kameleoon has developed a feature which allows you to tune how long an experiment influence its exposed visitors. We call it the attribution window and it directly influences how your visitor conversions are attributed to the experiment or not. Basically the attribution window defines the period of time during which visitor’s conversions and transactions are attributed to a given variation.
In Kameleoon, visitor’s conversions only count toward an experiment as soon as the visit has been targeted by the experiment or during this attribution window time. So it means that:
- Conversions happening in subsequent, non targeted visits are not taken into account in our report unless they happen within the time frame of the attribution window.
- If a conversion event fires before the targeting decision occurs, it won’t be counted in our report, unless the visit was already under the influence of the test.
In most analytics platforms, there is a good chance that conversions will continue to be counted even if visitors no longer meet the targeting conditions, so Kameleoon results page may show lower total conversions than in your analytics platform for a same event.
If you would like to know more on how Kameleoon counts conversion we recommend to read this article.
Conclusion
The vast majority of data (visit, visitor, event) discrepancies observed with our customers came down to root causes covered in this documentation. In case the guidelines above don’t point you to anything conclusive, our support team is here to help, so please feel free to reach out. When submitting support requests, please provide as much detail as possible including screenshots of your analytics platform results, code samples of your event tracking configuration, explanations on how your analytics platform has been implemented (classic or on premise).
