This weekend, we released some major changes to referrer and UTM parameter processing.
UTM parameters (utm_campaign, utm_source, etc) are over 20 years old now, and were originally designed to track online marketing. Over time though, their usage has become much more casual, as referrers have been slowly reduced to a rotting corpse thanks to Google, who started this whole thing with their “secure search” trial back in 2010.
Our initial support for UTMs was also released in 2010, but unfortunately this was 4 months before Google’s secure search trial. We would have designed the system entirely differently if it had been at least a year later, but since referrers were still the shit in January 2010, we decided to only track UTMs for their intended purpose, which was marketing. Back then, utm_campaign was the “umbrella” UTM parameter, so we made the unfortunate decision to only track UTMs if utm_campaign was present, otherwise they were ignored. Over time we made some minor improvements, but mostly it only worked well if UTMs were being used for the original purpose and design.
We’ve wanted to change this for a long time, and finally had the proper motivation: tracking AI, which we recently teased on Twitter (we’ll have more news about that next week).
In order to do exactly what we want, which is well beyond what’s shown above, we first needed to allow UTMs to do a lot more… so here we are.
The new Campaigns help page covers nicely how the new system works, but the main updates are as follows:
- utm_source can now set the third party referrer, as long as it passes anti-spam tests and there isn’t a referrer already. (This also means we now allow “non-domain” referrers. We’ll improve how these are displayed in the UI over time, e.g. removing the clickable link.)
- utm_medium will influence our traffic source classification, even if there isn’t a referrer. Previously, a referrer was required in order to be anything other than “direct” or “advertising”.
- utm_term will backfill search keywords into secure search referrers, including if there is no actual referrer but utm_source includes one such as “google”. Previously we did this for “google.com” referrers, but the code has been vastly improved to work better, and supports a lot more search engines now.
- Common UTM alternatives and misspellings are supported: ref, utm_referer, utm_search, etc
- New campaigns being logged may break away from existing campaign history if any of the values regularly logged are >64 characters.
- Our tracking code has been updated with better third party domain detection for outbound links. This was needed to support tracking outbound links for customers that included their domain name in “ref” or “utm_source”, which previously would have been ignored if the domain name was *anywhere* in the URL, instead of just the domain name itself.
- We’ve also improved referrer traffic classification, when UTMs aren’t involved.
One thing that’s not changing however, is that Clicky still only stores all actual UTM parameters “as is” when utm_campaign is set, as that is required to invoke the full campaign system.
Remember, this was all initiated because of our desire to track AI visitors. So part 1 of our AI initiative was included in this release, which is the new “AI” traffic source type (as shown in the tweet above). Part 2 will include tracking AI assistants, and much more. Stay tuned!
“Advertising” will soon be a separate flag from the actual source
One thing we’ve always disliked is how traffic sources are limited to one per visitor, in particular because “advertising” will override everything else. For example, if you ran ads on social media, those visitors would all be in the “advertising” bucket, and not the social media bucket with the rest of the people who clicked organic links. You could still filter by domain so you could overall traffic for one particular network, but clearly this needs improvement.
To address this, we’ve created a new advertising flag in the backend that is already tracking now. We plan to have paid vs organic be determined solely by this flag, so the visitor can always keep their main traffic source classification (social media, AI, etc) as determined by the referrer. This way you can see all social media users at once, and then filter down between paid or organic. Or you will be able to see all paid visitors, and then filter down to search, social, ai, etc. That is perhaps a month or two away, as we’ll need to make a few other changes in the meantime, and potentially update historical data too, but it will be a huge improvement.
Soon™️!!!
