Self-Deception: The Cookie Crumbles

Note: This article has been through several rewrites since it was first started. By and large, this is due to increased industry chatter regarding the issues surrounding cookie-clearing and cookie-blocking. Several weeks ago, the issue appeared to be one the interactive marketing industry would rather not see publicly discussed. However, since that time, several writers and industry leaders have taken notice of some of the effects of cookie-clearing and cookie-blocking on campaign metrics and have written several articles. Nevertheless, the issue is still underreported and healthy discussion needs to continue in order to resolve it.

The issue of cookie-blocking as a potential threat to the de facto method by which most online marketers track direct response metrics on their online campaigns has been on the radar screen for quite some time. The earliest I can remember this being an issue was in 1998 when rumors abounded that future versions of Microsoft’s Internet Explorer would block cookies by default. Although these rumors proved to be untrue at the time, the industry took notice of the issue. However, since then, cookie blocking and cookie clearing are much more commonplace with consumers.

To outline the problem from a technical perspective, ad servers employed by online marketers and agencies use cookies in a variety of different ways to serve and track online ads. When a consumer views an ad, an ad server will read or write a cookie (or both) as a way of later determining whether or not a consumer has been exposed to a campaign. Ad servers also read cookies to determine the appropriate destination URL that should be served to the user, should they happen to click on a served ad. Additionally, cookies are used to determine if a user takes action on an ad, regardless of whether they’ve clicked on the ad or not. Cookies also have applications in determining the appropriate ad to be served, in the event that a user has been exposed to an ad previously and the marketer wishes them to see a different message. Can you see the extent of the problem in the case that cookie blocking or clearing becomes commonplace? When web users block cookies, several unexpected things may happen, including:

1. The ad server may treat the consumer as a “new user” even if they’ve seen ads from a specific campaign previously. If ads are sequenced, the user will be exposed to the first ad in the sequence instead of later iterations as intended.
2. The ad server may also inflate its count of unique users exposed to the campaign (unique reach).
3. The ad server may direct the user to the wrong URL upon a user click.
4. The ad server will be unable to attribute actions to the ad that referred it should the user come to the advertiser’s site and take action.

If only a handful of consumers exposed to an ad campaign block or clear cookies, marketers tend to have few problems with a very small number of cookie-related hiccups within their campaigns. After all, server latency tends to be a much larger problem in the grand scheme of things and tends to cause more problems than blocked cookies. If you ask most professionals who work at ad serving and ad management companies, they’ll likely pooh-pooh the problem, claiming that such a low percentage of consumers block or clear their cookies that the effect is negligible.

However, the industry has seen evidence over the past couple years that the problem is bigger than we might have thought. Yet, the issue is being practically ignored. No one considers this issue a major agenda item at online advertising industry conferences, workshops, task forces and discussion communities. It appears many people would rather continue to put blind faith in the numbers, despite evidence that the technology model may be threatened.

The first threat to cookies comprises browsers and web clients that block or restrict cookies. The web clients that make up the overwhelming majority of the browser market have the capability to block or clear cookies. For example, Microsoft’s Internet Explorer provides customizable levels of privacy for the user. Set at the highest level, the browser blocks all cookies. Of course, this may present problems for the user in navigating registration-based sites or online shopping carts, but it is possible to surf the web and accept no cookies along the way. Mozilla Firefox has additional features and layers of customization pertaining to user acceptance of cookies. For instance, the browser features one-click clearing of cookies, as well as options that allow the user to selectively block. One such option, “Enable cookies, for the originating website only” would effectively block cookies from all ad servers.

The problem is not limited to browsers. Certain e-mail clients block ad server cookies in many cases, including Microsoft Outlook 2003. Recently, we encountered an issue in which a newsletter ad failed to click through to its proper destination URL. The problem’s source was Outlook’s blocking of the ad server cookie such that the ad server could not determine which ad was displayed to a user viewing an HTML newsletter. The ad server defaulted to the first click URL in the ad queue, which didn’t match up with the ad displayed, and directed the user to the wrong destination on click. After discovering the issue, it took a significant departure from the usual way online advertisers serve ads into HTML newsletters to fix the problem, yet I haven’t seen any sort of notice or service bulletin from the major ad serving providers to address the problem.

Perhaps one of the biggest effects cookie blocking has on advertising campaigns is skewing measurement of reach and frequency. If a web user clears his cookies after first being exposed to an ad campaign, that user is treated as a unique user twice. If the user clears cookies regularly, he may be counted multiple times in this way. Let’s say that Fred sees an ad for IBM on a website he visits once per day. The first time that Ed encounters that ad, he is counted as a unique recipient of the ad message by the ad server that served the ad. On subsequent days when Fred encounters the ad, the ad server realizes he has seen it before and attributes additional frequency to Fred and to the campaign. If Fred sees the ad seven times in seven days, the ad server counts Fred as one person with a frequency of exposure of seven. Now let’s suppose that Fred falls into the practice of clearing his cookies three times per week. Suddenly, Fred is seen by the ad server as three different people, each having been exposed to the ad between one and three times. Reach is inflated by 3X, while average frequency is undercounted by a factor of 2-3X. See how this can become a problem?

A similar effect is seen with syndicated research studies that rely on cookies to provide measurements of unique reach. Audience may be over- or undercounted depending on how often users clear cookies. The effect is one of tilting the playing field with regard to how sites perform against various audiences in comparison to one another.

The extent to which cookie-clearing and cookie-blocking is a problem is anybody’s guess. A scientific look at the extent of the problem would be incredibly difficult to execute, considering it would require visibility into the preferences of web users – how often they run spyware removal utilities, what browser clients they run and how they manage security using those clients, what e-mail applications they run, etc. It may be impossible to gauge this effectively and reliably without violating web users’ privacy.

Anecdotally, however, we should be looking at how our fellow web users are changing their behavior with regard to cookies. We know from our own personal experience that it is necessary to run anti-spyware utilities to keep our Windows PCs running smoothly and without crippling spyware cluttering up our systems. Two of the most popular free anti-spyware utilities, Spybot - Search and Destroy and Ad-Aware, have been downloaded nearly 170 million times from C|Net’s Download.com alone as of this writing. It should be noted that both of these utilities identify ad server cookies as spyware as a default, and will typically remove these cookies along with any rogue applications that may have infiltrated the user’s system. This alone should be an indicator that the problem can no longer be ignored.

On top of that, think about how many people have downloaded and migrated to Mozilla Firefox since it debuted, how many people heighten their security settings on their browser of choice, how many surf the Internet through a proxy, and how many are using e-mail clients that block cookies in HTML newsletters. While we don’t know how big these universes are, we at least know they’re rather large.

Jim Meskauskas suggested in a recent Online Spin piece that the solution is education. He would like to see the organizations of record for the online advertising industry launch an educational effort about cookies. I think we need much more than that, including:

• Negotiations between major ad serving companies and the anti-spyware software developers. These developers need to understand that cookies do not collect personally-identifiable information (PII) and should not be classified as spyware by default.
• Negotiations with developers of web browsers and e-mail clients to provide educational material to users of their products, such that they understand the ramifications of blocking or clearing cookies. Default security settings should not block cookies that do not collect PII.
• Development of new technologies in ad serving and tracking that do not rely on cookies to provide metrics.
• Healthy discussion regarding the extent of the problem within the industry.

Without these initiatives on the table, online advertising runs the risk of losing what gains it has made in terms of accountability.