Cookies and the Deletion Dilemma

A March report from JupiterResearch has triggered something of a tumult in the industries that rely on Web metrics with its findings that one of the key tools for measuring Web traffic may be much less reliable than previously thought. If those findings prove out, they could affect the way many areas of Internet marketing—including search marketing, behavioral targeting, affiliate marketing and site personalization—calculate their rates of return on investment in online advertising.

Basically, the conclusions in the Jupiter report, titled “Measuring Unique Visitors”, suggest that measuring Web audiences using cookies may be an inaccurate tool, simply because more users than previously thought either block those cookies or delete them more quickly and more often.

Since cookies are used to detect unique or first-time visitors to a site, those practices by consumers could inflate the number of unique visitors produced by an ad, including paid-search advertising. Cookies are also a way Web site operators can track returning customers and thus can help in calculating the average number of visits before a sales conversion and the lifetime value of customers. There too, unexpected consumer cookie behavior could throw those measures into doubt.

Eric Peterson, senior analyst for site technologies and operations at JupiterResearch, says he and his colleagues thought of testing for cookie deletion earlier this year when they received surprising reports from several Web merchants about steep spikes in their “anonymous visitor” traffic even as all the other data held steady. The research firm included some questions on consumers’ cookie handling in a general Web behavior survey and, when the responses suggested that something was going on, fielded a survey specifically on the cookie question in February and March.

The top-line findings of that survey indicate that 10% of consumers delete cookies daily, while 17% do so every week and 12% get rid of them monthly. “That’s a surprisingly high deletion rate,” Peterson says.

Although 52% of respondents to the survey said they had deleted cookies themselves in the previous year, it’s thought that the biggest cookie-killer has been the increased used of anti-spyware programs, many of which do not distinguish between cookies and the intrusive tracking programs known as spyware that hide applications on a computer, often without the user’s knowledge or consent. These applications can collect truly private data, re-route page requests, deliver unwanted pop-up ads and perform other illicit feats.

According to the Jupiter report, Web users’ fear of spyware is taking its toll on cookies, both at home and at the office. Fifty-eight percent of respondents said they had installed, updated or run anti-spyware programs in the year before they were surveyed; 41% had installed or updated a firewall, many of which also block cookies; 27% said they had installed, updated or run software specifically for deleting cookies. And 56% said they had deleted the cached or temporary files on their computers in that year, which would also have the effect of removing any cookies placed.

Other behaviors suggest that consumers are taking an active part in managing cookies, determining whether to accept them at all and, if so, from which sites. Jupiter found that 25% of users polled said they had blocked cookies from being set, while 28% had changed the cookies that they would accept or block. And 21% of survey respondents said they had added a site to the “safe” list in their browsers, allowing cookies to be placed.

As problematic as these behavioral findings might be, the survey’s revelations about consumer attitudes toward cookies points to an even bigger issue: namely, how users understand the reason for placing cookies. Basically, 52% of respondents told Jupiter that they were concerned about their privacy and security when online, and 44% thought that deleting cookies was one way of protecting their information and Web activity from intrusion. A full 38% of respondents said outright that they consider cookies placed at Web sites to be an invasion of their privacy.

“The pattern that emerged from these attitudinal questions is that consumers don’t in fact understand what cookies do,” Peterson says. “They lump them in with phishing, fraud and all the other dangers we hear about almost constantly to our privacy and security online and offline.” While phishing and other scams are growing harder to detect all the time, cookies are easily dealt with, either through anti-spyware programs or by dialing Web browsers up to the highest security levels. “Given something that’s easy to do and that they feel will protect them, more consumers than we expected are taking advantage of these options,” he says.

Software cookies serve two general purposes: to help Webmasters, merchants, advertisers and analysts track aggregate visitor activity; and to personalize Web sites so that consumers can avoid multi-step log-ins when they return to favorite sites. Registered subscribers to the Wall Street Journal Online, for instance, can avoid having to re-enter their ID and password by simply telling their browser to “remember me” on future visits. Cookies also let users of portals such as My Yahoo! select the content they most want to see on their start pages.

But Peterson says the results of the Jupiter survey indicate that consumers have made their own judgments about cookies—judgments that don’t necessarily jibe with reality. Forty-two percent of survey respondents said they doubt that cookies positively contribute to their Web experience by personalizing their Web sites.

“I don’t think that consumers are saying that they don’t value the function of cookies,” Peterson says. “They’re saying instead that they don’t believe that’s what cookies are for.” In a climate where Web users are hearing on all sides, from Congress and the Federal Trade Commission to the popular press and their own service providers, about threats to their privacy and personal data, consumers are taking what measures they can to feel more secure—even if those measures don’t actually do anything to deter the very real dangers.

“It’s like locking your house or your car when you leave it,” Peterson says. “A determined thief will get around those measures, but they seem like commonsense precautions.”

This consumer skepticism about the personalizing value of cookies underlines the distinction between first-party cookies—those left on a computer by the Web page currently being viewed, as in the My Yahoo! example—and third-parties cookies, in which viewing a Web page inserts a cookie from a domain other than that of the current page. For example, a visitor to a Web site on model railroading might get a cookie from “www.cookieads.com.” These cookies could then be used to amass aggregate data about users’ Web viewing behavior and to place targeted ads from the fictional CookieAds network.

Most browsers and firewalls and much anti-spyware distinguish between first-party and third-party cookies and allow users to keep the former while sweeping for the latter. But many of these also set “refuse cookies” as their default, requiring that users actively go in and adjust settings. Many also include explanations about the difference between first- and third-party cookies in their FAQ. The high rate of cookie deletion suggests that consumers are educating themselves about that difference and perhaps deciding that they want to be rid of third-party cookies in particular. One vendor interviewed for the Jupiter study reported that users’ blocking of third-party cookies had risen from less than 3% in January 2003 to 14% of visitors and 20% of purchasers in January 2005.

The Jupiter report has been something of a bombshell detonated among the complex of players in Web metrics and Internet advertising. But it’s been a peculiarly silent bomb so far, perhaps because the study identifies no easy solution to the cookie deletion problem. (Several Web analytics companies were contacted to participate in this story; all declined.)

The quickest response came from a handful of software developers who sell an opt-in system based on Macromedia Flash. These solutions leverage Flash MX’s shared objects to store and retrieve information, allowing cookies that have been deleted to be reconstructed. United Virtualities saw the Jupiter report as a timely moment to launch its “Persistent Identification Element” (or PIE, continuing the baked-goods theme). PIE and other technologies let Web sites use a line of Javascript code to place a “local shared object” on a viewer’s computer. Mookie Tenembaum, CEO of United Virtualities, says that PIE and

But the fact is that Flash shared objects can stand in for cookies today primarily because most consumers don’t know how to delete them. Given time and will, they will learn how to tweak their Flash browsers, or anti-spyware providers will learn to do it for them. Tenembaum himself foresees an ongoing battle between the factions that want to be able to measure Web use and Web users themselves, who seem motivated to keep their behaviors hidden. “It’s an arms race,” he says. “We come out with a new tool for analyzing the Web, and in time the anti-spyware developers will come out with a way to defeat it. By that time, I expect, we’ll have developed yet another tool that they can’t defeat right away.”

Peterson agrees that anti-spyware providers will naturally take aim at any technology that seems to promise a way out of the cookie-deletion dilemma. He points out that seven of the 10 anti-spyware programs rated most popular by CNET offer to delete cookie automatically as well as spyware. The other three, including the platform adapted for Microsoft’s MSN subscribers, take a “more responsible” approach by recommend that users manage their cookies manually and explaining their benefits.

In fact, Peterson worries that by simply mentioning the Flash stored objects alternative in its report, Jupiter may have drawn the attention of anti-spyware developers to a new and lucrative target. “Their vested interest is in finding more and more spyware on the Web, not less,” he says. “They’ve already benefited by labeling cookies as something intrusive to be deleted.”

Possible non-tech solutions to the problem of the disappearing cookies include testing cookie rejection rates over time to arrive at a “fudge factor” that will let Web advertisers more accurately calculate their ROI. Or Web site operators could convert to a registration model that would send visitors’ unique identifiers directly to measurement applications. Of course, this log-in approach can erode traffic to a site if visitors find the registration process cumbersome, exercising its own damping effect on ROI.

The most effective solution would be a campaign of user education on the nature and benefits of cookies—including possibly a discussion of the need to subsidize the “free” Web content that consumers like with ad revenues via third-party cookies. Lacking such a proactive campaign, and facing anti-spyware developers whose interests lie in ratcheting up the paranoia level about online privacy, Peterson admits that he doesn’t see a final resolution on the near horizon.

“I’ve been asked if I think the problem’s going to get better or worse,” he says. “I’d have to say I think it’s going to get worse, because the only real correction will be a change in consumer attitudes.”