Time, Budget Are Barriers to Web Testing by Online Retailers: Study

The majority of online retailers know that testing their Web sites and incorporating the findings of those tests can improve the user experience and produce an overall increase in site conversions. But only about one-third of ecommerce site managers say they test their sites actively, and barely more than that say they test “once in a while”.

Those are some of the findings of a new joint report from Web testing platform Amadesa and digital retail consultant The E-tailing Group. Those firms surveyed 160 leading Web retailers—both pure-play online and multichannel sellers– in the last quarter of 2010 and followed through with one-on-one interviews.

The study found that 71% of respondents said they had performed Web site testing at some point in the past. But asked about the current status of their Web testing efforts, 33% characterized the testing initiative on their site as “very active”, while 38% admitted to testing any part of their Web presence sporadically or “once in a while”. Eight percent said they intend to test portions of their site in 2011, and 7% said they have never done Web testing before. None of those polled reported having tested but then dropping plans for future tests.

The implication of the findings seems to be that many Web merchants treat testing like joining a gym for exercise: it’s something that can undoubtedly be good for their digital health, if only they could find the time, resources and money to do it.

“We found that a lot of Web marketers are doing [testing] to some degree,” says Rebecca Oistad, marketing director for Amadesa. “People who have been testers aren’t abandoning it altogether. It’s just that people are testing to different degrees of frequency. Most people were in agreement that they needed to be testing, although sometimes other priorities take precedence over a full-blown testing program.”

Thirteen percent of respondents said explicitly that while their companies believe in the benefits of testing their Web sites, “it just never has been a high enough priority to be activated.”

Among the small segment who told Amadesa researchers they have never tested, funding seems to be the biggest hurdle; 61% of the test-averse reported that they have no budget to support the initiative. And asked what could convince them to make Web testing a priority, 58% of the non-testing group said they would do so if they got the monetary resources to support the initiative.

But testing is apparently not part of the internal culture at some of these non-testing respondents; about a third said they don’t test because it’s just not an internal priority. The same number said there was no one on their management team who was functioning as a champion of Web testing either continuously or on a spot basis.

Finally, 27% said they avoided doing Web testing because they “believe other priorities will deliver great ROI.” And again, asked what might encourage them to begin testing their sites, 42% of this group said they would be convinced by research that linked testing firmly to ROI benefits for the Web property. That compares to only 21% who said they would be motivated by a decline in key performance indicators at the site.

Most of the respondents who do currently overlay testing on their Web sites do so monthly (28%), followed by 23% who say they run tests of some kind weekly. Only 13% of the e-retailers polled said they perform daily site tests, while 12% said they do so quarterly and 5% seasonally. Another 19% simply said they run Web tests “periodically”.

Asked which components of their Web site have been most improved by testing, the majority of merchants (61%) pointed to its positive effects on the user experience, while another 57% said they had seen an overall increase in site conversion, however they defined that.

Other more specific Web components may have contributed to those large improvements. For example, 325 of respondents who had tested said their search results or landing pages had been improved by undergoing testing, while a slightly smaller 31% pointed to testing’s benefit for their site navigation. About one fourth of respondents (2450 said testing helped to increase their average order volume, and one-fifth said testing at the Web site had cut down on shopping cart abandonment. Seventeen percent pointed to refinement of their product categories or classifications.

In terms of what testing modes they’ve applied, respondents by far (82%) favored straight A/B head-to-head testing of alternative content or formats, followed by segmentation testing that delivered different content or experiences to different users (37%). Multivariate testing—changing multiple factors in different combinations to determine the optimum pattern—was employed by 35% of poll respondents. And 27% said they were targeting differing content to users based on pre-established rules.

The popularity of relatively simple A/B testing bears out the great quantity of anecdotal opinion from merchants in the study that testing should not be allowed to get too complicated. As a result, Amadesa and The Etailing Group recommend that testing should start with short-term aims that can produce immediately actionable results.

For example, the study found that the majority of online merchants who tested on their sites in 2010 applied it to elements of their creative execution (65%) and the Web copy itself (50%), largely because these can be quickly optimized once the best formula is found. About 46% of merchants tested promotional options such as percentage rebates versus flat discounts, and 45% say they ran tests on broad marketing campaigns.

Also, 43% said they tested some aspect of their Web imagery in 2010, and 42% ran tests on the templates of such key pages as their home page, category or product pages, and their shopping cart. Forty-one percent said they tested their general user interface; 36% tested landing pages for search results or pay-per-click ads; and one-third ran tests on their site navigation.

On the relatively low end, 25% of Web retailers who used testing last year used it to try out pricing strategies, and 19% used it to manipulate brand strategy in some way. The same percentage tested Web content tactics such as applying video to a product page, while 17% tested tactical placements such as deploying customized product recommendations for their visitors.