Multivariate Testing Shows Sex Doesn’t Sell Cosmo Magazine

Anyone looking to design a direct mail prospecting package for Cosmopolitan magazine should know that sweepstakes don't boost responses, tote bag premiums don't justify their cost and above all, sex doesn't sell. At least, not on the outer envelope. (Inside the package is another story.)

That last point may be surprising, but the magazine conducted two multivariate tests within the past 18 months. The tests offer statistically valid results for the impact 12 changes had on response rates.

Head-to-head comparisons of a dozen elements would require more than 4,000 cells (test packages), each with a quantity large enough (5,000 at a minimum) to ensure confidence. Since most publishers might get queasy at the thought of mailing more than 20 million test packages, Cosmopolitan used a multivariate matrix to determine the impact of each proposed change.

Here's how it works: Imagine a grid with a list of elements—again, in Cosmopolitan's case, 12—running down the left side. For Cosmopolitan, these tests were:

* A higher price
* Adding a postage and handling fee
* A sweepstakes
* A "fast offer" (a high-value premium given to only a handful of respondents—in this case an iPhone or iPad);
* A paid-order premium (a tote bag)
* A three-year subscription option
* A change in creative that highlighted the one-year subscription
* A change in creative that highlighted the three-year subscription option
* An added letter from the publisher
* A peel-off involvement sticker (Place Sticker Here!) feature
* A fake barcode added to the outer envelope
* Changes in copy that stressed the magazine's sex-related content.

On the Grid
Next, imagine naming each test with a letter. Each test was given one column along the top of the grid. In Cosmopolitan's case, there were 16 such test cells, labeled A, B, C and so forth.

A grid with 12 elements tested among 16 packages yields 192 boxes. Each box will contain either a one or a zero (or a plus or minus sign, or what have you). A box containing a one indicates that particular test cell has incorporated the proposed change. A zero means that particular cell would rely on the design of the control package for that particular element, e.g. the standard price, no peel-off-sticker, no changes to the copy, no sweepstakes and so forth.

For an example of how such a grid might be structured, click here. This isn't the grid Cosmopolitan used, but it shows how the presence and absence of each test element is evenly distributed among each row and column.

By staggering which elements appeared in each of the 16 cells, Cosmopolitan was able to test the impact of all of them.

Among the 16 versions Cosmopolitan tested in a June 2010 mailing, test cell E, which featured the sweepstakes, the tote bag premium, copy highlighting the one-year subscription, the letter from the publisher and the sex-focused content generated a campaign-best 29.1% lift over the response to the control package.

Take a Step Back
"But you need to step back with a multivariate test," says Gordon Bell, president of statistics-based marketing consultancy Lucidview. The mailing tested only 16 of the 4,000+ combinations of new elements. The likelihood that Cosmopolitan had mailed the very best package was slim.

"Chances are there is some bad mixed in with the good," Bell adds.

The challenge, therefore, is to isolate which changes contributed to the lift from which had no effect and those which were detrimental. The matrix, or grid, was set up so that each test element was included in eight packages, and excluded from eight packages.

Lucidview averaged the lift rates of the packages in which each test element appeared. This allowed Cosmopolitan to assign a lift percentage to each proposed change, independent of the other effect of the changes.

How did each proposed change fare? Unsurprisingly, the suggested price increase generated a 37% drop below the control package's response rate. The tote bag premium boosted response rates by 11.8%. An increase in sex-focused copy pulled in 8.5% more responses than the control. Including a postage and handling fee resulted in a 77% falloff from the control's response rate.

The other suggested changes had no significant impact on responses. Given that some generated additional costs, it was easy to eliminate them as considerations.

The beauty of the multivariate matrix is that proposed changes can also be evaluated in combination with each other, in addition to a solo basis. Consider the tote bag premium. Including it raises response rates, while another test, the addition of the postage and handling fee, lowers response rates.

Isolating and analyzing only the test cells that contained both of these allows Cosmopolitan to determine whether there is an increase in response rate, and whether the additional revenue generated offsets the cost of the premium. By analyzing both of these, the magazine could potentially keep the postage and handling fee while including the tote premium and still come out ahead, if the bottom line profit and loss works. (As it happened, it didn't.)

Sexy, But Not Too Sexy
But Cosmopolitan wasn't done testing formats. Its designers were intrigued by the 8.5% boost above the control response rate the sex-related internal copy generated.

"Since the sex-focused copy was so successful, we decided to make the whole package sex focused," Vladimir Damianov, senior promotion director at Hearst magazines says.

Once again, Cosmopolitan rolled out a multivariate test mailing, this time pitting what it called the "sex package" against a mailer consisting of the top elements from the June test. Oddly enough, the sex package bombed.

Part of this, post-campaign analysis revealed, was that it went too far. One single design element, including a sex-related teaser note on the outer envelope, hurt response rates by 6.6%. "People did not want to be pointed in a more risqué direction right up front," Damianov surmises, as the packages that featured more sexual content on the inside did well.

The January test yielded other important learnings. Offering the tote bag boosted responses above the control by 17%, but the additional revenue generated didn't offset the premium's cost.

Cosmopolitan rolled out a new based on the winning elements of the 2010 test, with some tweaks to the copy, as its control in March. The new package is sexy, but not too sexy. Which is probably just as well: The publisher is likely excited enough by the boost in response rates.