I created a SlideShare to show you a side-by-side comparison of the two landing pages and an explanation of the changes I made to optimize the results. You can check it out below (spoiler alert: don’t view this SlideShare before you try to predict which landing page performed better on WhichTestWon.com, that is cheating!):

This case study ran publicly on WhichTestWon.com for one week between Wednesday, October 26th and Wednesday, November 2nd.  If you click through to it now you will see it has been moved to the “premium” section of their website where it is still featured but only for subscribers who pay for a premium membership.

I worked pretty hard to promote traffic to this post.  I issued multiple Facebook and Google+ status updates about it, Tweeted about it, posted status updates on LinkedIn, I posted links in relevant LinkedIn groups and comment threads, and posted comments on other marketing blogs under relevant topics.  And of course I blogged about it here.  Thanks to my friends and followers who shared the link and echoed the promotion (notably: Julie Danahy Hebeisen, Stacey Clement, Judi Cook, Reid Overcash, Hank Anderson, and Kevin Dickey).  Lisa Seaman, editor of WhichTestWon.com, tells me my case study received more traffic, submissions, and comments than any other case study in the past several months (bit.ly tells me my shortened link was responsible for 27% of the clicks).

It was fun to watch the results and the comments of the responders.  To my surprise, the response to the case study was split almost evenly, with 49% guessing that option A was the winner, and 51% (correctly) guessing that option B was the winner.  The thing that seemed to throw many people off is that “3 minute download” was part of the headline of the old page, and people thought this would be an important feature that would make more people respond. (Note: in the optimized landing page I did not delete this benefit, but I moved it from the headline to a less prominent position on the page.)

The truth is, the changes I made to the landing page were almost entirely textbook Landing Page Optimization fixes that are well-documented if you do your research (check out the SlideShare linked above, the last page of it has links to several good articles on Landing Page Optimization).

The fact that this audience of professional marketers could not tell which of two pages drove leads at double the rate of the other is a little bit shocking and disconcerting.  I think it reflects some general misconceptions about landing pages, and some lack of experience by some of today’s younger marketers:

After the test ran, I congratulated Lisa Seaman because they have created a website/blog that is so fun for marketers to interact with that it feels like a game.  You are challenged each week to use your marketing instincts to predict which version of an A/B test is the winner. Sometimes it is obvious, but more often than not it is difficult to predict the results and tricky to explain why one variation works better than the other. The moral of the story is you should always test, and you can really get burned if you just rely on your gut instincts.