What Should You Put on Your Home Page? (how Amazon.com does it)

Here's an excellent presentation (PDF) on how Amazon measures their Web site performance, with an emphasis on A/B testing -- running experiments where one element on a page is changed to see which performs significantly (statistically speaking) better.  My favorite bit (paraphrased from slide 22):

• Automation replaces intuition:
  • home page is made up of slots
  • anyone can submit content for a slot
  • best-performing content (based on A/B testing) wins

For those of you considering an e-commerce system (or upgrading the one you currently have), you should ask if/how the system supports A/B testing.  Of course, this requires that you are already generating significant traffic to your Web site, which means people can find it, and that they are effective when using it (i.e. they reach the goal that they had in mind when visiting).

I'd be interested in hearing from Cultivate, eWinerySolutions, or Inertia on the topic of A/B testing and how their customers might do it using their product offerings.  Measure and test is the heart of Web site success.

Thanks to Adam DuVander of Simplicity Rules for the pointer.

(PS - A simple way to start tackling this issue is to have a "Most Popular Pages" list in the sidebar of your home page above the fold.  At least this points visitors to pages that other visitors find interesting.  Bottom line: you need to respond to how users are voting with their clicks.)

Comments

[Paul Mabray]

Mike,
Great question. We have about 10% of our winery partners doing A/B testing and our system can support it (especially the new bells and whistles coming in our new release).

However, we actually discourage wineries from doing A/B testing UNTIL they have proper resources (time and people) and a solid amount of traffic.

The initial A/B testing we recommend is with email offers. It is easy, and a winery can see quick wins. We then break up our clients based upon SKU totals to help them in promotional A/B testing (free shipping call out vs. a discount call out vs. an accolade call out). Finally, for our larger clients it is product A/B testing with offer A/B testing in conjunction.

I have to again reiterate - we discourage our wineries from doing A/B till they are sophisticated enough and after they have a strong plan and program in place. Otherwise they are distracting themselves from doing the essentials to chase something that is incremental lifts in sales. As you've found Mike, the wine industry currently isn't hiring the web/e-commerce expertise to make this a core focus and thus it becomes an interesting exercise, not a core function.

From my point of view, I'd just like to see the industry write a good plan and metric themselves against it for direct sales and include the website/e-commerce part in that plan. "Fail to plan and you plan to Fail" seems one of the core reasons web/e-commerce initiatives fail. I am hoping that if the industry can gain some success as a result of their own work, in a year everyone will be talking about A/B testing, personalization, and more. My fingers are crossed.

Inertia - Powering the Wine Revolution

---Paul Mabray - CEO

[eljefe]

It seems like there is a little bit of blurring between "home page" and "landing page" going on. If I am doing A/B testing of an email campaign, I would be using landing pages. If I am talking about the default landing page - the one you get when you type in the URL of off our cork - that's the home page. In fact, I bet it would be better to speak of it as the default landing page... with any page on your site being a potential entry point from a search result, it seems you really need to think about what needs to be on every page...

[Mike Duffy]

Jefe: good distinction, thank you. It brings me back to something I ask when consulting: Do you know the most popular page on your site? Frequently, that's the home page. Do some optimization of that page (A/B testing being one approach), and then maybe the next one. Your biggest impact comes from improving the most-seen pages on your site. That's why it's important to look at your server logs (or use Google Analytics) to understand where people start their visit to your site.

What have your experiences been with this sort of effort?