The Essential MVT Roadmap: Test Planning Secrets to Ensure Success

You already know that running multivariate tests can help you improve the usability and effectiveness of your site, but where do you start? Typically you want to examine your web analytics framework and truly understand the most important metrics, known as key performance indicators (KPIs), behind your business goals. Those KPIs, in turn, can be correlated to site factors that can be tested. This blog post will look at how to turn metrics into testable factors, A/B versus multivariate testing, and 5 key errors to avoid when getting started.

Starting at the beginning

If you don’t already use a web analytics framework for your website, it’s high time you start. This framework will make your goals and their measurement explicit and keep you focused on what’s important. Almost every website wants to achieve one or more RACR goals; that is, Reach, Acquisition, Conversion, and/or Retention. If you are focused on more than one of these goals for your business, you’ll want to set up a different framework for each of them, as you’ll be focusing on different metrics and therefore different factors to test.

Framing the big picture

 Let’s look at a very simple web analytics framework for Conversion using a hypothetical online retailer as an example:

With this framework, we now have enough information to figure out which site elements, or factors, to test in order to understand which of them influences visitor behavior. Obviously, you will want track those pages and areas of the site that users click on in the conversion funnel, such as the “Buy Now” button and resulting “Thank You” pages.

Here are some of the things you could test and measure in support of an e-commerce website:

• What elements of the website led to the most “Add to Cart” clicks, followed by successful order completion pages (e.g. “Thank you for your order”)?
• Which combination of product information such as graphics, descriptions, layout, and color increased average order value?
• What combination of factors relating to site search most successfully brought users to pages from which they ultimately purchased products?
• Consider testing coupons and promotions.
• Test offers such as free shipping or financing.
• What about credibility factors, such as logos denoting secure credit card processing?
• Does the availability, placement, or look and feel of customer reviews and testimonials make a difference on purchase decisions?

Those are just a few things to think about. You’ll want to start with the factor(s) you think is most important to your KPI(s) and decide what experimental design is best. With A/B testing, you test one factor, such as a call-to-action button or a hero shot, against one or more variations to see which is most persuasive. While A/B testing allows you to test just one factor at a time, multivariate testing enables you to test many changes simultaneously. Evaluating the impact of combinations of factors and variations often reveals significant interaction effects that can have a dramatic impact on your conversion goals.

Common errors to avoid

There are common mistakes that are easy to make when running multivariate tests. Here’s one of them:

Improper factoring caused by poor or no isolation of individual test changes; for example, changing a headline’s text, font color, and font size, all at the same time as an A/B test instead of a multivariate test.Why is this problematic? Because it’s difficult or impossible to isolate the impact of each individual change — i.e., was it the font color and/or the text that caused the visitor to behave differently?

About the Author

Eric Hansen is the CEO and founder of SiteSpect, and the chief architect of the firm’s non-intrusive technology for multivariate testing, behavioral targeting and digital marketing optimization. He is a frequent speaker at conferences covering web analytics and optimization, and writes regularly on topics dealing with the intersection of marketing and technology.