3 Frequently Asked Questions About A/B and Multivariate Tests

A/B/n and multivariate testing is one of the most important CRO (conversion rate optimisation) activities for continually improving your website, and yet for some it can be difficult to get started with.

In this post I’ll share three frequently asked questions we hear time and time again from our clients when just starting out with A/B and multivariate testing.

1. We would love to get started with A/B/n testing and MVT but it’s just too difficult on our website platform?

Actually A/B/n and multivariate testing tools will work on almost every website platform  Most solutions just require a simple piece of JavaScript code across all pages of the site to get going (in a similar way to setting up your Google analytics).

Once the site has been tagged you can do pretty much all your testing without bothering the IT/web-development team again.

How is this possible?

A common misconception is that A/B/n testing requires you to actually build an alternative ‘B’, ‘C’, ‘D’ page for each and every test, which would be a very challenging feat if you were working with say an ecommerce platform or complex CMS and if you had infrequent code releases.

Whilst a redirect test (splitting traffic between two physical webpages) is sometimes useful for testing a whole different journey (like an alternative checkout process for example), the majority of tests you’re likely to carry out will not require an actual alternative webpage to be built and hosted.

This is because the modern client-side approach taken by most of the new A/B/n or multivariate testing tools work by altering the original page in the browser using JavaScript (JQuery) and CSS, to achieve the desired layout, image, or content changes.

Therefore version ‘B’ is created and launched entirely through the MVT software and does not require your IT/web-development team’s involvement to upload alternative pages to the server or make any code changes.

2. Which A/B/MVT tool should we use as there are so many to choose from?

There are many to choose from now that the A/B/MVT market is becoming more commoditised  (as you can see from the growing list of suppliers on www.WhichMVT.com). More choice and greater competition is good news for the newbie testers out there, as it’s bringing down prices, helping to improve the quality of the tools and making them much easier to use.

But it’s not always easy to make a decision on a tool that meets all your needs. So other than reading the Econsultancy MVT buyers guide here’s a few pointers to help you with your decision. For those stuck using an unwieldy clunky tool maybe it’s time to change?

Ease of use

This is one of most important considerations when choosing a tool. This is important for both setting up the experiments (WYSIWYG interfaces, code view for advanced users, setting up conversion goals, targeting etc.) and also monitoring and measuring results.

Even if you don’t intend to manage and implement all the tests yourself in-house, it should still be relatively easy for you to carry out say a simple image replacement or messaging test.

Testing alternative creatives for example can be very useful for the marketing team, so why shouldn’t they be able to set up and launch these simple tests themselves?

Also just like with analytics tools, you should expect from a decent testing tool a great test reporting page where you can clearly see the conversion goal metrics for each design variation and corresponding uplift and confidence.

You shouldn’t have to wait for an excel report once the test is finished.

Useful tip: Many of the best testing tools out there are pretty confident in their GUI and will let you to try it out for yourselves before signing the contract.

Testing more than one page at a time

Running tests on multiple pages at a time is essential for optimising most kinds of websites and it should be very easy to do in your chosen testing tool.

There are so many possible tests that will rely on this functionality, such as global header and footer changes, category page tests (on all category pages), IA/navigation changes and product pages.

Also, for many testing ideas it will be essential that what you’re testing (a new button style or change to title for example) is changed in every page whereit exists.

If your tool doesn’t allow you to easily test design variations of multiple pages at the same time then you will miss out on so many testing opportunities.

Conversion goals

Although it’s important to decide on key success metrics for each test, your chosen software should allow you to easily set-up multiple conversion goals from page visits to revenue tracking and even tracking clicks on elements.

A few example conversion goals you may wish to track:

Primary success metrics:

• Sales/orders.
• Revenue.
• Leads/sign-ups.
• Click-through rates.
• The whole checkout funnel pages (e.g. basket, login/register, delivery, payment pages).

Secondary conversion goals (points of interest):

• Tabbed navigation clicks.
• Interacting with the zoom.
• Clicking on thumbnail images.
• Writing a reviews and comments.
• Clicking on top navigation versus footer navigation.
• Filling out forms.
• Clicking on size guides.

Segmentation, targeting and personalisation functionality

Most of the time when optimising your website you’ll probably want to test out your ideas with all your website visitors. However, there will be occasions where your test idea will be aimed at a particular segment of users i.e. a personalisation test.

For example:

• Does making help content more prominent for new visitors increase conversion?
• Would a location-specific message help to engage your users better?
• Is targeting special offers via a voucher code to only a specific value segment a good business decision?
• Would increasing the tappable areas of CTA’s for iPad users prove to be a good idea; and therefore do these tests show that you need to look into building an optimised tablet version?
• Personalising the experience for those coming from a particular PPC campaign by continuing the message or theme – these simple ‘landing page’ tests can be extremely effective.

There are so many ideas you may want to test out with regards to making your website content more relevant to your users and more effective at getting them to convert through website personalisation.

Therefore consider selecting a testing tool with segmentation, targeting and personalisation functionality.

Integrating with your analytics package

If your testing tool integrates with your analytics package then this’ll allow you to analyse the effect of a new design variant B, C and D, in more detail than what’s available in your testing tool reporting interface.

Here are a few useful examples:

• Ecommerce product category to see if version B is actually helping to sell more of a particular product.
• Visitor segment (e.g. new, returning, device, and many advanced segments). This type of analysis can give great insight for personalisation ideas.
• Channel (e.g. natural search, paid search, referral traffic).

Sanitised example of integrating an A/B/MVT tool with Google Analytics

It’s not all about the software

The most important thing to remember is that the A/B and multivariate testing software is just one part of the conversion rate optimisation toolkit, so don’t blow your entire budget on the technology.

Even the most sophisticated A/B/MVT tool still needs a good team behind it, so make sure you leave budget for the right people and the right expertise. 

You’ll have three options when it comes to whose going to run the testing programme:

• Self-managed, choosing just the testing tool and building up an in-house team.
• Fully managed by the MVT supplier. Some MVT companies not only sell the tool but also provide you with professional services to run the testing.
• Fully managed by an independent conversion rate optimisation agency and signing up to an MVT tool separately.

The agency therefore is not tied to the testing tool so if you should you decide to change tools – for example you wanted to change tools in favour of one that offered extra features, you can still carry on using the same team even though the tech has changed.

3. How will we be able to measure the ROI of our testing process?

With A/B/n testing and MVT you can not only determine whether a new version of a webpage, design or combination of elements is better than the control (the original page), but you can measure how much better it is.

You can measure what the new conversion rate (CVR) is, the new average order value (AOV), revenue per visitor (RPV) or how many new leads the winning variation has given you during the test period.

Once you have this comparable and objective data along with a high enough statistical significance to be confident in the result (95% recommended), the great thing is that as well as having a good business case for permanently implementing the winning changes or design, it is now so much easier to work out the ROI of your whole testing process.

The important thing is to include in your ROI calculation all the resources and activities that led to the testing ideas such as the customer research, surveys, usability testing, analytics and your optimisation team of experts.

Not to mention the costs of the development to permanently implement the changes. When you are able to do this systematically, then your conversion rate optimisation process is in a very good place.

Conclusion

Hopefully this post has answered some of your testing questions. And for those who are not yet testing, maybe it’s provided you with that extra nudge that you need to finally take the plunge.

I can assure you that once you get started with a good testing tool and a great optimisation team to run the testing programme then there’s no going back to a ‘cross your fingers and hope for the best’ approach.

If you have any other tips or words of encouragement for newbie testers then please add your comments below.

This article first appeared on the Econsultancy Blog.

About the Author

Chris Gibbins is the Director of UX & Optimization at Biglight.co.uk. He specialises in creating amazing customer experiences across every channel and device for his retail clients. With a background in art and design, his passion for understanding customer behavior and optimizing digital experiences started about 10 years ago after being introduced to the fascinating field of usability and eye tracking. Chris has been known to say that if he could, he would A/B test everything! He recently moved to BigLight after 8 years heading up UX, CRO and personalization at RedEye.