Celebrating Failure: Why the H@^& Did That Test Fail?

After attending Conversion Conference East in 2011, my ideas of testing on the web took a decidedly scientific turn. The Oxford English Dictionary defines the scientific method as: “a method or procedure that has characterized natural science since the 17th century, consisting in systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses.” Wikipedia continues, “The chief characteristic, which distinguishes the scientific method from other methods of acquiring knowledge, is that scientists seek to let reality speak for itself, supporting a theory when a theory’s predictions are confirmed and challenging a theory when its predictions prove false.”

There within lies the reasons why many tests fail; an incorrect initial hypothesis. From numerous tests, we’ve found that the hypothesis creation has a major impact on the way a test is run, what is tested, how long a test runs and just as important, who’s being tested?

When you don’t apply a discipline of looking at your testing efforts strategically, identify specific goals that they will serve towards (improving user experience, increase form completion, reduce exit rates, etc.) and think about the expected learning and actions outcomes prior to the test, you would be increasing your chances for failure.

Unfortunately, GIGO really applies here – Garbage in Garbage Out!

From years of testing, we’ve seen many people rush into a test due to third party influences. For instance, a Senior Manager believes adding a cat above the Call To Action (CTA) will increase conversions by 8%.

Or your observations show that of your visitors who use site search 80% convert, therefore let’s make the site search the only option on the home page! Sure, your website is about pet products, but will this cat drive an increase in conversions? Should dog people see the cat? Is a visitor at 3am going to be distracted by the cat? Yes, visitors that use site search convert higher, but are your internal teams solely using the site search due to the inability to find things organically within the site?

Many times the hypothesis created can be a detriment to understanding what is really happening on your website. That’s why it’s super important to understand the root cause and solve it rather than go for band-aid solutions to the problem. Good thinking, good strategy, good planning will result in good results, good learning, good actions. Bad planning, bad strategy equal a recipe for failure.

About the Authors

Eric Brandt is Internet Marketing Manager for TSIA.com, a for profit membership based organization serving technology companies. Prior to this, he was Director of Data Science for Internet Marketing Inc. He has managed many successful Organic Search Engine Optimization, Paid Search, Social Media and Online Reputation Management campaigns. His diverse experience and specialization in the integration of online marketing with traditional marketing efforts, website usability and analytics analysis were critical to the success of Internet Marketing Inc.

Eric graduated with an International Business degree with a concentration in Marketing and a French Minor from James Madison University. Right after graduation, he worked for a Washington, DC think tank, servicing large accounts including Pepsi Bottling, Federal Deposit Insurance Corporation, American Express Company, Citigroup Inc. and the New York Stock Exchange.

In his free time, Eric is an adrenaline junkie who enjoys bungee jumping, skydiving and various thrill-seeking adventures. He is also a budding entrepreneur/inventor working on personal online and offline projects. Eric enjoys traveling and foreign languages.

——————————————

Nazli has over 7 years of digital marketing experience focusing on site optimization, behavioral targeting and web analytics. In her previous role she had identified Adobe Test&Target as a solution to improve site conversion rates and leveraged this solution to support higher lead goals. Her work with T&T not only contributed to 3 times higher improvement in conversion rates but also got recognized with “Best Lead Generation Test” and “Best Segmentation Test” awards by WhichTestWon.com.

In March 2010, Nazli received an invitation to join the Omniture Summit where she presented at the “Backstage with Test&Target ROI Rock Stars” session. In 2011, she also presented at eMetrics Marketing Optimization Summit and on “WhichTestWon.com” on some of the best practices in testing. She also now serves as a member of the WhichTestWon.com Leadership Advisory Board.