Writing a Good A/B Testing Hypothesis

Hypothesis

A sound hypothesis is what makes or breaks any conversion optimization experiment. Without having a relevant, unambiguous, measurable and specific hypothesis to start off with, you’re basically groping in the dark with your experiment.

Also, in order to run the best experiments, you should ensure the hypotheses you’re using are founded on psychological theory, web-analytics data, qualitative research and learnings from previous experiments.

Basic structure

Every good hypothesis is structured like the following: “If {I did this}, then {this will happen}”. Below you can find a couple of examples to clarify this basic structure:

  • If I add a testimonial to the product page, then visitors will feel reassured and buy more widgets.
  • If I include our speedy delivery in the value proposition, then visitors will see this benefit and the bounce rate lowers.
  • If I change the CTA text to ‘Add to cart’, then visitors will be less anxious and add more items to their cart.

Relevant

Before you decide whether your hypothesis is scientifically correct, you should first have another look at the relevance of what you’re going to test. Knowing “Which image visitors like best” or “What the best headline is for the About Us page” without knowing what KPIs these metrics contribute to, is pretty much equal in relevance compared to not run any tests at all.

Make it relevant

improvement Say one of the KPIs (Key Performance Indicators) of your website is the number of items that you’ve sold, and another is the number of five star ratings that you receive. With these KPIs in mind, you should have a clearer idea of what you should be testing. Without knowledge of your KPIs, the results you are getting from tests might be interesting, but wouldn’t allow for structured improvement towards your goals.

Unambiguous

Also be careful not to create ambiguous hypothesis such as: “If I add a USP, then visitors will buy more items”. The problem with this hypothesis is that you won’t be able to find out what USP (Unique Selling Point) works best. Is it one mentioning your free returns, your response time, etc.? Perhaps the test results will even show that one USP shows a significant increase in conversions, while another USP shows a significant decrease in conversions. Because it hasn’t been specified which USP will be tested, one cannot properly confirm or reject such a hypothesis.

Make it unambiguous

By adding which USP is added, you will clarify the hypothesis. For example if you were to use the following hypothesis: “If I add ‘Free shipping’ as a USP, then visitors will buy more items” it would quickly become unambiguous which USP will be used in the experiment.

Measurable

Furthermore you should make sure the change in behavior you’re predicting can be measured with the tools at hand. For example, consider the hypothesis: “If I add ‘Free shipping’ as a USP, then visitors will be happier”. Not even regarding the fact that this hypothesis isn’t necessarily relevant, it probably isn’t measurable well either.

Make it measurable

measurable By expanding the hypothesis to “If I add ‘Free shipping’ as a USP, visitors will be happier and will view more pages per visit” you’ve created a measurable hypothesis. Do keep in mind that even though this hypothesis is now measurable, you should still consider whether or not ‘view more pages per visit’ is relevant in achieving the goals set for your website.

Specific

Lastly you should verify if the hypotheses that you write are specific, but not overly specific. For example a hypothesis such as the following sounds great at first glance: “If I add ‘Free shipping’ as a USP, then 5% more people will buy my widgets”. But what happens when the test results come in and it appears that only 3% more people bought your widgets? Strictly speaking, you should reject this hypothesis, even though the results were pretty much as you had predicted them to be.

Make it specific

By changing the hypothesis to “If I add ‘Free shipping’ as a USP, then more people will buy my widgets” or “If I add ‘Free shipping’ as a USP, then at least 5% more people will buy my widgets” it will become unambiguous, measurable and specific.


Dutch translation: Een Goede A/B Testing Hypothese Opstellen


Theo van der Zee

Author: Theo van der Zee

He is the founder of ConversionReview. He has been building and optimizing websites for 15+ years now, and doing so with great success.

On top of his digital skills, Theo is also a trained psychologist and frequent speaker at events around the world.

With UserFeedback you can

Find Out What Your Users Want

UserFeedback logo
  • Decrease your bounce rates
  • Improve user experience
  • Get lower ad costs
Learn more