Validating Experiments Before Running Them


Most people start optimizing their website by testing the first thing that comes to mind. While this process seems like an efficient way of running tests, it isn’t an effective solution. In order to improve this process, you should make sure that tests are validated and founded in qualitative and quantitative research.

Not only does this verify that what you’re testing is an actual hurdle for users, it also raises the chances of finding a winning variation! As a bonus, by testing validated problems, you’ll always be learning (because all tests are relevant).

Quantitative research

Any research that is based on large numbers is said to be quantitative. This could involve both statistical analysis of surveys, database queries, or extracting findings from web-analytics data and the like.

Web analytics data

Most companies these days are using some kind of software to monitor the visitors or their website. By far the most common tool to do this is called Google Analytics. By analyzing the data that you’ve gathered, you can validate whether or not a proposed test is likely to gather valuable insights or fix critical problems on the website. For example, you might want to have a look at the Page Value, Bounce rate and Exit rate of a page that is being tested. Alternatively, for checkout funnel pages, you could check if the page for which a test is proposed has a large dropout rate. Perhaps you will notice other pages that are in far more need of improvement!

Customer data

customer data Another potentially valuable source is the customer data that you already have. This data could include information on what types of people are purchasing your products or services. For example, it allows you to see if your product is purchased more by men or by women, what age groups are currently buying from you, their demographic locations, etc. By having a hard look at your customer data, you should be able to see if a test that is targeted at female visitors will be effective on a part of the website where mostly male visitors are shopping.

Qualitative research

This type of research is mostly focused on smaller numbers. Compared to the (mostly anonymous) data studied in quantitative research, it’s also far more personal.

Expert heuristic analysis of user scenarios

This involves getting an expert in usability, user experience, user interface, etc. involved in your project and request a website review from them. Together with this expert, your team will scrutinize your website. By looking at common user scenarios (paths a user takes through the website in order to achieve their goals) you can see roadblocks that are blocking them from converting. These roadblocks could range from incomprehensible forms to unclear call-to-action buttons, to broken navigation links, etc.

User testing

Another form of qualitative testing is user testing. In this form of testing real visitors are being monitored. Services such as allow you to see what your visitors are doing (and thinking!) while surfing on your website. Aside from this technique you could also employ methods that monitor the mouse movement and clicks of your visitors. This could reveal patterns in their behavior and unearth common barriers that the encounter on their path to conversion.

Dutch translation: Experimenten Valideren Voordat je ze Draait

Theo van der Zee

Author: Theo van der Zee

He is the founder of ConversionReview. He has been building and optimizing websites for 15+ years now, and doing so with great success.

On top of his digital skills, Theo is also a trained psychologist and frequent speaker at events around the world.