Selecting Winning Variations For A/B Tests


Picking the best variations for a test is both an art and a science. It requires you to analyze data and predict which variations are likely to perform best. This research could involve both theoretical psychological models and inquiries to actual visitors or people from the target audience.

In the end it’s all about having the smallest set of variations (in order to minimize testing runtime) with the highest potential for learning and optimization.

Theoretical research

The work of others can be a great help in deciding which areas of your website to test, and which variations to try.

Psychological theory

First, we’ll discuss psychological theories. These theories are often the results of fundamental or applied psychological research and used to be found exclusively scientific journals and books. However, with the ever-increasing popularity of the internet, this type of research is more accessible than ever for everyone. Great sources of applied psychological theories online are the Zurb triggers and the Wheel of Persuasion.

Articles and blogs

A second source of theoretical resources is articles and blogs. Some of my personal favorite blogs on conversion optimization are ConversionXL, Unbounce and Optimizely. Each of these blogs offers research-based articles on a broad spectrum of website optimization strategies and tactics. They also link out a lot to other websites, which makes them great starting points for further exploration on the topic of website optimization.

Involving people

After you’ve done your research on a given topic you want to test, you might want to involve real people to find the best variations.

Usability testing

Traditional usability testing involved recruiting test participants, getting them to a designated location, and recording and scoring their behavior. Luckily, the internet has tremendously simplified this process. Sites such as UsabilityHub and Feedback Army allow you to have real visitors take a look at your site or mockup and provide feedback. Often these visitors will provide valuable insights into which variations might or might not work in a live environment.


Another way to gather insights from your users is by asking them questions directly. Tools such as Qualaroo and HotJar allow you to unobtrusively ask questions from visitors of your website. By asking actual visitors that are on your website already, you can gather data directly from your target audience.

Size of changes

Getting the ‘size’ of changes you make in your variations right can be difficult. Of course you want the change to be big enough to be able to measure a change in behavior from your visitors. On the other hand you also want to be able to pinpoint which aspect of the variation has changed the visitors’ behavior (in order to extrapolate your results). While sometimes a change as small as modifying one word can show significant and relevant effects, at other times you’ll need to make bigger, bolder changes to see a change in behavior.

Dutch translation: Winnende Variaties Selecteren Voor A/B Tests

Theo van der Zee

Author: Theo van der Zee

He is the founder of ConversionReview. He has been building and optimizing websites for 15+ years now, and doing so with great success.

On top of his digital skills, Theo is also a trained psychologist and frequent speaker at events around the world.