An important aspect of conversion optimization testing is deciding on which type of experiment you’re going to run. This basically boils down to two different options, A/B testing or Multi Variate Testing.
A/B
A/B testing is the most popular type of conversion optimization. It also happens to be the least complex form of experiment you can run on your website or app.
How does it work?
With A/B testing, you’re testing two (or more) different variations of a website or app against each other. Using testing software such as Optimizely or VWO you split your traffic in two equal groups. Half of the traffic will be shown variation A of your website and the other half will see variation B. The testing software will then log the actions that visitors in each of these variations will take (such as signing up to your newsletter or purchasing your products). After the experiment is completed, you can analyze the results to decide with variation performed best and have your IT team show the winner to all your visitors from that point on.
Sometimes aimed at improving, not learning
If you want to learn something from an A/B experiment, you can only change on factor at a time. For example, you can only change the headline text and not its size; only change the CTA button’s color and not the text on it, etc. Because if you were to change multiple factors at the same time, you can’t ascribe any potential gains of losses to a specific change in your website.
A/B/n testing
Most people who think about A/B testing assume there can be only two variations in the experiment. While I must admit the name is extremely misleading, there is not reason why there can’t be more than two variations in an A/B experiment. As long as you’re comparing only one factor per page, A/B testing will allow you to test as many variations as you like in an experiment. Do keep in mind that the more variations you create, the longer the experiment will need to run in order to reach proper levels of statistical significance and power.
A/A testing
If you want to verify if your A/B testing tool is working properly, you could run a so-called A/A test. With an A/A test you’ll be testing two identical versions of your website or app. Because these variations are identical, with enough time and conversions (be aware of repeated significance testing) you should notice that the variations will perform identically. If they aren’t performing identically after enough time has passed, you might want to contact your testing tool provider and ask them for a clarification.
MVT
The abbreviation MVT refers to Multi Variate Test. Multi variate testing is the more complex brother of A/B testing, and is commonly used by more advanced conversion optimizers.
How does it work?
With multi variate testing, you’re testing two (or more) different variations of a website or app against each other. Notice how this is exactly the same describing as I’ve used with A/B testing. The difference with multi variate testing however is that you can test multiple factors on a page. For example, you can test both the length (long or short) and the placement (left or right) of a descriptive text. The testing software will use this input to generate four variations for you, namely:
- long x left
- long x right
- short x left
- short x right
Can be aimed at improving and learning
With multi variate tests you’ll be able to single out which factor caused the change in behavior of your visitors that led to an increase or decrease in your conversion rate. The testing software will enable you to see whether it was the longer text, the placement on the right, or perhaps the interaction effect of both switching to a longer text and placing it on the right. With this added benefit of being able to learn more also comes the serious downside of needing more variations per experiment. This in turn will require quite a larger amount of traffic to your website in order to reach a satisfactory level of statistical significance and power for each of your variations.
Dutch translation: A/B of MVT: Wanneer Kies je Welk Type Test?
He is the founder of ConversionReview. He has been building and optimizing websites for 15+ years now, and doing so with great success.
On top of his digital skills, Theo is also a trained psychologist and frequent speaker at events around the world.