Planning Conversion Optimization Experiments


Every experiment (from a simple 2-variation A/B test to a complex multi-page MVT) starts with planning. In this article I suggest you should pay attention to the following aspects while making your testing plan.

First of all, clearly define the objectives of your tests, prioritize them, and document your learnings properly. Make sure to interpret your results correctly by having precise know-how of the tools you’re using, and keep in mind odd variations in traffic source types. Last but not least, don’t forget to include the activities of different departments of the organization in your plan (management, marketing and IT).


As stated in the introduction, you should start planning your conversion optimization experiments with the formulation of objectives, or goals of your test. What are you trying to achieve with this particular test? And how might that aid you in learning about your overall conversion? Are you trying to learn about the behavior of your visitors, or merely want to increase profit? In general we see that the more you want to learn about your visitors behavior (as opposed to increasing profit solely) the smaller the differentiations are you should be testing.

Winning or learning

winning When you’re testing a full redesign of your checkout funnel as an A/B test, the only real conclusion that you’ll be able to draw is whether or not the redesign performs better than the control. If however, the redesign gets split up into several A/B or Multi Variate tests, you can start drawing more fine-grained conclusions based on the results. On the other hand, such smaller tests often take more time to set up, more runtime to complete the experiment, and therefore less fast results (and potential gains) for the company.


After having defined you objectives, you should start prioritizing your tests. After all, you can only run a limited number of tests simultaneously. You might wonder how to prioritize. There exist several frameworks that attempt to assist you in this. One well-known framework is the PIE framework by WiderFunnel. In this framework you can determine a PIE (Potential, Importance and Ease) score for each of your tests.

  • Potential: How much improvement can be made on the pages?
  • Importance: How valuable is the traffic to the pages?
  • Ease: How complicated will the test be to implement on the page or template?


In order to keep track of your experiments it’s important to document both your testing roadmap and the results that come from your tests. While at first the process of documenting this knowledge might seem time-consuming or bureaucratic, you’ll soon learn that the work you put into it will pay out multifold later on.

Testing roadmap

By setting up a testing roadmap such as described in this article A/B testing in 2 hours you’ll be able to structure your conversion optimization strategy. By keeping this testing roadmap (which contains items such as screenshots, hypotheses and dependencies) up-to-date you’ll always know what action you should take next. The testing roadmap should contain every test, from the first results that you have acquired, to the latest idea mentioned by a colleague during a brainstorm.


After having completed the preceding three steps, you should ensure that you have proper know-how of the available testing tools. For example, it is useful to know if it is possible to test proposed changes to a webpage with the testing tool that you’re using. And do you know how long the test will run? If the test performs as expected, will you be able to draw results with enough significance, power and relevance? By using tools such as the A/B test calculator and Test duration calculator you’ll be able to tackle such questions.


Aside from knowing what website optimization tools to use and how to use them, you should also keep an eye on odd variations in traffic source types. Under normal circumstances, you’ll get a similar composition of different types of traffic sources on your website or app. In other words, you’ll roughly get the same percentage of visitors on your website through a link, as direct traffic, by organic search queries, via PPC ads, etc.

Unbalanced traffic mix

Hence, a temporal imbalance in the composition of traffic sources is no cause for concern, because all traffic is equally distributed over your variations. Therefore, each of the variations will receive a similar chunk of the imbalanced traffic mix. However, when one of your variations somehow interacts with a certain segment of traffic, having an imbalanced traffic mix could cause problems. For instance, if one of your variations promotes the superior technical qualities of your product and one of your blogs goes viral on several technology-related websites. The amount of technologically minded visitors would therefore increase sharply, which might skew the results of your experiment.

Include other departments

Last but not least, don’t forget to include the activities of different departments of the organization in your plan. The most important departments to include in your plans are marketing, IT and management.


Conversion testing during promotions provides you with a great opportunity to learn more about your visitors. However, you got to make sure you’re planning ahead when you’re testing during a promotion. By constantly coordinating your tests with other departments in your organization, you’ll make sure to stay on top of all marketing related activities. This will enable you to keep your tests in sync with the release of new collections, product launches, policy changes, promotions, etc.

Corporate visual identity

The webpage or app you’re testing is most liked designed in accordance with the corporate visual identity set by the organization. So when you’re developing testing variations, make sure you attune your ideas with the design department at some point during this process.


Another crucial department in the organization that you should keep in touch with regularly is IT. Their releases can break your tests, and vice versa. It’s therefore vital for both parties to have activities in sync. What you could do to keep updated is to subscribe to their project management feed, obtain the release calendar and try to set up regular meetings with IT staff.


By keeping yourself in the loop with IT, you will also be aware of any planned downtime due to feature releases or software security updates. Unfortunately not all downtime is planned. Therefore, if your organization isn’t using a downtime monitor already, I’d suggest you to look into one right away. Services like Uptime robot and Pingdom monitor the website realtime and in case of any downtime will send you a warning immediately. This information allows you to factor in any unforeseen downtime when you’re analyzing the results of your experiments.


manager While staying in touch with the IT and marketing department are important I’d suggest also keep in good graces with management, because ultimately they will decide whether or not your efforts will be financed. If approval from management is absent, your job as a conversion optimizer will become a lot more difficult. Sometimes it’s necessary to start your testing adventures off the radar far away from upper-management, because they haven’t been convinced yet of the importance of a data-driven testing culture in their organization. However, don’t make this your rule of thumb. In order to ensure you’re able to take your testing efforts to a higher level in the future, you should strive to get support from the highest level of management you can possibly get.

Theo van der Zee

Author: Theo van der Zee

He is the founder of ConversionReview. He has been building and optimizing websites for 15+ years now, and doing so with great success.

On top of his digital skills, Theo is also a trained psychologist and frequent speaker at events around the world.