Would you like your tests to run in less time with equally reliable results? How about being able to benefit more from the holiday season? You can do this both by profiting from the natural ebb and flow in traffic volume caused by holidays and (offline) promotions.
Instead of stopping your tests during these times, you should continue running them. Better still, prepare for the traffic spikes in advance, and set up specific tests for each holiday, event, promotion or campaign!
Why test during holidays or promotions?
One of the most important factors that determines the estimated duration of an A/B test is the daily amount of visitors. By running your tests during holidays and promotions you’ll be ensured of plenty of traffic. You can use this traffic to shorten the duration of your tests. You could even run additional tests.
Testing on specific holidays
Apart from using the extra traffic to run more tests, you might also consider running tests during a specific holiday. For example, does your website convert better when you fit it in a red-and-green theme around X-mas? Or perhaps a jolly Santa in the top-right corner would persuade your visitors to convert more? While results of these tests might or might not apply to other holidays (more on this later), they will probably apply to the same holiday next year. Just to be sure, you could repeat the test for a small percentage of traffic in the next year to confirm the results are in fact still the same.
Predicting high traffic volumes
Some traffic spikes (such as those caused by viral content hits) are often impossible to predict. However, other events such as yearly returning holidays can be predicted well in advance. Try using the following techniques to predict time periods in which high traffic and sales volumes are expected. Knowing well in advance when high traffic is expected will allow you to set up your experiments and make sure they work flawlessly while in execution.
Holidays and events
Holidays and events refer not only to (inter)national holidays, but also specific events such as Black Friday and Cyber Monday. Apart from these generic holidays and events, you should also keep an eye on industry related happenings. Examples that come to mind are the Super Bowl or Eurovision Song festival.
Marketing calendar
The marketing department of your company will be able to provide you with this information. By using marketing data you’ll know when large marketing campaigns of your company will be launched, and you will be able to predict traffic spikes with pin-point accuracy.
Previous years
By analyzing existing data such as the demographics of your long term visitors and sales data in your web analytics tool, you should be able to see cyclical spikes in traffic. While most of these spikes are probably linked to either a holiday or a promotion, some spikes in traffic might be unexpected. Apart from running extra conversion experiments during this time, you might also consider setting up additional promotions targeted at whatever event is causing the traffic spikes.
How to minimize the risk
During these high-traffic times there’s a lot at stake. How can you minimize the risk of visitors seeing a malfunctioning website or that your website sees an incremental decrease in the number of conversions?
Double quality assurance
Of course you should always be doing proper quality assurance for your tests. However, with the risks involved in conversion testing during the holiday season, I’d advise that you go for another round of quality insurance. Be sure to check your variation on as many devices and browsers as possible, get multiple people to vet the variations, and so forth.
Coordinate across departments
Also be sure to coordinate your tests with other departments in your organization. You wouldn’t want a specific campaign to go live that contradicts any modifications made in a specific variation.
Spread out the experiments
By making sure to spread experiments you run over time across the different types of pages on your website, you’ll hit two birds with one stone. Firstly you’ll ensure that the number of tests you can run independently from each other is maximized. After all, multiple experiments running on the same page should in most cases be conducted as a MultiVariate Test (MVT). Secondly you can spread the risk involved in running these tests. Because not all visitors hit every type of page, some might not be bothered by a broken or underperforming variation in an experiment. Make sure not to run experiments that might interact with each other (i.e. contradictory messages on banners).
Start small
Try to run each experiment on a subset of the available traffic at first. Assuming you’ve got plenty of traffic you could try running the experiment on 10% of the available traffic. After at least ~100 conversions in each variation you can then decide to upscale the traffic volume. A particularly large difference in the conversion rate between variations (or in segments such as mobile or browsers) could then warrant more pre-testing.
Don’t stop too early
Be wary not to stop the tests too early though, because this might lead to an increase of Type 1 and Type 2 errors. Also, while it’s technically possible to alter a running experiment (such as aborting a low performing variation), this practice is advised against.
Empower others
Another way to minimize the risk is by empowering other members or your organization. For instance, you could grant a junior conversion specialist access to your testing tool. He/she would only be able to pause an experiment. If something goes wrong with an experiment and for whatever reason you’re unable to pause or stop it (for example you’re traveling overseas or got ill) then your colleague can pause the test.
Monitor closely
Closely monitor your conversions across various segments on your website Make sure you actively monitor both the results of your tests and your website in general. Try building custom dashboards based on your web analytics information to closely monitor your conversions across various segments on your website. This will allow you to spot problems as quickly as possible and allow you to act upon them.
Conclusion
During holidays both the stakes and risks of conversion tests are high. By making sure to minimize the risks using the techniques described above, you can harvest the advantages that testing during these merry times can bring.
Dutch translation: A/B Testen Tijdens Feestdagen of Aanbiedingen
He is the founder of ConversionReview. He has been building and optimizing websites for 15+ years now, and doing so with great success.
On top of his digital skills, Theo is also a trained psychologist and frequent speaker at events around the world.