Not all visitors are created equal; therefore you shouldn’t pretend they are in your tests. To optimize your testing plan, you should recognize that different visitors to your website have different characteristics and different goals.
Before or after the test
The first thing to ask yourself is if you want to segment your traffic before or after you run your test. Both have their own upsides and downsides.
Segmenting before your test
Most popular testing tools will allow you to segment your traffic before your test. By selecting the characteristics of the visitors that you want to target, you can only show your experiments to specific visitors.
Upsides and downsides
The major upside of this is that it will allow you to run more simultaneous tests because the tests aren’t shown to visitors for which they aren’t relevant to. On the other hand, if you have made a mistake in deciding the correct segment upfront, you might be missing out on valuable data afterwards. Because you have decided not to show your test to these visitors, this data can’t be retrieved afterwards either.
Segmenting after your test
Similar to segmenting before your test, you can also segment your traffic after your test is completed. One of the most important things to keep in mind here is that segmenting your traffic means fewer conversions per variation. This in turn means lower power and less chance to have reached statistical significance for your individual goals. So, when segmenting your traffic after your test, make sure the segment in case has enough conversions per variation.
Upsides and downsides
The best thing of segmenting after your test has completed is that all the data is available. By comparing data between, for instance, new visitors and returning visitors you’ll be able to find discrepancies between them. On the other hand, you should be cautious not to ‘slice and dice until you reach results’. Meaning that if you segment you data in enough different ways, somewhere you’ll find a significant result. This however doesn’t mean that this is also a relevant result worthy of investing further resources into.
The most commonly used way of segmenting your visitors is by the technology they are using to visit your website or app. Refer to the manual of your testing tool to find out how to segment your tests based on these characteristics.
- Browser: Google Chrome, Firefox, Safari, Internet Explorer
- Operating system: Windows, Mac OS, Linux
- Device: Desktop, tablet, mobile
- Source: Search, referral, direct
- Location: Country, state, city
Apart from the technological segments defined above, you can also segment users based on what you know about them.
- Account: Logged In vs. Guest Users
- Frequency: Repeat Shoppers vs. First-Time Visitors
- Status: VIP, regular, budget
- Last purchase: Never, this year, this month, this week
- Tier: Percentile of spending
Dutch translation: Bezoekers Segmenteren Voor Betere Test Resultaten
He is the founder of ConversionReview. He has been building and optimizing websites for 15+ years now, and doing so with great success.
On top of his digital skills, Theo is also a trained psychologist and frequent speaker at events around the world.