How we make split testing on low-traffic sites worth your time
When people are thinking about making a change, whether it’s a new platform, a new page or a new headline, someone always throws out the same comment: “Great idea. Just A/B test it.”
That assumes a lot, mostly that you’ve got the traffic.
“Just A/B test it” needs a lot of visits
To get true statistical significance, you need thousands of visits per variation.
If the change isn’t very big, like going from a 3% to a 3.5% conversion rate, you need about 20,000 visits per variation. If you’re doubling your conversion rate, you still need about 1,000 per variation.
With a fairly small change and not much traffic, your test could need to run for years. That obviously isn’t a good use of your time, and so much changes in a few years that the result wouldn’t mean much by the end anyway.
Sub-goals give you data from day one
AB Split Test has worked with thousands of agencies and small business websites over the years. What we’ve realised is that the primary goal is important, and it’s the one you really want to optimise. But well-placed sub-goals get you way more data from day one.
Scrolling, clicking, visiting pages, adding to cart, or just spending more time on the site are all signals, and they add up to something quite useful. AB Split Test automatically adds sensible sub-goals to every test, so you get these insights without setting anything up.
Audiences can cancel each other out
Splitting your traffic into audiences matters too: mobile, desktop, people coming from ChatGPT, all that sort of thing. People react in opposite ways, so sometimes one group cancels the other out and the test looks flat.
Look at audiences separately and you’ll sometimes see huge swings. So every split test automatically gets these audiences too, and we look at the results for you to pull out the insights. You don’t need to dig through all of it yourself.
Go big or don’t test at all
Everyone talks about changing a button colour. On a small or medium website, it’s just not going to make a difference.
You need to test big, bold changes, like a new offer, a new page structure or a wildly different headline angle. Moving a full stop, or swapping it for an exclamation mark or a question mark, isn’t going to do it.
No change is a result
You’re rolling out new features, new contact forms, different pricing pages. Maybe you want to try out Astro or whatever.
Often there won’t be a big change. The figure people usually quote is that only around 1 in 5 split tests show a visible uplift, even on huge sites.
A flat test still tells you what you’re doing isn’t tanking your conversions or upsetting your customers. With enough sub-goals in place, you’ll see people are still browsing and clicking around your site like they were before.
After a little while, AB Split Test will tell you there’s no huge effect either way. Then you can go with whichever version you prefer.
Keep the learnings from every test
Keep all the information from those no-change tests. We build up learnings as we go, so each test and each improvement we suggest is based on more data. That’s built in.
Say you tried a really casual headline against a formal one, and the casual one didn’t increase engagement. That’s probably worth not trying again. Try a different formal one next time.
CRO is a feedback machine
Like I said, even a test that doesn’t reach significance teaches you something about how your visitors behave and what they respond to.
Every site update becomes a check-in with your customers. If you’re an agency, it shows your clients you’re making changes based on data. That builds trust, and it shows your interests are aligned, because what you both care about is their revenue going up.
What you learn from a proper CRO process also goes way beyond the website. It feeds into your messaging, your pricing, which features you build next, and how you handle sales conversations.
Built for starting at zero
Low traffic doesn’t make testing pointless. It just means you test differently.
Most other services aren’t focused on this. We’re focused on marketing agencies, and marketing agencies take people from zero to millions. You need to be able to start at zero to get to millions.