Quick Fix: The A/B Test

a b testEver heard of an A/B test?

What it is is testing one thing against another. Variable versus control – just like in science. It’s a trusty old tool in the hands of the rainmaker.

The way it works is, you have your control – the thing you know, the one that’s always consistent – say, the same banner advertisement you’ve been using on your site for who knows how long, or your old website design layout, or offline maybe it’s the ad you’ve been running in print for ages. Then you have your variable – that new, unpredictable thing – the new banner ad, the redone website design, the fresh print ad you’ve just begun running.

And in an A/B test, those two go head to head.

What’s the value in that? Numbers. In using an A/B test, you know exactly, by the numbers, which one was more effective – the control (the original) or the variable (the new and unknown version).

Why’s that good? Well, if you’re testing, say, an advertisement on your website that visitors click on and it takes them to a sales page, and you found you were getting 50% more clicks from one of those advertisements than you were from another… well, wouldn’t that be something you’d want to know? Couldn’t that make, potentially, a huge difference in how much business you’re doing?

Sample Size: The Secret of the Split Test

The A/B test – also called the “split test,” because you’re splitting your testing between two items, the “A” and the “B” of the A/B test – is used to get as close to random, objective data as you can achieve.

Split testing isn’t something that became big until the Internet age. It simply wasn’t possible, or practical, to do testing like this at anything approximating a large scale. Before that, it was mostly guesswork – you felt like you got more electronics sales on the days when you were wearing a red shirt, so you started wearing red more often. And maybe red made an impact, or maybe the sample size was just too small.

Thing is, sample size is king when it comes to testing. Don’t believe it? Check this out:

a b test

That’s an actual graph of my results from a 6 month long split test I did on a sales page for a business of mine. I was testing two totally different long-form sales pages – see the results for yourself.

At first, the original version outperformed the variable. I was surprised by this, because I thought the variable was better and more tightly written and designed. But the original version had a video of me speaking personally to the buyer, and at the time a lot of my buyers were people I’d built a relationship with through the website. My guess was, as traffic to the site increased and I got more and more buyers, the more professional sales page would prevail.

And, despite that big early lead in conversions to sales (as evidenced by the big blue spike up there – that’s the original version outperforming the variable), with time the variable eventually caught up and surpassed the original sales page’s numbers.

Here’s the rub: if we quit testing after one month, we would’ve gone with the original sales page.

Even though, in the end, it was the newly designed page that performed better.

That’s what you need to be very, very aware of with A/B testing: sample size is king. The more data points you have to make your decision on, the better that decision will be.

In split testing, this is called “confidence” – basically, the better your data, the more confidence you have. And with big decisions, you’re going to want a lot of confidence.

Why Should You Use an A/B Test?

I’ll use an online product sales page to start with, because that’s the easiest example to give, but then we can move onto some other examples so you know what else you can use A/B testing with.

Let’s say you have a sales page where you sell a video learning program on trading stocks for $140. And let’s say your current sales page is converting at 1%.

That is, 1% of all buyers who land on your page buy your video learning program. So for every 100 people who go there, 1 one of them buys.

Now let’s say you build another page that converted at 1.3%. Not very exciting, right? 1.3% vs. 1% – not exactly something you write home to Mom and Pop about.

But it should be. Want to know why? Here’s why:

If you’re getting 2000 people a month to hit that sales page, that means with your original sales page you’re doing $2800 / month in revenue. But with the new sales page, at 1.3%, you’re suddenly and instantly doing $3640 / month in revenue. Is that worth being excited about?

And the bigger your traffic numbers get, the more exciting the difference becomes.

Split tests are a way to add a multiplier to your revenue generation. They let you take everyone who’s coming in, and get more money per set of eyes on the screen than you’d otherwise have.

What Can You Use A/B Testing To Do?

So what can you actually use A/B tests to collect data on? Anything that might impact your conversion, essentially. And conversion can mean:

  • Who buys a product
  • What product they buy
  • If they sign up for your newsletter
  • If they click on another link on your site
  • If the click on an ad and go to a sales page

So you might measure how many clicks this advertisement gets versus another. Here’s another statistic of mine from the same site:

a b test

What that one tells us is there’s essentially a dead heat between the Original Ad and Ad #2, and that Ad #1 is lagging somewhat behind in performance. So if we want to make a cut somewhere, we cut Ad #1 and overall clicks go up.

You can also use an A/B test to see which website layout keeps users on the site longer, which one converts the greatest proportion of them to buyers, or which one generates the most clicks.

You can use an A/B test to find out which offer is the most attractive to your users – if you have an offer that reads “Download PDF Now” does that outperform or under perform “Download Free Gift Now”?

You can even use A/B testing in print ads – for instance, if each advertisement sends readers to visit a different website or page on a website. It gets a bit more complicated with print – you probably want to put the same ad in multiple places all going to different pages first, just to get an idea of how much traffic each channel usually gives you. Then you start playing around with variables.

But split testing is incredibly useful and valuable for finding out more about your audience, your website, your products, and your pitch.

And it’s incredibly useful for making your business work better.

Tools You’ll Love

It used to be that to do an A/B test, you had to know HTML, because you had to do some coding on your pages and your site to make it work.

These days, it’s a breeze. Using the following split testing tools and resources you can become a master split tester in almost no time flat:

Google Website Optimizer

Makes A/B testing a snap. You go in, set up an experiment, tell Google what your three pages of interest are:

  1. Your control / original
  2. Your variable / new
  3. Your conversion page / where people end up when they click

Then, Google gives you some code, you copy it and paste it into the header of the page, and then… you’re done. The experiment starts running, and Google does all the work, sending your visitors to this page or that page and tracking the results.

How well does Website Optimizer work? It occasionally misses conversions here and there, but it’s one of the more reliable systems out there. For the price (free) and the usability (it’s a piece of cake), you can’t beat this one.



Split Tester

This one’s really handy for doing some fast math on whether your sample size is big enough and you have enough conversions to draw conclusions from.

Statistics are tricky because you always want to think the answers in front of you are definitive when they aren’t. The only way you can know if you’ve got concrete, actionable numbers is by probability, and that’s where Split Tester comes in.

All you do with this one is put in your numbers, and Split Tester tells you how confident you can be those numbers were correct, based on number of clicks versus number of impressions (the function knows the number of impressions by taking the number of clicks divided by the click through rate or CTR).

Here are my results using that sales page test I showed you earlier:

a b test

Guess I ought to go run that test another 6 months or so.



Visual Website Optimizer

I haven’t moved to this one yet, but as I understand it Visual Website Optimizer (VWO) is the heavy artillery in the split testing arsenal.

Every review I read on VWO is shining. People love it, and it gets results.

Here’s a case study on an established ecommerce site that bumped sales up 20% using VWO to redesign its homepage:

The Business Case for A/B Testing

(note: the individual posting this case is the CEO of VWO, so he isn’t an unrelated party, though I don’t think it matters much as he’s just presenting a case study)



Test Results In: A/B Testing Puts Businesses on Rocket Fuel

If you have an established business and you want to start boosting and refining its returns and conversions, you need A/B testing.

And if you have a new business and you’re still just trying to figure it all out, you also need A/B testing.

Heck, no matter where you are, if you have a website – and almost everybody does these days – you need to get started doing split testing if you’re not already.

It’s not just for immediate returns – testing is a medium-term strategy. It doesn’t produce overnight results, unless you already have large enough traffic that you can quickly get large numbers of data points.

Testing’s most useful function is this: it takes what you already have and makes it work better.

  • You don’t have to go build a new sales channel
  • You don’t have to go build new marketing channels
  • You don’t have to launch a new product line
  • You don’t have to revolutionize your site
  • You don’t have to pay outsources exorbitant rates
  • You don’t even have to come up with any brilliant new ideas at all

All you’ve got to do is be able to tweak and refine and try new things, and then set up the test and let it just run. And then it takes what you’ve already got – and it tells you how to make it better.

Nothing quite like it.

And if you’re ready to start split testing and really start switching your business into overdrive, you definitely don’t want to miss all the great tips, tools, techniques, and insights on offer in my newsletter – sign up now to start getting it delivered to you today:

Get Industry-Leading Tips Straight to Your Inbox

See you next time, same time, same place.

Chase Dumont


Related posts:

This entry was posted in Quick Fix. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

CommentLuv badge