Beginner’s Guide to A/B Testing

There are two main ways to make more money from your website.

The first is to drive more traffic, and the second is to convert more visitors into customers.

A/B testing is a great way to get those additional conversions.

If you want to start A/B-testing your website, that’s great.

It’s an effective method to figure out what changes you need to make to your website to achieve your goal.

Want to improve conversions?

Maybe changing the color scheme, button size, or button placement can impact the results.

A/B testing is the best way to figure this out and continuous A/B testing will make your website more efficient..

But before you start A/B testing, you need to understand how it works. It’s not just a one-time thing.

We’ll explain how you can use this strategy to improve your business and help you boost your conversion rates.

Let’s start with the basics.

How A/B testing works

A/B testing isn’t difficult to understand. You start by creating a hypothesis about a certain element and then run a test to see if your theory was right.

To do this, you create two different versions of your website. Half of your site traffic will get sent to one version, and the other half will get directed to the variation.

Here’s a visual representation to give you a better understanding of what I’m talking about:

Once you set up the test, you wait to see which variation has higher conversion rates. Then you draw conclusions and update your website with the version that converts the most.

What’s tricky about A/B testing is deciding for how long to run tests and choosing what elements to test.

Honestly, there’s not one right answer to those questions. But I’ll give you some guidance to help you run these tests efficiently.

Let’s begin.

Set a clear goal for your testing procedure

Here’s a visual representation of what your procedure should look like:

The first thing you need to do is determine which conversion to improve.

Don’t change every aspect of your website.

That’s an ineffective approach and won’t give you measurable results.

Instead, make an alteration to something specific that’s related to your goal.

If you’re unsure where to start, here are some examples of different components you can change on your website:

Test your CTA buttons

When you’re trying to get higher conversions, it’s best to start with the button that gets people to convert.

Obviously, these buttons are different for each company in every industry. It depends on your goal on a specific landing page as well. Some of you may be trying to drive a sale, while others may be trying to get site visitors to opt in to their email subscription lists.

The key here is picking one element to start. You don’t want to test two theories at once. Testing multiple hypotheses doesn’t give you conclusive results because you won’t know which variation impacted the results.

That’s why you should be running tests continuously to maximize the efficiency.

Start with something subtle and easy such as the size, color, or placement of your CTA button. For example, you could hypothesize that a larger CTA button will have higher conversions.

You can also test the actual text written on your CTA button. Try testing power words against action words.

Here are some of the power words that convert the most:

Are any of these words used in your call to action?

Here are some more statistics about the conversion success of words that prompt an action:

Based on the numbers in these graphics, you could hypothesize that a call to action that says “Sign up for free” will have higher conversion rates than one that says “Download now.”

But there’s only one way to find out. Test it.

After you test one of these and are satisfied with the results, move on to another test. If you started with the size of your CTA button, move on to the color. After that, you can test the text or placement of the button.

Test all the CTAs this way on each page of your website.

Fewer than half of websites have CTA buttons that can be spotted in less than 3 seconds. Testing your CTA will definitely give you an edge over your competitors.

Evaluate your headings

After you’re satisfied with your CTA button analysis, move on to other components of your website that stand out the most.

Your headlines and subheadings definitely jump off the page, so it makes sense to test those next.

If your headlines aren’t worded properly, visitors may not even read all the content on the page. While testing the CTA may seem more important, visitors could miss out on your entire value proposition if the headers don’t keep them engaged.

In addition to conversions, you should be looking at analytics that show for how long each visitor stays on the page.

If one headline produces significantly longer average page viewing time than the variation, it will positively impact conversion rates.

Here’s an example of an A/B test of a website’s headline:

As you can see, the two pages are identical except for the headings. In the variation, the main heading and subheading have different wording.

The test yielded conclusive results. Website visitors who saw version A filled out the form at a 27.76% higher rate than the ones who saw version B.

If your tests are only within 5% or so of each other, you may not be able to say that one is definitely better. To find out for sure, you can run an A/A test before your A/B test to see what your standard deviation is between the same versions of a web page.

Improve your checkout process

For those of you with an ecommerce website, you need to find ways to minimize shopping cart abandonment.

Using A/B tests on the layout of your checkout process can really help you maximize conversions. You’ll be able to tell which elements are working and which ones can be tossed away.

We’ll give you some ideas for the elements you can start testing.

Are you accepting coupon codes? If you have an option for visitors to input a coupon code during the checkout process, it could send them searching for a code.

But if these codes aren’t always readily available and they’re just something you offer to a disgruntled customer, it shouldn’t be a primary element of your checkout process.

Test it out to see what kind of results you get.

You could also test a guest checkout vs. login to checkout conversion rates. We’re assuming you’re offering a secure checkout, so use A/B tests to see if the size and placement of your security badges have an impact on conversions.

Test every element on the page. Something small could make a huge difference.

Here’s a great example of something subtle that iHerb used when A/B testing their checkout process:

Take a look at the left side of the screen. They have a shipping cost calculator. All the customer has to do is input their zip code to get an estimate on their shipping costs.

But the Heavy Duty Bag promotion draws attention away from the calculator.

Here’s another version of the page:

As you can see, the image was removed, and the “Calculate” button was placed much closer to the “Proceed to Checkout” CTA.

OK. So technically they changed two elements on this page, which we said you shouldn’t do. But this was subtle enough to be effective.

We are big advocates for removing clutter from your pages, so we think it was a smart decision to get rid of the image in addition to moving the placement of the calculator.

Find out which images convert

You definitely want to use images to help you improve your website. It’s just a matter of deciding what images to use and where to place them.

If you’re on the fence about a decision, or you have some images already on your website, run A/B tests to see which ones have the highest conversion rates.

For example, you could test an image of a man vs. a woman to see if they yield different results. Or you could test an image of the same model but with different facial expressions, such as smiling vs. a serious face.

Is it better to have one big image as the background for your website? Or will a white background with the image in the foreground have higher conversions?

The only way to find out for sure is to test your theory.

You should also consider the size and position of your images in relation to other elements on your website.

Continuously run these tests to maximize your conversion rates.

For example, let’s say you find out that a photo of a man converts higher than a photo of a woman. Now you’ve got to find the most optimal position of that photo on the page, so your tests will continue.

Test different color schemes

Colors can make a huge impact on how people see your website. That’s because different colors impact us differently.

We’re programmed to associate certain colors with certain events. For example, what color does everyone wear to a funeral (at least in the West)? We’re not trying to sound grim, but that’s one of the reasons why we automatically associate the color black with death.

Test colors for every component of your website, such as the color of the text, the menu icons, and CTA buttons.

Here’s an example that tests the colors of the call to action on this website:

The pages are identical, including the text. The only thing that’s been changed is the color. Refer back to what we said earlier about our minds automatically associating certain colors with certain things or occasions.

Well, we’re programmed to go on green and stop on red. So a valid hypothesis for this A/B test would be that a green CTA button will have higher conversions than the red one. You’ll have to run a test to find out if that’s true for your website.

Here are a few other suggestions to get you brainstorming.

• awards and mentions in the media
• testimonials and social proof
• videos
• advertisements

All these components can affect the behavior and actions of your visitors.

Ultimately, these actions can impact your conversions.

Once you set a goal, you can form a hypothesis to test to determine whether that solution will help you reach those goals.

For example, let’s say your goal is to increase conversions.

Your hypothesis is that increasing the size of your call-to-action button and making it more prominent on your homepage will increase conversion rates.

Then you devise a split test to test that hypothesis.

Here’s an example from Yuppiechef:

Yuppiechef hypothesized that their website users were too distracted by their navigation menu.

They thought that visitors had too many options to click, so they weren’t selecting the CTA button.

What did they do?

Yuppiechef removed the navigation bar for their variation page of the A/B test.

The test layout resulted in a 100% increase in their conversions.

Highrise used A/B testing to test a hypothesis about the header on their homepage:

Altering this heading increased clicks by 30%.

To sum up the process:

• set a goal
• come up with a hypothesis (what elements should I change to try to achieve this goal?)
• run an A/B test
• analyze the results

If you change too many components of your website, it will be extremely difficult to accurately test your hypothesis.

Select a platform to run your A/B tests

Okay.

You’ve got a goal and a hypothesis.

But how do you implement these tests on your website?

Not everyone who operates a website is a computer engineer or programmer.

Fortunately, you don’t need to be either to run an A/B test.

There are plenty of tools and resources that can help you do that.

Here are some places to start.

Visual Website Optimizer is a great option that you can try.

It enables you to carry out and measure tests the same way you can with Google Content Experiments, but it also includes a what-you-see-is-what-you-get (WYSIWYG) editor. The WYSIWYG editor enables you to make changes without having to change the underlying HTML or CSS code to run a test.

You can change a headline or button copy and run a test without having to enlist IT resources to set it up.

You can also edit the CSS and Javascript if that’s something you know how to do, but it’s not required for many tests.

Thus, the benefit of VWO and many other testing platforms is that they enable you to run tests without using IT resources, something that is in short supply for most businesses.

Optimizely also offers A/B testing.

Like with Google Analytics, you can run multivariate tests in addition to A/B tests with Optimizely.

It’s easy for you to edit and change virtually every element of your website’s design.

You do need to sign up for a paid subscription to use their platform.

However, Optimizely offers you a free 30-day trial to check out their software—you don’t need to commit to a subscription right away.

Optimizely generates a line of code for you to insert into your HTML.

It’s easy to follow their instructions, and you’ll see results based on your testing in real time.

If you have a mobile application, Optimizely allows you to run experiments on your app as well.

Unbounce is another popular choice for A/B testing.

Here’s what they offer.

You can build a landing page with high conversion rates.

Integrate your analytics, marketing automation, CRM tools, or email campaigns with their software.

Their A/B testing lets you optimize conversions, converting traffic into leads and sales.

We like their drag-and-drop format to customize your website.

This feature makes it easy to make changes to your A/B tests.

It’s another paid subscription software.

If you’re on the fence about which software to use for your A/B experiments, I would definitely recommend trying one of the options we just discussed:

These are all easy to use, regardless of your goals.

Prioritizing The List

At this point you’ll usually have a short to semi-long list of tests you can run.

It’s important to prioritize the tests based on which one you think will have the biggest impact on your conversions.

Do you think adding a testimonial is more likely to lead to a win, or do you think new call-to-action button copy will make the difference?

Either way, the goal is to come up with a list of tests you can run, and then to prioritize them based on which one you think will make the biggest difference.

It’s also good to keep track of the test ideas that you have.

You may want to test a new headline, try new call-to-action button copy, and consider adding a free shipping icon in a prominent place on your site.

These are all great ideas, but if you test them one at a time, you’ll need to create a list so you can keep track of your ideas and not lose any really good ones.

The best way to do this is by creating a spreadsheet where you can log and prioritize your ideas for future tests.

This is an example of a spreadsheet you can create to keep track and prioritize your testing ideas.

Something else to keep in mind is that testing one item at a time can lead to incremental improvements, but big changes are more likely to lead to a big win.

For example, testing ten small elements may improve conversions by 25%, but one big change can cause them to go up by 125%. It’s recommended that you experiment with bigger changes in order to get bigger wins, but when first getting started, small tests are good to run as a way to get your feet wet and to learn how to run and measure tests.

Setting Up The Test

The next step is to set up the test. For our experiment we’re going to use Visual Website Optimizer as our testing platform but this will vary depending on what platform you use.

First, we’re going to log in to VWO and click “Create Test” to get started.

Next, we’re going to select “A/B test” in order to set up an A/B split test.

You can always run a multivariate test or a split URL test if that makes more sense for the test you’re running, but we recommend starting out with A/B tests until you get more comfortable with running an experiment and evaluating the results.

The next step is to enter the url we’ll be testing.

Next, we’ll use the VWO WYSIWYG editor to create a variation with the new headline. As you’ll notice, it’s really easy to change the headline, and the font and size are the same as the current version of the site.

You can use html to customize the new version, but simple tests can be run without needing to write a single line of code.

Now that we’re on step four, we’ll enter the actions we would like to track. The first step we’d like to measure for this test is the number of people that go from the homepage to the next step in the funnel.

After that, we’d like to track the number of people that sign up for a free account.

It’s important to remember that the final conversion goal is what really matters.

It’s great to get more visitors to click from the homepage to the signup page, but what matters even more is the number of people that end up signing up for a free trial.

And then after that, what matters the most is how many people sign up for a paid subscription.

Those are the conversion goals that matter the most for a SaaS business, not the number of people who click through from the homepage to the signup page.

With that said, it’s useful to know what the conversion rates are for each step of the process.

Sometimes you may notice that there’s a bottleneck between the homepage and the signup page, so you come up with a test that you hope will get more people to click through.

But if you do, remember to always test for the final conversion since just because more people click through doesn’t mean more people will sign up for your service or for a free trial.

And just because more people sign up for a free trial doesn’t mean more will become paid customers.

You need to measure each step of the process, and you should always remember to choose the winner based on the results that will have the biggest impact on your bottom line.

Now finally, for the last step of the process, we’ll name the test, add some notes, determine what percentage of traffic to use, and decide whether or not to filter out any types of traffic such as location, paid, organic, etc. Here’s a further description of each:

• The name is simply what you’d like to call the test so you can differentiate it from the different tests you’ve run. A simple name like “Crazy Egg Homepage Headline” works just fine.
• Notes are where you can keep track of the hypothesis for the test or any details about the different variations. This part is optional and can be used if you’d like to record notes about the experiment you’re running.
• The percentage of traffic gives you the option of not testing on all of your traffic. Most of the time you’ll leave it at 100%, but if needed for whatever reason, you can always lower it so not every visitor is part of the experiment.
• Target test to a segment allows you to filter out certain kinds of traffic. You could filter out mobile traffic, desktop traffic, or international visitors, depending on what makes sense for your business. For example, if you run a U.S. based site that only sells to customers in the U.S., you can filter out international traffic so it doesn’t impact the results in any way.

Your new variation is now ready for testing.

Now that it’s created, VWO will automatically show the new version to roughly half of your traffic and the original version (the control) to the other half.

After enough traffic has been sent to both versions, a winning variation will be selected which you can then implement permanently (or at least until the next winning variation is found).

If the new version does not improve conversion rates, then you can stop the test and move on to the next option in your list of conversion tests for your site.

Understand the statistics behind your data

All right, as we said before, your A/B test will help you test your hypothesis.

Once you have the results, you’ll need to make sense of them.

This is a basic statistical experiment.

If you slept through your high school or college statistics course, we’ll give you a quick refresher so you can effectively interpret and analyze the results.

Here are some basic terms to get familiar with:

• mean
• variance
• sampling

The mean is an average value of something.

Variance measures the average variability of your results.

The higher the variability, the less accurate your mean (or average) will be for the experiment.

You can use an A/A test to detect any natural variance on your website.

Here’s an example of A/A testing to determine the variance.

The two homepages above are identical.

However, the one on the right had 15% more conversions.

You can do the same on your website by splitting the traffic between two identical pages.

It’s important to know this information before you start the A/B test.

Here’s why.

Let’s say the A/B test yields a 15% higher conversion rate for the page you’re testing.

Well, if your natural variance is already 15%, the A/B is inconclusive.

If you don’t know your variance from the A/A test, it could potentially give you a false positive result when you run the A/B test.

Your sample size is also important.

There’s no fixed number of visitors you need to get or set number of days you need to run your test for.

Continue your A/B test for as long as you have to.

Here’s an example of some results you may see after a couple of days:

At first glance, it appears that your variation was unsuccessful.

But your sample size isn’t large enough yet.

Here are the results of that same test two weeks later:

Sure, you want to stay on top of your data so you can measure the outcome.

But don’t do this too soon.

If you ended your test after the first couple of days, you would have missed out on all this additional information.

After running the test for two weeks, you can see there was a 25.18% improvement in the variation of your control page.

It’s statistics 101.

Yes, I know you’re excited to see the results so you can come up with a finalized page.

Rushing won’t help.

Take your time so you can get accurate results.

Use A/B tests to improve your emails

Everything we’ve discussed so far is related to your website. But that’s not the only platform you use to get conversions.

A/B testing is not limited to your website. You can also test factors within your email marketing campaigns.

Here too, you can focus on your CTA, colors, images, headings, and text.

But you can test other factors as well.

For example, use A/B testing on your email subject lines. The content of your message can be the same, but see if you have a significant difference in open rates and conversions based on the subject.

Test the word count of your marketing emails.

Recent studies suggest that emails between 50 and 125 words have the highest conversion rates.

Think outside the box when you’re running these A/B tests.

Don’t assume you’re only allowed to test elements of your website. Email campaigns are another viable option.

Revisit your early tests

A/B tests don’t last just a day or two.

Typically, you’ll want to run each test for at least a few weeks to make sure you’ve got a large enough sample size to yield conclusive results.

Let’s say you run 4 or 5 different tests on your CTA button. That alone could take 3 months. Then you test your headlines, images, colors, and checkout process.

By the time you get through all these tests, a year or two could have passed since your initial test.

Well, don’t stop now. Go back and see if your CTA is still as efficient as possible.

Conclusion

A/B testing is one of the best ways to increase your conversion rates.

But you don’t run just one test and call it a day. This process needs to be a continuous part of your marketing strategy. Always strive to make improvements to your website.

Test things like your CTA buttons, headlines, and checkout process.

You can also test visual elements such as your images and color schemes on your pages.

A/B testing isn’t restricted to your website. You can apply this strategy to your email marketing campaigns as well.

Once you finish testing something, move on to the next element. After you’ve tested everything, start at the beginning.

Now that you know all about A/B testing, you are ready to start. You know everything from what A/B testing is to how to run and optimize tests.

If you haven’t started yet, you should consider running your first test in the next few weeks. It will help you reduce your acquisition cost per customer and give you a leg up on your competition.