Knowledge Bridge | Time for A/B testing

//Tomas Ulej /February 1 / 2014

Time for A/B testing

Out of all the tools available for media websites, only a few are more useful than A/B testing. A/B testing allows for much better decision-making along with fine-tuning the website into perfection. At the same time, it’s one of the easiest things to do. To master it, you don’t need to read five books nor attend a specialized training. If you are not utilizing it yet, we recommend you start – ideally now.

So what is A/B testing? Simply said, A/B testing is a method which allows you to measure the effect of changes to your website on the visitors without actually making them.

Imagine as an example a web page with an article to which you make a small change – let’s say you move a picture from its current place in the right column into the center. Now comes the trick: you display the new layout version to a very small portion of your visitors. What will happen? Will visitors stay on the page for a longer or shorter time? Will a smaller or a larger percentage of readers finish reading the article?

After a few thousand views of both versions, you will be able to neatly and relatively exactly compare and determine whether the change is – or is not – an improvement. If indeed it is, you know what to do – change the picture position for everyone. If not, no problem – you can test some more.

You can test any changes this way – always on a small percentage of visitors and find out whether something works even before widely implementing it. And what is important, you are not limited to testing just one change but for example five different options. Using the same example as above, you can test positioning the photo in various places (left column, right column, top, center…) and by trial, find out where its positioning works the best.

Simply speaking, A/B testing gives you a luxury that we would probably welcome in all parts of life – the possibility of not having to make a decision every time we want to do something and then wait and worry whether we have made the right decision.

Instead, thanks to A/B testing, you have the option to choose the best solution from the multiple possibilities before making it available to all your readers.

Why should we run A/B tests?

A/B testing is one of the most practical tools for improving your website, primarily for the following reasons:

Low cost, quick start: one of the best things about A/B testing is that, unlike using questionnaires or usability tests, you can start immediately. And if you decide to use one of the free services that are available, you won’t even incur any costs. No complicated searches for participants or long preparation times. One test idea and an observation of certain rules discussed below and you can be on your way. You don’t have to hire an expert. You can easily do it just with this article.

The right sample: With a survey or questionnaire, you will always face questions such as “do we have the right sample of people?” or whether the group of people surveyed had been “representative”. There are no such questions in A/B testing – you know 100 percent that you have the right sample. It’s the people that come to your website, exactly those that you are improving your site for. There is no need to look for anyone elsewhere.

Very practical: Standard questionnaire surveys often bring theoretical findings, such as people are more interested in sports news or perhaps that they would welcome more videos in articles. But what should an ideal piece of sports news look like? Where exactly should the video be in the article? A/B testing, together with usability tests, offer very practical findings, which are – as a bonus – all immediately applicable. They not only provide you with information about what to do but also how exactly to do it.

Getting started with A/B testing

So how do we do A/B testing?

Hypothesis. At the beginning of each test, having a “hypothesis”, i.e. a question you want answered by the test, is very important. For example – “Would visitors click more on an article if it had a larger headline?” Any new function you would like to implement on the website is also a hypothesis – e.g. will the function bring a better reaction from the visitors? You probably already have a number of ideas as to what to test. You can also find some tips below.
Alternate versions. The second step is the proposal of changes with which we will verify the hypothesis – let’s say the creation of alternative versions of the original website with new functionality or design. At this point, it is important not to limit yourself to just one alternative to your current site. You can come up with as many versions as you would like. This way, you can be sure that you have not missed the best solution possible. For example, if you are testing a larger headline for your article, why not include various fonts, colors and/or size? The test will let you know the best solution.
Test variables. Often, defining the test variables is a much underestimated step. A test variable is anything you can measure and which allows you to objectively determine which page is better. It could be the number of clicks on a certain link, the time spent on a page or even whether a visitor had completed a certain operation (confirmation of a subscription payment). People frequently measure the number of clicks, but is that always the best? Always consider carefully what to measure and test more variables. That way you can see whether your new version won’t do more harm than good (e.g. the change forces more readers to click but they leave the page immediately because they hadn’t found what they were looking for).

Which tools to use for A/B testing?

Once you have done all three things listed above, only one thing remains – deploy the test. Let’s look closely at how to do it.

Ideally, you would use one of the ready-made solutions available. There are plenty of them and some are even for free. And it is certainly simpler than having the A/B testing tool custom programmed directly for your website – a solution which is possible, but definitely not simple.

When choosing, look mainly for simple implementation (do you have a programmer at hand or will the tests be administered by someone who knows programming?). Of course, the price is also crucial. Paid tools often offer free tutorials or assistance, so free does not always equal the best.

Here is a list of some of the best known A/B testing tools:

Google Analytics is among the best known A/B testing tools. One of its main advantages when compared to competing products is the fact that it’s free. And it works in a simple way: basically, you create and add all versions of your web page to the server – if the current page is at article.php, you create article1.php, article2.php, etc.

What follows is an easy step-by-step process of implementation:

If you do not have an account in Google Analytics, create one and after registering, click “Experiments” in the “Behavior” section of your website profile. Click on “Create experiment”.

Type in the URL of the website you want to test (in our example, it’s article.php) and select the measured variable – for example average visit duration, revenue, etc. Decide the percentage of your users you want to include in the experiment – determine the number according to the number of users that usually visit your site – if you have a lot of visitors, you can test the page on 1% of your visitors. If, however, you have few visitors, you could wait for the results from a 1% sample for a long time.

Then you just add the URLs of the changed pages to be tested. The last step is to add a special JavaScript code to each one. The code will determine which page is shown to which user.

And that’s all. You can find more details on how to do A/B testing with Google Analytics here.

Optimizely.com is one of the best known paid tools for A/B testing. Other paid tools mentioned below all work in a similar way. Compared to Google Analytics, they all have one significant advantage – you do not need a programmer to run the test – it’s enough to include in the page one very simple JavaScript code. Everything else can be “clicked” and selected without needing to know the source code. It’s almost as if you were editing text in Word or some other text editor – you can play around and change the text, the font size, anything. You choose the percentage of visitors to which you’ll display the tested version and select the measured variables. The tool saves the changes and, after starting the test, it will display your altered pages to random visitors.

Other popular A/B testing tools include Unbounce.com or VisualWebsiteOptimizer.com. Choose the one that suits you best. An overview of the best known tools is here.

Understanding the results

Most of the split testing tools offer results as you go along and even calculate which version has the highest chance of winning the test. At the same time, the tool helps to determine the right sample. At this point in the process, you don’t need to do anything else.

This, however, can become a double-edged sword. Despite the simplicity of A/B testing, it can be spoiled (just like anything in life). Watch out for these common mistakes:

One variable, one hypothesis – one of the most common mistakes in A/B testing is that we want to test too many things at the same time. An example would be two completely different article visuals (each has a completely different layout, ordering, as well as elements). Remember: one A/B test, one hypothesis you are testing, one change. This is a way to avoid potential problems of not being able to understand what actually brought the change when reading the test result. Was it the background color? Or omission of a large ad space? Always change only one thing. Then test it.

Attention – sample size too small. The second most common error is impatience. Frequently, one of the tested versions can have much better results very early on. Do not stop the test! Continue to the end of the test – it can often have very different results at the end than it seemed at the beginning. Contemporary A/B testing tools contain a function which automatically calculates the number of users you need to test in order to ensure the test is right. Here you can find a calculator which helps you to find out the approximate sample size even before running the test. If you are testing a large number of versions, it pays off to wait a little longer for results. Also, you may want to repeat the test one more time – test fewer versions the second time (e.g. original vs. winner and runner-up from the first test).

Poorly chosen metrics. Even if you manage all of the above well, there is still a chance you are failing in the selection of the metrics with which you’ll be measuring the effect of your changes. Is the version that gets more clicks really better for you? Or are you more interested in whether visitors, after they click, subscribe to your publication? Number of clicks tends to be an alpha and omega for online media so pay attention to other criteria that may prove significant for you.

Testing tips

You surely have many ideas on what to test. A/B testing is practical any time you want to introduce something new and you discuss in your team how to proceed the best way. Don’t limit yourself to new functions you’d like to add to your site though. Think about things already present on your site, especially:

Things that seem self-evident to you (if you hear someone say “people love xxx”, “the best way to do this is xxx” without having A/B test results to prove it, don’t just believe the statement – try it).

Things on your site you think are done really well (this link has many clicks, the ad in the right corner works beautifully) – what if these things can be done even better? Don’t forget that every hundredth of a conversion counts!

Things you think don’t matter (font color, font size, or moving an ad box 10 px more to the right). If you test a lot, you’ll find out that many of the results may not make much sense to you, yet they work. Do not rely on your rational criteria only and do not think that minute details don’t matter. They do – test everything.

Here are a few tips for tests you can run right now:

A correct looking lead on the home page – is a smaller headline, full-width photo and no lead, or a larger headline, smaller photo and more text better? Or should it be altogether different? Come up with all different possibilities and find out what users prefer.

Ad formats – you know that when there are too many, they don’t work; when there are only a few, conversion is higher but revenue smaller – so what is the right ratio? Try out different ad distribution on your page and select the best one.

Homepage layout – there’s an everlasting fight within online media about the homepage. Each service, each journalist wants to see his or her article there, yet they cannot all be there. What is the right number of articles for visitors to still click? Which section is better suited for the right column and which will do better in the middle? The simplest way to find out is trying it.

Test the wording of all buttons on your page. Compare “Enter the article discussion” to “Discuss the article now” for example. Test what works best and you may be surprised.

You can find lots of ideas and inspiration for A/B testing at www.abtests.com. It includes a number of case studies, so you can see the findings right away. Many of you may be surprised by them.

Still not enough? Suggested reading about A/B testing includes Always Be Testing: The Complete Guide to Google Website Optimizer. Although recommendations related to the implementation of Google Analytics experiments (the tool was formerly known as Website Optimizer) have become obsolete, the book offers hundreds of ideas and ways to improve your web site.
So what are you waiting for? Dive into A/B testing now!

Article by Tomas Ulej