As is usually the case though, the most effective and helpful tool – or rather a technique from those mentioned above – is the one that also costs the least. Have you done usability testing yet? No? In that case, if you’re going to do one single thing, throw away all the other tools and techniques and start doing usability tests – they will take your website the farthest.

In this article, we’ll take a look at what you need to do and what you should know before you start testing.

What we’ll need

A usability test is nothing more than tracking a user during the time he or she is using your site. Really, that’s it. No measuring of eye movement, level of stress or anything else that needs hi-tech tracking – just a simple recorded (shot of the computer screen they’re sitting behind and their voice) dialogue between you and them. And of course the tasks you request them to do on your website.

So, no measurable indicators – CTR, time spent on site, etc.? Precisely! Questionnaires or Google Analytics will provide you with lots of data, but it doesn’t get much better than seeing the person in action with the possibility of asking what they think. I guarantee that after a usability test, you’ll know right away what needs to be done even without knowing exact percentages or having data to feed into a graph.

Like with any cooking recipe, we too will start with the ingredients we’ll need. There isn’t much of it, yet it’s still a good thing to specify them.

• A quiet room – it may sound strange but if there is something that can ruin the result of a usability test, it’s having someone disturb you. Are you in an open space office? If so, forget right away that you would do a usability test at your work station. An ideal place for a usability test is a small room, two chairs and a desk – no disturbing elements. Let people know not to disturb you during the time of the test. Turn off your phone.

• A standard computer and browser – although the object of testing during the usability test will be your website, people you’ll be working with will have a feeling you are testing them. Despite being told otherwise, they will not get rid of this feeling. Don’t add to their stress by making them learn to use your gaming mouse or keyboard with different characters. Use the most standard computer most people are used to. In the author’s country, this would be a laptop with Windows, a plain mouse and Chrome browser. Have more browsers prepared and let the participant choose which to use. The more they feel at ease and as if working with their own computer, the better for them and you too.

• Microphone and a program for recording your computer screen – During the test, you will record what the participant is saying as well as the browser window they’ll be surfing in. This is important because you won’t have time to take notes and will not remember all the issues that will come up. Laptops, as well as desktops, often have built-in microphones; however, it has proven useful to the author of this text to have an external microphone. A simple cheap mic for Skyping will do very well.

Selection of a recording program may be something of an obstacle as there aren’t many good quality solutions. For us, a good choice proved to be Camtasia Studio. It can record the screen as well as sound and later allows the recording to be cut, highlight parts of the video such as a mouse click, rolling over an element, etc. The program is not exactly cheap but has a 30-day trial version, so you can start with tests for free and later decide whether investing in the software is worth it.

What will we test?

Question “What will we test?” may sound ridiculous to you. You probably already know about the weak spots on your website – perhaps you were notified by email from users, or you’ve seen something in the statistics. Maybe you’ve asked yourself already why no one clicks on the Editorial Team link at the bottom of the page, while at the same time you get asked by users whether you’ve considered having one.

If you haven’t done usability tests before, I would guess the first ones you do will be of a very general character. You will want to look at your website as it is and its key functions, and find out whether users know how to use them or if they have a problem using them, and last but not least, how you could improve and simplify the way your website is used.

Later, you will probably concentrate more on in-depth issues and usability tests will focus on only one of the sections or just an individual function of your website. Either way, your task now is to transform all intentions into assignments for the participants in your test. How do you go about it?

Close your eyes and imagine all the key functions that make up your website. You can forget about the details for now – the most important elements will do for now. Imagine an average person and what he or she actually does on your website. Usually arrives, looks at the homepage, clicks on an article. Then what? Perhaps he wants to enter the discussion? Or maybe he wants to sign into your system email and read messages? Obviously, it’s different for every website but you should always keep in mind the most frequent and concrete scenarios of your website’s visitors.

Once you have them, you can start turning these scenarios into a story with assignments, ideally with some update theme. Set up concrete and fun assignments.

For example: “You’ve seen on TV that the government wants to increase taxes for people with higher income. You missed some parts of this information, therefore you decided to visit our website and confirm the missing details. Where would you look for this information on our website? Find out the percentage of this planned tax increase and the date the tax law should become effective”.

Notice that I haven’t used in this assignment the name of the section (go to section Economy, in the left-side menu click item xxx). This is exactly the method you should use too, as normally a user doesn’t know where things are. Since nobody tells users on the homepage where to find the information they are looking for, they have to find it on their own. Effectively, the user will now test your site for you and whether you have developed it well. Following the user’s path to finding the right link will show you how to make your website better.

Prepare about ten assignments in a similar manner. For each of them, try to create a specific situation and make it interesting or fun so as not to bore the participant. Be sure to make it clear exactly when the given assignment can be considered finished.

Are you testing one specific part of your website? It’s good to have the first two assignments of a different kind – “to trick” a little, but also for loosening up the user. When testing a specific thing, I personally try to hide it by breaking it into a number of different assignments so I don’t give away my intention. Ideally, at the end of the test, the participant shouldn’t really know what the main purpose of the whole test was.

Ready for some inspiration? A complete usability test can be found here: http://www.youtube.com/watch?v=QckIzHC99Xc

Selecting participants

How many people need to take the test in order to reach the optimal results? It may come as a shock to you, but 6-7 people will be enough.

Are you wondering why so few are needed when for any A/B testing questionnaire you need to have at least a few hundred participants to consider the data reliable? The answer is that when it comes to usability issues, we are all very much alike. You will find this out as soon as you start testing: the second participant will have very similar problems with your website as the first one, and this will repeat until you get to the last one. Using, say, 20 people instead of ten of course means finding slightly more issues on your website, but I guarantee that testing 6-7 people is absolutely sufficient for finding the most significant ones.

Which people to choose? You may be tempted to strictly adhere to demographic distribution, so here is my tip – don’t take it too seriously. (In the end, it will be hard to stick to a demographic distribution of users in a sample of 6-7 people.) Include men and women, younger and older, but the best thing you can do is to call on users that could potentially have a problem with your site.  If you call on a programmer to take part, he’ll probably be able to manage almost anything, but how about a user who isn’t so good with computers in the first place? You are better off choosing participants that will show you as many deficiencies as possible.

An important rule: never invite someone who is a “heavy user” of your site – a member of your editorial team for example, if you have a news website. Of course, there are exceptions when this rule does not apply (e.g. intranet testing), but for the majority of websites, inviting outside people – strangers not familiar with your website – is crucial. Ideally, of course, you would call on people that do not know your site at all. This, however, cannot be done in all cases – if your website targets brokers, an ordinary person won’t be able to help you much.

Another thing to consider is the question of payment for taking the test. You’ll find out in US literature that you should. My personal experience is to invite people without the promise of remuneration but rewarding them at the end anyhow (usually not with money though, but rather in the form of a small gift such as a book).

You may now have a good idea of who to invite. If not, here are a few tips from practice:

• Create a site with a simple questionnaire for people who would like to help with website development and link to it from your site. In a few hours, you are sure to have perhaps hundreds of contacts for willing people.

• If there is a place where users come, e.g. the reception area of a newspaper office where they renew subscriptions, position yourself there and ask directly whether they have time to take the usability test. Another way to do this is for the person on the desk in reception to ask and take the interested users directly to your office.

• Place an ad in the classifieds or post it on your Facebook page.

If you don’t want to do any of the above? We all have friends and acquaintances who don’t even know where we work or have never visited our websites – use them. Just one rule to keep in mind: never, never invite the same person for tests twice, especially if it’s the same test!

Carrying out a usability test

You have a quiet room, you have a computer ready (you’ve tested the recording and mic functionality), you have people coming and you have the assignments ready for them. What now? It’s time to carry out the usability test.

At the beginning of the test, you have to tell people what awaits them. In order not to influence the results, it’s ideal to tell everyone the same thing – that is, read the same text to all of them. You don’t have to come up with the wording on your own; usability expert Steve Krug has done it for you already, so don’t worry. You can download the English version of it here.

The text describes what will happen during the usability test and also includes a printable form of agreement for making sound and video recording. So now you can smoothly begin your assignments.

Now comes the most important part. You could have done everything perfectly up to now, but this is the point when everything can go wrong – your communication during the test. The following are the three most fatal mistakes. Avoiding them and sticking to the following method can make usability testing a brilliant tool.

• Do not help. Do you remember how you deliberately didn’t include specific instructions about what to do? This is exactly what you shouldn’t do either while the test is in process. Let’s say a participant has a problem with completing your assignment (can’t find a link to a page or has forgotten her password). Do not help her under any circumstances. You can try a formulation such as, “Try thinking about it some more”, “What would you do if I were not here?”, or “Unfortunately I can’t help you but as soon as the test is over, I’ll answer any of your questions”. If the participant cannot complete the assignment for a long time, end it. It is difficult not to help a person who is obviously in trouble. I personally managed to “fail” a few times when I administered my first usability test. While it is difficult, it is very important because if you help the participant, you may as well throw the results away because you can’t be sure that you didn’t bring the user somewhere she wouldn’t otherwise have got to on her own. There of course are situations when you can help – for example if a strange window, not relevant to your website, pops up, or the mouse is disconnected, etc. – but don’t ever say, “And why don’t you try clicking here?”

• Ask what they’re thinking about – keep on prompting the participant to say what he’s thinking about, to verbalize what he’s trying to do as much as possible. The instruction to do so was already included in the introductory text you read out at the beginning, however, some participants will tend to forget – you should do all you can to make sure there isn’t silence in the room and the participant speaks as much as possible. That is the key to improving your website – every one of the participant’s thoughts will move you forward.

• Praise and eat humble pie – many people you invite will say, “I’m not smart enough for this” and when they can’t do something, they will repeat the phrase. Participants will constantly feel as if you are testing their abilities. Use every opportunity to disagree with them: “I’m not sure where to click?” “Thank you – you have just found a huge shortcoming in our website. Actually, you’re doing it right, and it’s us who developed the website who got it wrong because you can’t find what you’re looking for.” After the test is over, tell them again how much they have helped you. You can even show them the right solution to their task.

After the test

The first thing that shows you’ve done the usability test right is that the results are absolutely horrible. Users had problems finishing your tasks, you’ve noticed mistakes you made in planning the site, users do things in a very different way than you thought when building the site. These are the results you’re looking for. If you, the website developer, feel miserable, be assured that this is exactly the kind of feeling you needed to reach.

In order to transform the usability test results into a better website, do the following:

• Share the results with everyone – Do not keep, under any circumstance, the test results just to yourself. Let as many colleagues as possible see the video recorded during the test. Definitely show it to designers, project managers and programmers. Don’t be shy to show it to other people who worked on the development of the site. Practically anyone from the team can learn something from a usability test. Before you show them the video, give them a pen and paper and ask them to take notes with ideas on what to change on the web in order to prevent some of the problems the next time around. I’ll bet everyone will have a paper full of various thoughts. Are you all too busy to watch a 5- or 6-hour long recording? Cut the usability test video – leave out the parts where participants succeed in their tasks and pick out the “cherries on the top”.

• Have a discussion and make a list of changes. After watching the video, compare your lists and write the problems on a big board. A huge discussion is surely to arise. A long list will be created. Set a priority for each item – mark the issues according to how big they are.

You will notice as you go along how the view of the web changes in the team. At the beginning of the exercise, someone may say something like, “Are these users completely stupid?” And then, when they see that every test participant had a problem with the same thing, they will go quiet. You will notice the effect of a usability test on the work of the team in the future. A programmer will try to simplify everything as much as possible right in the beginning. A project manager may want to run a usability test during the design stage. The owner will approve a budget for usability tests.

The possibility of seeing users utilizing your site in real time is something that will completely change the way you work.

How to solve issues discovered

One of the things that happens to you after a usability test is that you suddenly get a feeling that you completely need to change your website. You’ll probably feel that it’s badly designed; programmers may arrive with an idea to re-program the whole site. One of the worst things you could do, though, would be succumbing to these temptations.

Any usability issue found can be resolved in two ways: the complex way (total redesign) or the dilettante/superficial way (do the absolute minimum in order to remove the problem). Based on a number of years of experience, I recommend choosing the second option.

If you don’t believe me, take a look at the following list:

Dilettante solution vs. complex solution

• Much less people will be affected by the issue / No one or almost no one will be affected by the error
• Easily implemented / Requires much effort to implement
• Finished in a few minutes, a day the most / Can take a month or two, and may actually never be finished

Complex solutions seem to be clean and thorough. In reality though, it means you are starting from zero. In the end, the fact that you have done it over again doesn’t mean that you have not made the mistakes again or haven’t created new ones. On the contrary, if you redo something completely, you will end up facing a dilemma at the end what to do with the new issues.

Usability expert Steve Krug introduced a rule he calls “Do as little as possible”. Always look for the fastest solution to fix an issue. Only when that doesn’t work, use something more complex.

Here is an overview of Krug’s points:

Do not redesign, tweak (Krug’s No. 1 rule of successful mistake correction)

• Everybody hates redesigns, so if you suggest a wholly new solution, they will complain anyway.
• You may solve your problem with a redesign, however, it’s almost certain you will make at least five other, perhaps more serious, mistakes.
• Do the smallest changes possible – tweak until the problem is solved. If that doesn’t help, revert to a more serious solution.
• It often means that you want to increase the font, change the background color of a message or something along those lines.

Do not add, removing something is much better (Krug’s No. 2 rule of successful mistake correction)

• People usually feel that adding something will work better, let’s say a new notice or phrase. Often, it’s much better to remove something. Most mistakes are actually caused by organization on the site being so complicated that it becomes chaotic. This is a very frequent reason for mistakes.
• Always consider the list of issues in relation to effectiveness. If there are small issues which affect only a few users and would take too long to fix, don’t bother with them. There are probably ten larger issues that can be addressed quickly. Focus on these first and then on the others.

How do you find out whether you have improved the product? Simple, run the same usability test again … with different participants, of course.

Now you know everything important about usability tests. If you’d like to know more, I recommend two of Steve Krug’s books: Don’t make me think and Rocket surgery made easy. They are short, funny and will tell you everything you need to know.

Mr Haile said: “We’ve found effectively no correlation between social shares and people actually reading.” Chartbeat, which measures real-time traffic for many online publishers,  later clarified that he was talking specifically about tweets, though they expect research to show the same pattern for Facebook.

Some industry leaders have been calling for online publishers to move beyond measuring simple user numbers and to start measuring engagement instead, through indicators such as social media shares and likes. They argue that this gives publishers and advertisers a better insight into how relevant consumers find their content: people would only share an article they like.

But if it’s true that there is no correlation between social shares and genuine engagement, the industry may have to think again about the next step in understanding how to measure the real value of content.

So what is A/B testing? Simply said, A/B testing is a method which allows you to measure the effect of changes to your website on the visitors without actually making them.

Imagine as an example a web page with an article to which you make a small change – let’s say you move a picture from its current place in the right column into the center. Now comes the trick: you display the new layout version to a very small portion of your visitors. What will happen? Will visitors stay on the page for a longer or shorter time? Will a smaller or a larger percentage of readers finish reading the article?

After a few thousand views of both versions, you will be able to neatly and relatively exactly compare and determine whether the change is – or is not – an improvement. If indeed it is, you know what to do – change the picture position for everyone. If not, no problem – you can test some more.

You can test any changes this way – always on a small percentage of visitors and find out whether something works even before widely implementing it. And what is important, you are not limited to testing just one change but for example five different options. Using the same example as above, you can test positioning the photo in various places (left column, right column, top, center…) and by trial, find out where its positioning works the best.

Simply speaking, A/B testing gives you a luxury that we would probably welcome in all parts of life – the possibility of not having to make a decision every time we want to do something and then wait and worry whether we have made the right decision.

Instead, thanks to A/B testing, you have the option to choose the best solution from the multiple possibilities before making it available to all your readers.

Why should we run A/B tests?

A/B testing is one of the most practical tools for improving your website, primarily for the following reasons:

• Low cost, quick start: one of the best things about A/B testing is that, unlike using questionnaires or usability tests, you can start immediately. And if you decide to use one of the free services that are available, you won’t even incur any costs. No complicated searches for participants or long preparation times. One test idea and an observation of certain rules discussed below and you can be on your way. You don’t have to hire an expert. You can easily do it just with this article.

• The right sample: With a survey or questionnaire, you will always face questions such as “do we have the right sample of people?” or whether the group of people surveyed had been “representative”. There are no such questions in A/B testing – you know 100 percent that you have the right sample. It’s the people that come to your website, exactly those that you are improving your site for. There is no need to look for anyone elsewhere.

• Very practical: Standard questionnaire surveys often bring theoretical findings, such as people are more interested in sports news or perhaps that they would welcome more videos in articles. But what should an ideal piece of sports news look like? Where exactly should the video be in the article? A/B testing, together with usability tests, offer very practical findings, which are – as a bonus – all immediately applicable. They not only provide you with information about what to do but also how exactly to do it.

Getting started with A/B testing

So how do we do A/B testing?

1. Hypothesis. At the beginning of each test, having a “hypothesis”, i.e. a question you want answered by the test, is very important. For example – “Would visitors click more on an article if it had a larger headline?” Any new function you would like to implement on the website is also a hypothesis – e.g. will the function bring a better reaction from the visitors? You probably already have a number of ideas as to what to test. You can also find some tips below.
2. Alternate versions. The second step is the proposal of changes with which we will verify the hypothesis – let’s say the creation of alternative versions of the original website with new functionality or design. At this point, it is important not to limit yourself to just one alternative to your current site. You can come up with as many versions as you would like. This way, you can be sure that you have not missed the best solution possible. For example, if you are testing a larger headline for your article, why not include various fonts, colors and/or size? The test will let you know the best solution.
3. Test variables. Often, defining the test variables is a much underestimated step. A test variable is anything you can measure and which allows you to objectively determine which page is better. It could be the number of clicks on a certain link, the time spent on a page or even whether a visitor had completed a certain operation (confirmation of a subscription payment). People frequently measure the number of clicks, but is that always the best? Always consider carefully what to measure and test more variables. That way you can see whether your new version won’t do more harm than good (e.g. the change forces more readers to click but they leave the page immediately because they hadn’t found what they were looking for).

Which tools to use for A/B testing?

Once you have done all three things listed above, only one thing remains – deploy the test. Let’s look closely at how to do it.

Ideally, you would use one of the ready-made solutions available. There are plenty of them and some are even for free. And it is certainly simpler than having the A/B testing tool custom programmed directly for your website – a solution which is possible, but definitely not simple.

When choosing, look mainly for simple implementation (do you have a programmer at hand or will the tests be administered by someone who knows programming?). Of course, the price is also crucial. Paid tools often offer free tutorials or assistance, so free does not always equal the best.

Here is a list of some of the best known A/B testing tools:

Google Analytics is among the best known A/B testing tools. One of its main advantages when compared to competing products is the fact that it’s free. And it works in a simple way: basically, you create and add all versions of your web page to the server – if the current page is at article.php, you create article1.php, article2.php, etc.

What follows is an easy step-by-step process of implementation:

If you do not have an account in Google Analytics, create one and after registering, click “Experiments” in the “Behavior” section of your website profile. Click on “Create experiment”.

Type in the URL of the website you want to test (in our example, it’s article.php) and select the measured variable – for example average visit duration, revenue, etc. Decide the percentage of your users you want to include in the experiment – determine the number according to the number of users that usually visit your site – if you have a lot of visitors, you can test the page on 1% of your visitors. If, however, you have few visitors, you could wait for the results from a 1% sample for a long time.

Then you just add the URLs of the changed pages to be tested. The last step is to add a special JavaScript code to each one. The code will determine which page is shown to which user.

And that’s all. You can find more details on how to do A/B testing with Google Analytics here.

Optimizely.com is one of the best known paid tools for A/B testing. Other paid tools mentioned below all work in a similar way. Compared to Google Analytics, they all have one significant advantage – you do not need a programmer to run the test – it’s enough to include in the page one very simple JavaScript code. Everything else can be “clicked” and selected without needing to know the source code. It’s almost as if you were editing text in Word or some other text editor – you can play around and change the text, the font size, anything. You choose the percentage of visitors to which you’ll display the tested version and select the measured variables. The tool saves the changes and, after starting the test, it will display your altered pages to random visitors.

Other popular A/B testing tools include Unbounce.com or VisualWebsiteOptimizer.com. Choose the one that suits you best. An overview of the best known tools is here.

Understanding the results

Most of the split testing tools offer results as you go along and even calculate which version has the highest chance of winning the test. At the same time, the tool helps to determine the right sample. At this point in the process, you don’t need to do anything else.

This, however, can become a double-edged sword. Despite the simplicity of A/B testing, it can be spoiled (just like anything in life). Watch out for these common mistakes:

• One variable, one hypothesis – one of the most common mistakes in A/B testing is that we want to test too many things at the same time. An example would be two completely different article visuals (each has a completely different layout, ordering, as well as elements). Remember: one A/B test, one hypothesis you are testing, one change. This is a way to avoid potential problems of not being able to understand what actually brought the change when reading the test result. Was it the background color? Or omission of a large ad space? Always change only one thing. Then test it.

• Attention – sample size too small. The second most common error is impatience. Frequently, one of the tested versions can have much better results very early on. Do not stop the test! Continue to the end of the test – it can often have very different results at the end than it seemed at the beginning. Contemporary A/B testing tools contain a function which automatically calculates the number of users you need to test in order to ensure the test is right. Here you can find a calculator which helps you to find out the approximate sample size even before running the test. If you are testing a large number of versions, it pays off to wait a little longer for results. Also, you may want to repeat the test one more time – test fewer versions the second time (e.g. original vs. winner and runner-up from the first test).

• Poorly chosen metrics. Even if you manage all of the above well, there is still a chance you are failing in the selection of the metrics with which you’ll be measuring the effect of your changes. Is the version that gets more clicks really better for you? Or are you more interested in whether visitors, after they click, subscribe to your publication? Number of clicks tends to be an alpha and omega for online media so pay attention to other criteria that may prove significant for you.

Testing tips

You surely have many ideas on what to test. A/B testing is practical any time you want to introduce something new and you discuss in your team how to proceed the best way. Don’t limit yourself to new functions you’d like to add to your site though. Think about things already present on your site, especially:

• Things that seem self-evident to you (if you hear someone say “people love xxx”, “the best way to do this is xxx” without having A/B test results to prove it, don’t just believe the statement – try it).

• Things on your site you think are done really well (this link has many clicks, the ad in the right corner works beautifully) – what if these things can be done even better? Don’t forget that every hundredth of a conversion counts!

• Things you think don’t matter (font color, font size, or moving an ad box 10 px more to the right). If you test a lot, you’ll find out that many of the results may not make much sense to you, yet they work. Do not rely on your rational criteria only and do not think that minute details don’t matter. They do – test everything.

Here are a few tips for tests you can run right now:

• A correct looking lead on the home page – is a smaller headline, full-width photo and no lead, or a larger headline, smaller photo and more text better? Or should it be altogether different? Come up with all different possibilities and find out what users prefer.

• Ad formats – you know that when there are too many, they don’t work; when there are only a few, conversion is higher but revenue smaller – so what is the right ratio? Try out different ad distribution on your page and select the best one.

• Homepage layout – there’s an everlasting fight within online media about the homepage. Each service, each journalist wants to see his or her article there, yet they cannot all be there. What is the right number of articles for visitors to still click? Which section is better suited for the right column and which will do better in the middle? The simplest way to find out is trying it.

• Test the wording of all buttons on your page. Compare “Enter the article discussion” to “Discuss the article now” for example. Test what works best and you may be surprised.

You can find lots of ideas and inspiration for A/B testing at www.abtests.com. It includes a number of case studies, so you can see the findings right away. Many of you may be surprised by them.

Still not enough? Suggested reading about A/B testing includes Always Be Testing: The Complete Guide to Google Website Optimizer. Although recommendations related to the implementation of Google Analytics experiments (the tool was formerly known as Website Optimizer) have become obsolete, the book offers hundreds of ideas and ways to improve your web site.
So what are you waiting for? Dive into A/B testing now!

Internet is often described as ‘the most measurable medium ever created’, thus real-time analytics is designed to monitor very precisely the current data on the behavior of visitors in order to help you optimize your website, content and overall online operations by delivering live and granular data. Through knowledge of what users are doing on your website right now, it is possible to understand the consequences of short-term changes and content updates, and respond to them immediately. It’s about moving with a greater speed towards previously unknown questions, defining new insights and reducing the time between what is going on on your website and your proper response or reaction within minutes.

Omnipresent generation of data in online environment is not just about transforming your content itself, it is, at the same time, also the transformation of workflow in news organizations and helping to develop effective content production strategies related to actual demand. New tools – like real-time tracking – are being deployed in newsrooms to provide insights around how your content is performing on site, in social networks, news aggregators and search engines. A new role in digital newsroom has been created: an audience editor is in charge to deliver and interpret real-time perceptiveness into the heart of the news workflow. This change in interaction with live audience improves how internal resources are deployed and what new perspective is explored in the newsroom.

Real-time analytics allows digital editors, content producers and analysts to easily inspect content viewing patterns on their own websites and to monitor visitor’s activity trends as it happens on site and react within minutes accordingly. The reports are updated continuously and each user’s activity is reported seconds after it occurs on your site. For example, you can track how many people are on your homepage right now, what is their geographic locations, the traffic sources that referred them to your site and which pages they’re interacting with after leaving your homepage. With real-time analytics, you can immediately and continuously monitor the effects that site changes and content updates have on your traffic, whether a new and changed content on your site is being viewed, what content is being consumed massively and monitor the immediate impact on traffic from a blog/social network posts or tweets. We can also see when you have stopped receiving visits from the social network, which helps you to realize when to reengage.

Better understanding of content consumption trends in progress leads your editors to more relevant, high-interest content creation. It helps to draw your attention to a particular information demand even on a kind of seemingly unimportant topics (you can also track and see, based on the traffic referrers, how people got to an article which led to rapidly growing reader’s interest trend). This way you may find that readers are significantly more attracted not by a present lead article and cover story on your homepage but greater information demand is focused to an article that is not listed on your homepage. Based on this finding, you can decide to place a link to the article on a location of your website through which people tend to visit your site. At the same time, you can instruct editors to supplement and enrich the article with related links, images, video or to add to the article a chart or infographics, and thus creating a compelling reader’s experience.

Based on continuous (a second-by-second basis) measurement of visitors’ behavior, it is easier to understand which various factors contribute to your visitors’ satisfaction (e.g. increase time spent on site), and how to convert a random visits to a loyal, returning visitor. More granular up-to-date data will generate fast yet informed decisions for your news desk. Soon, you will realize that just refining the title of an article can make a huge difference in terms of numbers of visitors accessing the article.

How to choose the right tool to fit my needs?

Before you start to compare the different real-time analytics tools, their features and the granularity of the reported data, think about what questions these tools should answer and solve for you. You must have a clear answer to questions like these: What criteria do you use to decide which articles to publish on your homepage? What are the types of things your editors disagree about internally and you have to decide intuitively? What everyday decisions have a major impact on the way your site looks and feels? According to what criteria do you control and assess the quality of content processing on your website? Do you know which time during the day is the most effective use of social networks? Who will be in charge of working with real-time analytics in the newsroom and what are his/her expectations? What would be needed to simplify and streamline the work?

Once you can answer these questions, you have got quite a clear understanding what features are a must for your daily use, so you can start comparing different real-time analytics tools.

Real-time analytics tools

There is a wide range of on-site measurement tools from different vendors that provide real-time analytics software and services. Data from your site is usually gathered via page tagging – an embedded tracking code, usually written in JavaScript or Java; increasingly Flash is used. (Of course, there are also other data collection methods, e.g. packet analyzer.)

Here you can find a very useful comparison chart of different available real-time analytics platforms: http://clicky.com/compare/. Another well worked-out comparison might be found in this Google Doc. Yet, other good up-to-date overviews of available web analytics tools are: Top 30 Web Analytics Tools and 53 Alternatives to Google Analytics.

Case studies

Reynolds Journalism Institute  hosted Journalytics Summit in September 2013. Following presentations are case studies given by representatives of real-time analytics company and their clients:

Outbrain (Visual Revenue) case study

Dennis R. Mortensen, Outbrain

Rhonda Prast, The Kansas City Star

Chartbeat case study

Joe Alicata, Chartbeat

Joy Mayer, Columbia Missourian

Parse.ly case study

Andrew Montalenti & Mike Sukmanowsky, Parse.ly
Adam Felder, Atlantic Media

Related article:

Real-time analytics can help you use scarce editorial resources more effectively

Both on the speaker’s dais and among participants, there were several clear themes for news media publishers – “big data”, paid content, branding for news media, content packaging for targeted audiences, as well as clever approaches to creating new revenue from print.  All of these themes support the view presented by Earl Wilkinson, INMA global CEO during his closing presentation.  Wilkinson underscored that digital is here to stay.  The presentation, distribution and management techniques characterized by successful digital media companies will progressively dominate in the media management world.  Wilkinson emphasized that the adoption and evolution of these techniques are at different stages of development around the world.  But in every case, digital content creation and distribution and the supporting tools will be the key to future success in digital as well as print.

First among the digital themes has to be “big data” and its role in developing advertising and audience.  Frode Eilersten, Schibsted’s newly appointed Executive Vice President for Strategy and Digital Transformation presented the media group’s investment in data acqInnuisition and analysis to support advertising sales, product development and marketing.  Eilersten who recently joined Schibsted from US consultancy McKinsey highlighted the investment required in data acquisition and analytics, but also the need to build an internal business culture to make use of the data to solve complex business problems.  Data in particular was the “secret sauce” in advertising networks that allowed a publishing company to create extra value from the advertising presented to their audience.  Data and research was presented as one of the required success factors in the development and launch of paid content at sites as different as the New York Times and the TB+ premium content model developed by Tønsbergs Blad, from a small town southeast of Olso.

Content marketing or native advertising was another important area of discussion.  In content marketing, the role of advertiser and publisher are increasingly blurred. Advertisers are now supplying, choosing or at a minimum approving content for publication on news media websites in exchange for an “advertising” fee.   The advertiser recognizes that strong editorial brands offer special “brand benefits” to their audience.  By associating themselves closing with strong news brands, advertisers enhance their own brand credibility and recognition.  There were many potential implications of this discussion.  The movement of advertisers to create their own media and sidestep news media for brand advertising was one important implication.  Bennetton’s Colours magazine or Google’s advertising and marketing quarterly Think+ were cited as examples of this trend.  For publishers, content marketing creates the need for a clear and well articulated plan for how to create and present “advertiser content” in order to maintain the overall credibility of the news brand.  Finally, speakers highlighted how publishers will increasingly need to think about investing in the brand of their news media in order to create value in the publishing brand not in just the audience for an individual story.

Finally, the conference came back to its roots and presented some interesting programs to enhance the reception and profitability of the printed product.  Two examples of this approach stood out.  First, presenters from Die Welt Kompakt and NRC.Next presented content packages developed to target young professionals.  But other examples targeting children as well as women reminded the conference that targeted content well-delivered can still create an audience attractive to advertisers.   The second approach to the product development came from Sandy MacLeod, Vice President Consumer Marketing and Strategy at The Toronto Star who presented several examples of well researched and well delivered content products like improved TV guides and puzzle books provided for a price.  MacLeod made the compelling case that there is still revenue available for the print product, if you look for it.

The INMA European News Media Conference provided a wealth of examples for regional media companies to pursue.  For many in the audience the challenge is how to minimize the risk to developing, customizing these models to their different media markets.  For large companies like Schibsted, Axel Springer or Sanoma Corp, there are ample corporate profits available to experiment without too much risk.  Internally these companies have the ability to raise internal start-up capital and to recruit and train media managers with digital capabilities.   The challenge for smaller companies lies in finding resources, both talent and capital, to develop experiments.  But the need to continually experiment, measure, improve or discontinue new products was presented repeatedly as one of the fundamental requirements to success in the emerging multiplatform world of news media.

How all kinds of data is helping newspaper companies go after more and more subscribers? Read more: The newsonomics of “Little Data,” data scientists, and conversion specialists by Ken Doctor

The study revealed that “internet users (in the country) are mainly from the ‘working middle class’, as opposed to upper (as defined by living standards measures)”, Joanna Wright wrote in the Media Online. Internet penetration is still low in the country, with upper estimates standing at 35.2 percent, according to the Stats SA 2011 Census. The Census also looked at where South Africans had access, either at home, at work, via mobile phone or some other location. It is important to note that twice as many South Africans had access via mobile phones, 16.3 percent, than had access at home.

However, while internet penetration might be low, this new study by online research firm Columinate, which was commissioned by the Digital Media and Marketing Association in South Africa, shows that use is expanding beyond higher income levels.

The Columinate study also challenged another assumption about the digital market in South Africa, that it is predominantly young. The study found that 11 percent of users were over 50 years old, and most, 60 percent, were between 25 and 49 years old.

“(It’s) not necessarily a youth market. You are accessing South Africans who work, who have disposable income,” said Elna Smit of Columinate, the online research group responsible for the study.

And internet users in South Africa are well educated. The research found that 13 percent of internet users in the country have a degree, while only one percent of the total population has a higher education degree.

Publishers need to leverage data to woo advertisers

Research like this is key to helping you win over advertisers who might be sceptical about the reach of digital advertising as your market transitions to digital media.

“Marketers often complain that digital won’t give them the mass market, the bulk of South Africans,” Smit told Media Online, but added that advertisers need to look beyond simple reach. “You should be asking: What percentage of the people who spend on your brand are you reaching?” Smit said.

To win over advertisers, you need the data to answer this question. In the early days of the digital transition, audiences might appear small when compared to traditional media, but if you have effective audience measurement tools, then you can show how targeting can deliver a more relevant message to a more receptive audience.

You can begin your audience measurement work by investing in services such as ComScore, and you can also monitor your social media metrics using Facebook’s built in tools or other social media metrics tools such as Chartbeat. Data is becoming even more important to ad sales as even real-time bidding or programmatic buying reaches larger emerging markets such as Russia and Malaysia.

Even as you grow your audience, you will still need to make sure that you speak not just of the size of your audience but the income, age and geographic distribution of your audience. News publishers and broadcasters who invest in knowing their audience will find it easier to win over advertisers and compete more effectively against targeted advertising from search engines and social networks.

With the current level of excitement about data journalism, teams around the world are developing not only new ways to analyse and display data but also new forms of storytelling that leverage its power. Data journalism is not simply about charts, graphs and visualisations, it can also be used to weave together images and text with those numbers to create incredibly rich stories.

One of the most recent examples is Connected China, a project from Thomson-Reuters, that looks at the relationships between China’s leaders, its institutions and its elites. Reg Chua, the data and innovation editor at Reuters, explained the project on his personal blog:

It’s a little hard to sum up simply; at one level, it’s a microsite that focuses on looking at power in China, explaining how it flows, the key players and institutions, and their relationships, featuring stories and rich multimedia (including fantastic archival footage.) But it’s much more than that: It’s also a series of innovative data visualizations that pull from a rich, underlying database of people, institutions and relationships to illustrate the connections, careers and positions of key officials in China. And more than that: It’s a great example of how the combination of data. visualizations, stories and multimedia can be much more than the sum of their parts.

The entire micro-site is driven by a huge database:

It’s an amazing database: Tens of thousands of entities, 30,000 relationships, and a million and a half words (not to mention the array of news stories, photos and videos also featured in the app.) The team structured tons of information – connections, the importance of job roles, etc – with an editorial sensibility. In other words, they applied news judgment – but rather than use it just in stories, they used it to structure data.

That is a pretty impressive amount of data, and the power isn’t simply in its size but also the relationships that it helps uncover. Those relationships provide context and show connections that would be hidden to all but the most knowledgeable China watcher. While this project operated on a huge scale, smaller data journalism projects can still deliver important context, such as:

• This award-winning project in Novosibirsk in Russia that mapped 1068 accidents involving pedestrians in 2011.
• Angélica Peralta Ramos of La Nación in Argentina discussed a project the newspaper did to analyse public transportation subsidies in their country.
• At MDIF’s Media Forum 2012, Roby Alampay of InterAkyson, the online division of TV5  in the Philippines, spoke of how they convinced the Metropolitan Manila Development Authority to digitise the traffic reports that they were already collecting. Previously, they had just written the reports on a whiteboard, updated every 15 minutes. Old data was simply erased. Roby said: “Render what is digital.  Digitize what is not.” He donated three cheap PCs to the MMDA so they could collect the data they were throwing away and used it to create a traffic tracker in Manila.

With all of the demands on news organisations, one of the biggest challenges of the digital transition is how to set priorities. Let’s look at some smart ways to use data and how to make a strong business case for developing data journalism in your newsroom.

Reuse your databases – One of the things that digital journalists learn quickly is that it is difficult to justify creating complex, time consuming and expensive digital projects that have only a single use. If you gather or develop databases, think of how to use them and maintain them on an ongoing basis. What data sets are key to your readers? Education? The environment? Political trends? Funding for key local services?

Also, think of the various applications you can create. Don’t just think of a chart on the website but also maybe a searchable form or even a mobile application that you can create using the data.

Building databases takes effort, and knowing the kind of databases that will generate the most applications might help you decide which ones to develop and which ones aren’t worth the effort.

Think of internal and external applications – When you’re considering applications, think of internal as well as external applications. When you look at Connected China, you can easily see how this would be an extremely useful resource for Reuters’ journalists as well as its audiences. When you are thinking of which data projects to do, consider the value that your newsrooms will get from them as well.

Assess potential revenue streams – The days when journalism could afford to be completely divorced from business realities is over, and you will need to be clear on what revenue stream or streams can be generated from your data projects.

Always ask:

• Does the data have commercial value?
• Are there obvious sponsors for the data?
• Could you build an app with the data that might be a premium product?

Start small before taking on big data – Connected China is a massive project beyond the scope of most organisations, but it still illustrates important concepts about context and the importance of structuring data that can be used on much smaller projects. Structured data can be used to add context to local stories. Also, it’s important to realise that you can build up databases over time rather than thinking that you have to build big right away. There are a number of ways that you can begin to add data to everyday stories which will help you understand how your audience responds to such projects and also guide you in deciding which data projects to pursue.

Measure and learn – Monitoring how your audiences respond is key. Many news organisations find that data projects are extremely popular with audiences. The Texas Tribune in the US has found that its focus on data is appreciated by its readers. A year after it launched in 2009, the Columbia Journalism Review noted that its dozens of interactive databases accounted for three times more traffic than its stories.

When the Guardian launched its famous Data blog, editor Simon Rogers thought it would have a very small, niche following of “techies and developers”, but he has said:

In fact it has become one of the most popular on guardian.co.uk and has developed a very mainstream audience.

This isn’t to say that your audiences will embrace data in quite that way, but you should measure how they engage with your data projects. Do a few small ones and use your analytics to find out which ones work and try to figure out why. This will help you to decide the next step on your data journalism development.

Data journalism is opening up opportunities to tell better stories, to engage audiences more deeply and also to improve commercial results for news groups who understand how to build in commercial goals to their product development. News groups that are developing data journalism projects are finding success in telling stories and also in increasing their audiences.

News organisations could learn a few things from Barack Obama. The Obama campaign built a huge data-driven analytical system to bring together vast amounts of data. “We are going to measure every single thing in this campaign,” Obama campaign manager Jim Messina told Time. This allowed the campaign to precisely target potential supporters, register them to vote and make sure they voted on election day. These innovations were one of the reasons that Obama was able to win the close election.

Some forward-looking news organisations are already using similar data-driven techniques to improve the performance of their content, their mobile apps and their business. Being able to effectively measure not only the size of their audience but also how the audience responds to and engages with their content is helping news outlets to clearly know whether they are meeting their editorial and business goals.

In the battle for audiences’ time and attention – and advertisers’ money – news organisations’ ability to compete with digital competitors will be determined by how effectively they are able to use audience data.

Making audience data a priority

For digital news organisations such as PolicyMic, keeping close track of their web analytics allows them to monitor the pulse of their website.

All nine employees of the digital news startup not only have access to the site analytics, but in addition, “every editor wakes up to an automated e-mail with a complete rundown of our analytics from the day before. We make sure the daily analytics are simple, accessible, and actionable,” said PolicyMic co-founder Chris Altchek. Media Development Loan Fund has invested in PolicyMic through its Digital News Ventures fund.

Altchek sees a clear benefit from their focus on data. “Analytics helps us drive growth month over month. We’ve grown from 100,000 unique visitors per month to 5 m in 10 months this year by learning every day from our analytics,” he said.

What is the job of the content?

These kind of results are one of the reasons that news organisations are increasingly using audience data to improve their editorial and financial performance.

Analytics software and services can allow you to measure a myriad of performance indicators about the performance of your content including raw audience numbers, location of your audience, basic demographic about your audience and referrals. Referrals are how your audience is finding your content, through search, social media or by directly visiting your site. Your first step will be to decide what to measure.

Barack Obama’s campaign wanted to measure everything, but for media and news organisations, while it’s possible to measure everything, it’s not practical. “You have to do prioritisation, you can’t track 30 to 40 metrics,” says Karl Schneider, the editorial development director at business-to-business publisher Reed Business Information.

To prioritise which of the myriad of metrics you want to focus on, you’ll need to determine your goals and which measurement techniques are best suited for each goal.

This is a radical shift in the way that editorial organisations have traditionally used metrics, and Schneider explained why and how editorial organisations are making this shift from simply measuring the size of the audience to measuring how audiences are responding to content.

In the past, they used standard industry metrics: page views, unique users and some engagement metrics, such as time on page and social media shares. He conceded that not a lot of thought had gone into why they were using these metrics. “We had been thinking for a while that the things we measured weren’t very well tied to the commercial success of the business,” Schneider said.

He compared the traditional thinking about metrics to a company designing a vehicle. To understand if they had a good design, they might choose a few criteria. An engineer might choose top speed, but first you need to figure out what kind of car you’re designing. If the car is a sports car, top speed might be a good criterion, but if we’re trying to build a delivery van, top speed wouldn’t be an appropriate metric for success. The first question that the company would need to answer would be what the vehicle would be used for. What is its job?

Similarly, before even thinking about metrics, RBI started to think of the role of the content. What were they trying to achieve with the content, both for audiences and also for RBI as a business? Once they had answered the question of what job the content was doing, what their goal was, “often, the appropriate metric would fall out straightaway, would be obvious,” Schneider said.

It got him thinking about why editorial organisations don’t ask what the role or job of the content is, and he said:

Traditionally the job is always the same, because the job is the same we never have to ask that question. Traditionally, the job is always to deliver the audience. … In the past, our monetisiation was relatively straightforward. You would either charge for it … or we sold advertising against it.

The package was undifferentiated and the advertising was relatively undifferentiated. In print, you couldn’t target advertising. For both of those business models, the job of the content was to deliver the audience. Measuring the issues sold, readership, was appropriate for both of those business models.

“But we took that thinking online without really thinking that especially, more recently, the way that we monetise the content is much more sophisticated,” he added. Now, in digital, monetising the content might include getting the reader to sign up for recruitment advertising or getting a reader to register for a paid event. They might also capture details about the reader that they can sell on for lead generation. He added:

Getting (people) to turn up isn’t good enough in those business models. If someone turns up and reads an article, watches a video, if that is all they do, we typically don’t make money. Obviously, we still have advertising, but even there, it’s more sophisticated. We are starting to want to target advertising. If they turn up and tell us something about themselves so we can target advertising, that is a more valuable thing for them to do than if they turn up and just disappear again.

Choosing the right metric

To help them understand the job of the content, what they want to achieve, RBI is now piloting an approach that involves meeting with managers responsible for the revenue streams that support the content. They discuss what the content does or could do to add to the revenue streams.

For example, for an events business, the obvious thing that content can do is to attract people who might sign up for that event to a page where they can buy tickets. Many of the revenue streams at RBI work on this model of using content to increase conversion to either a paid event or paid content. Measuring how effectively content drives these conversions is one key metric.

They also look at how content can build a brand relationship with the audience. “In most of our markets, the main experience our audience has of the brand is the content,” he said.

Brand relationship is harder to measure than conversion. They are currently developing key performance indicators (KPIs), and “you try to look for indicators that are highly correlated with the change of perception that you are looking for,” Schneider said. If someone values content enough to share it with their friends and professional contacts, RBI is finding that this is a better measure of brand relationship than simple pageviews.

As a business, they have started to change how they perceive success. For instance, when comparing the performance of content, they might have two pieces of content. The first one had twice as many pageviews, but the second one delivered twice as many conversions for people who signed up for a paid-for event.

“In the old way of thinking, we might think that the first one was a better piece of content, but in maybe today, maybe we would say the second one is a better piece of content because it delivered more people to a conversion we care about it because it delivered revenue to us,” he said. The first piece of content might have delivered more people but it wasn’t our core audience, he added.

They have had to prioritise because in some markets, they might have six or seven revenue streams with four or five different roles of content for each. What are the key revenue streams? What are the key roles to make a difference for those revenue streams?

Informed innovation

RBI and other news organisations are using this audience data to help them set priorities, improve their site and app design, and even help them improve the performance of their content. Using audience data involves a process of continual optimisation, using the metrics to review and revise your approach.

Before joining RBI, Schneider was a research physicist, and he brings a decidedly scientific approach to how he is using audience data. He views the KPIs they have chosen as part of a hypothesis to be tested.

After six months, they will review the KPIs. If all of the KPIs are up and the revenue is up, they will know that they have chosen the right metrics, but if not, they will have to review what they are measuring.

At PolicyMic, they using their analytics to constantly review and improve their editorial performance. Altchek said:

After every big event, we create a report and blueprint based on analytics, so when a similar event happens again, we have a solid editorial plan that improves based on past performance. We therefore assign stories, change headlines, and re-write introductory paragraphs to fit trends we’ve discovered studying past analytics.

At the Daily Telegraph in the UK, data is helping to refine their mobile products. In a piece for the International News Marketing Association, The Telegraph mobile director, Mark Challinor, said:

Instead of trial and error, we guide our app development and mobile offering with key metrics, as agencies and advertisers are demanding a more targeted mobile audience, and readers have come to expect mobile news access in differing ways.

They used data from a six-month free trial period of their iPad app to improve the app and also to test advertising formats. Monitoring how users navigated the app helped them deliver an improved experience when they launched the paid and bundled app after the free trial. Like many other news organisations, they noticed a spike in evening usage of the app. “With that in mind, we introduced a night-reading mode, turning the iPad into a true lean-back device users read in bed,” Challinor said.

They also found that there were more users over 50 than in any other age group, which led them to focus web and smartphone content and campaigns on younger readers than the on the iPad.

The Telegraph also shared its analytics insights with advertisers. “The data obtained from analytics let advertisers create more effective ads and help brands make real and engaging connections with our readers,” he said.

Making data manageable

To help teams manage the data, RBI has also created dashboards to help them easily monitor the key metrics that they have determined they want to track. They have created one top level dashboard using their metrics software, Site Catalyst from Adobe. The top level dashboard provides the key raw metrics about whatever KPI they are trying to measure. They track it over time to see if the conversion rate this month is better than the conversion rate last month.

A second dashboard looks at the content on a much more granular level. It looks at both categories of content and even individual pieces of content that performed well or poorly in terms of the KPIs. “That is the most important dashboard in terms of driving up performance,” Schneider said. Content teams can use this dashboard to find characteristics in common for high-performing content.

This allows them to test content and make informed decisions about what content to focus their time and resources on and what content they choose not to do.

Site Catalyst is used by many large media organisations, but Schneider said that they aren’t using all of the capabilities of the service yet. News organisations could do much of what RBI is doing in its pilot using simpler statistics packages, such as Google Analytics, he said.

How have editorial teams responded?

This revolution in audience data is providing new clarity to editorial organisations in their digital decision making. However, even for advocates such as Schneider, he worried how editorial teams might respond.

“That was my biggest concern how we would sell this to the content teams and how it went down. My fear was that it would be really negative. My hope is that it would be neutral,” he said.

His experience with creative staff including journalists is that they hate vague success criteria but they also hate micromanagement. RBI isn’t telling journalists and editors that they must create a certain type of content. Rather they are setting success criteria and giving editorial staff the freedom to decide how they achieve those goals.

The analytics are leading them to experiment with content types that they haven’t done much of in the past such as data visualisations. One of RBI’s revenue streams is that it has a robust data business, and these visualisations are being used to attract new subscribers to these data products.*

Although the rationality of data might seem cool, or even cold, in contrast to the passion that most journalists feel for their work, we all want to succeed, and news organisations and other publishers are finding that audience insights are helping their content to reach more people and their businesses perform better. These data-driven strategies can be an important competitive advantage for news organisations as they make the digital transition.

* Disclosure: Before joining MDLF, I worked with RBI to train their staff in data journalism techniques including how to create charts and graphs and other visualisations.