About authors:

Clémence Petit-Perrot is the Children’s Radio Foundation’s Learning and Innovation Director. She oversees the development all new initiatives within the organisation. Part of her portfolio includes piloting technological solutions like WhatsApp to increase listeners engagement and measure the radio shows’ impact. Before joining CRF, she was the Southern Africa correspondent for Radio France Internationale (RFI). She also worked for the South African production company DOXA, producing social documentary films and leading a digitisation project of anti-Apartheid audiovisual archives.

Linda Daniels is a journalist by training and has worked in print, digital and broadcast media. She has reported on a range of issues, which include climate change, Intellectual Property and South African politics. Her work has appeared in local and international publications. Between 2013 and 2018, she worked at the Children’s Radio Foundation as the Radio Capacity Building Associate and managed the WhatsApp Integration project.

Please download and share the guide. We would love to hear from you – send any comments or suggestions to us at mas@mdif.org.

[pdf-embedder url=”https://www.kbridge.org/wp-content/uploads/2019/04/WhatsApp-for-Radio-Toolkit.pdf” title=”Guide #6: WhatsApp for Radio Toolkit by Clémence Petit-Perrot and Linda Daniels”]

This explains why in many countries, digital journalistic enterprises launched when social media was already mature rapidly run ahead of legacy newspapers, even those that made big cash injections into their digital operations. Of course, successful digital media must produce good journalism, but their true secret is creating a conversation around it.  They are open to their public and easily let them know who they are. In one example in Eastern Europe, despite the traditional formality of many East European media, a new digital outlet had no problem sending a video to their audience of the editor sitting in her kitchen apologising saying she was sorry for a boring newsletter they had sent. In Latin America, new digital outlets have also successfully broken with the formal, ceremonial tone so characteristic of serious media there. Reporters tell the stories behind their best stories; introduce themselves with slang, as if to friends; constantly correct their mistakes; and when they have a conflict of interest about an issue, are candid about it. They let their public know that the media is only human.

These journalists offer their audiences a new, more transparent, and freer horizontal culture. However, sometimes, even those passionate journalists forget it takes two to tango. They want to tell their readers a lot about themselves, but do not care to listen.  Recently I saw journalists from Central America and the Middle East marvel at how little they knew about their readers after taking an intensive “read your analytics” course.  They said that knowing their Google stats and monitoring their following on social media makes a big difference to knowing how their stories are received.

But they, along with other media, including the largest US newspapers, have been realizing that tracking graphs and trends is not the same as talking with your public. (“We can count the world’s best-informed and most influential people among our readers”, said the New York Time’s 2014 innovation report. “Yet we haven’t cracked the code for engaging with them in a way that makes our report richer”).

Media in digital era know now they should invite readers to discover the world with them: open doors so that their audience can check the public discourse with them (like many of the 100+ fact-checking outlets around the globe are doing today); know the experts among their readers so that they bring insight into their news; call upon those with a generous heart to help them go through the millions of documents they just got from a source and build a database; ask the furious and the bullies, who write insults under their articles, where does their anger come from and, listen; open a space to let readers decide which reportage they should do; invite first-hand witnesses to document a problem they are investigating… the list of how much they can enrich their journalism is endless.

For those journalists with blinders who believe that engagement with audience is the business of marketers, Monica Guzman in her great guide about audience engagement  published this year with the American Press Institute proves them wrong. It is not about delivering a product, it is about making sure your readers know you respect and value them, she says, “showing them that together, they have important things to teach each other.”

Around the world independent journalism becomes stronger on the shoulders of the communities they serve.  Eldiario in Spain and Mada Masr in Egypt define themselves as a culture, a way of being, a clique, an idea of the society they want to be. And they build this dream together with a community that feels invited to be part of their world, well-treated, partaker, equal, like in any really good conversation. The “talky” public is here to stay and those journalists who fail to see their luck in this new era are likely to fade away.

This story originally appeared in https://medium.com/@OSFJournalism of the Open Society Foundation’s Program on Independent Journalism and is reprinted with permission.

You can subscribe to the newsletter here.

Bilton quotes Andrew Jack, head of aggregation and curated content at The Financial Times, which recently launched its latest daily newsletter, First FT: “There’s still a very strong appetite for email. People want something that’s always there and easy to access. It also gives us a way to push news and comments to readers in a new form.”

Benjamin Mullin of Poynter.org explains how effective email newsletters are proving for BuzzFeed: “so far in 2014 newsletter traffic to BuzzFeed is up 700 percent over 2013. And newsletters … are now in the ‘top five or six’ biggest drivers of traffic, behind sites including Facebook, Pinterest, Google and Twitter.” BuzzFeed now has 14 newsletters targeted at different audience segments.

While FirstFT aggregates articles from a variety of sources, not everyone agrees. Bilton gives an overview of four of the leading media newsletters.

Quartz: Daily Brief

• “We wanted the newsletter to feel like a personal email from a savvy, intelligent, well-connected person.”
• Aggregated sources
• 95,000 subscribers
• Sponsored and native ads

Vox: Sentences

• “Vox Sentences is the fastest way to get caught up on the most important news stories after a long day”
• Delivery: 8pm.
• Aggregated sources
• Banner ads

The Financial Times: FirstFT

• Morning briefing aimed at global decision-makers
• 50% FT sources, 50% external
• Some ads but goal is to drive FT subscriptions by building a relationship.

 Time.com: The Brief

• A daily snapshot of Time’s content to build the audience’s habit of using Time.
• Links to 12 Time stories, no aggregation
• 650,000 subscribers, with a 40 percent open rate.
• Carries ads and pushes readers to subscribe to its magazine.

The Knight News Challenge has been supporting news media innovation since 2007, funding more than a hundred projects to the tune of $37 million. Some of its winners have developed into successful businesses (though not necessarily doing what they originally planned), while others have sunk without a trace.

The report looks at 28 projects such as FrontlineSMS, iWitness, and Zeega and picks out important lessons that contribute to successful media innovation.

Target users with “a need you can feel”

Successful projects scale because they have identified a core audience and proven need, whereas others fail “because they developed a tool without first identifying target users”.

Get the interface right

“An intuitive user interface is vital for attracting and retaining users.” But don’t underestimate the time and expense involved in developing what appears to be a simple design.

Successful projects may appeal to a different audience than first imagined…

Tools developed to help media outlets with tasks such as visualising data have struggled to get traction in the newsroom but have found success in other industries. “Small budgets in journalism and a lack of technical understanding among journalists can inhibit adoption.”

…Anticipate resistance to innovation

Successful startups realise that their innovation may cause disruption and meet institutional resistance. Successful innovators anticipate resistance and plan ahead for it, for example by identifying a wider potential audience beyond the initial target.

Identify staffing needs early

Many startups rely on a mix of full-time paid staff and a community of users and evangelists to develop and promote services and tools. Identify which parts of your project need paid, full-time staff and those that can be carried out by volunteers, and allocate resources accordingly. The contribution of unpaid supporters can be undermined by insufficient core staffing.

As is usually the case though, the most effective and helpful tool – or rather a technique from those mentioned above – is the one that also costs the least. Have you done usability testing yet? No? In that case, if you’re going to do one single thing, throw away all the other tools and techniques and start doing usability tests – they will take your website the farthest.

In this article, we’ll take a look at what you need to do and what you should know before you start testing.

What we’ll need

A usability test is nothing more than tracking a user during the time he or she is using your site. Really, that’s it. No measuring of eye movement, level of stress or anything else that needs hi-tech tracking – just a simple recorded (shot of the computer screen they’re sitting behind and their voice) dialogue between you and them. And of course the tasks you request them to do on your website.

So, no measurable indicators – CTR, time spent on site, etc.? Precisely! Questionnaires or Google Analytics will provide you with lots of data, but it doesn’t get much better than seeing the person in action with the possibility of asking what they think. I guarantee that after a usability test, you’ll know right away what needs to be done even without knowing exact percentages or having data to feed into a graph.

Like with any cooking recipe, we too will start with the ingredients we’ll need. There isn’t much of it, yet it’s still a good thing to specify them.

• A quiet room – it may sound strange but if there is something that can ruin the result of a usability test, it’s having someone disturb you. Are you in an open space office? If so, forget right away that you would do a usability test at your work station. An ideal place for a usability test is a small room, two chairs and a desk – no disturbing elements. Let people know not to disturb you during the time of the test. Turn off your phone.

• A standard computer and browser – although the object of testing during the usability test will be your website, people you’ll be working with will have a feeling you are testing them. Despite being told otherwise, they will not get rid of this feeling. Don’t add to their stress by making them learn to use your gaming mouse or keyboard with different characters. Use the most standard computer most people are used to. In the author’s country, this would be a laptop with Windows, a plain mouse and Chrome browser. Have more browsers prepared and let the participant choose which to use. The more they feel at ease and as if working with their own computer, the better for them and you too.

• Microphone and a program for recording your computer screen – During the test, you will record what the participant is saying as well as the browser window they’ll be surfing in. This is important because you won’t have time to take notes and will not remember all the issues that will come up. Laptops, as well as desktops, often have built-in microphones; however, it has proven useful to the author of this text to have an external microphone. A simple cheap mic for Skyping will do very well.

Selection of a recording program may be something of an obstacle as there aren’t many good quality solutions. For us, a good choice proved to be Camtasia Studio. It can record the screen as well as sound and later allows the recording to be cut, highlight parts of the video such as a mouse click, rolling over an element, etc. The program is not exactly cheap but has a 30-day trial version, so you can start with tests for free and later decide whether investing in the software is worth it.

What will we test?

Question “What will we test?” may sound ridiculous to you. You probably already know about the weak spots on your website – perhaps you were notified by email from users, or you’ve seen something in the statistics. Maybe you’ve asked yourself already why no one clicks on the Editorial Team link at the bottom of the page, while at the same time you get asked by users whether you’ve considered having one.

If you haven’t done usability tests before, I would guess the first ones you do will be of a very general character. You will want to look at your website as it is and its key functions, and find out whether users know how to use them or if they have a problem using them, and last but not least, how you could improve and simplify the way your website is used.

Later, you will probably concentrate more on in-depth issues and usability tests will focus on only one of the sections or just an individual function of your website. Either way, your task now is to transform all intentions into assignments for the participants in your test. How do you go about it?

Close your eyes and imagine all the key functions that make up your website. You can forget about the details for now – the most important elements will do for now. Imagine an average person and what he or she actually does on your website. Usually arrives, looks at the homepage, clicks on an article. Then what? Perhaps he wants to enter the discussion? Or maybe he wants to sign into your system email and read messages? Obviously, it’s different for every website but you should always keep in mind the most frequent and concrete scenarios of your website’s visitors.

Once you have them, you can start turning these scenarios into a story with assignments, ideally with some update theme. Set up concrete and fun assignments.

For example: “You’ve seen on TV that the government wants to increase taxes for people with higher income. You missed some parts of this information, therefore you decided to visit our website and confirm the missing details. Where would you look for this information on our website? Find out the percentage of this planned tax increase and the date the tax law should become effective”.

Notice that I haven’t used in this assignment the name of the section (go to section Economy, in the left-side menu click item xxx). This is exactly the method you should use too, as normally a user doesn’t know where things are. Since nobody tells users on the homepage where to find the information they are looking for, they have to find it on their own. Effectively, the user will now test your site for you and whether you have developed it well. Following the user’s path to finding the right link will show you how to make your website better.

Prepare about ten assignments in a similar manner. For each of them, try to create a specific situation and make it interesting or fun so as not to bore the participant. Be sure to make it clear exactly when the given assignment can be considered finished.

Are you testing one specific part of your website? It’s good to have the first two assignments of a different kind – “to trick” a little, but also for loosening up the user. When testing a specific thing, I personally try to hide it by breaking it into a number of different assignments so I don’t give away my intention. Ideally, at the end of the test, the participant shouldn’t really know what the main purpose of the whole test was.

Ready for some inspiration? A complete usability test can be found here: http://www.youtube.com/watch?v=QckIzHC99Xc

Selecting participants

How many people need to take the test in order to reach the optimal results? It may come as a shock to you, but 6-7 people will be enough.

Are you wondering why so few are needed when for any A/B testing questionnaire you need to have at least a few hundred participants to consider the data reliable? The answer is that when it comes to usability issues, we are all very much alike. You will find this out as soon as you start testing: the second participant will have very similar problems with your website as the first one, and this will repeat until you get to the last one. Using, say, 20 people instead of ten of course means finding slightly more issues on your website, but I guarantee that testing 6-7 people is absolutely sufficient for finding the most significant ones.

Which people to choose? You may be tempted to strictly adhere to demographic distribution, so here is my tip – don’t take it too seriously. (In the end, it will be hard to stick to a demographic distribution of users in a sample of 6-7 people.) Include men and women, younger and older, but the best thing you can do is to call on users that could potentially have a problem with your site.  If you call on a programmer to take part, he’ll probably be able to manage almost anything, but how about a user who isn’t so good with computers in the first place? You are better off choosing participants that will show you as many deficiencies as possible.

An important rule: never invite someone who is a “heavy user” of your site – a member of your editorial team for example, if you have a news website. Of course, there are exceptions when this rule does not apply (e.g. intranet testing), but for the majority of websites, inviting outside people – strangers not familiar with your website – is crucial. Ideally, of course, you would call on people that do not know your site at all. This, however, cannot be done in all cases – if your website targets brokers, an ordinary person won’t be able to help you much.

Another thing to consider is the question of payment for taking the test. You’ll find out in US literature that you should. My personal experience is to invite people without the promise of remuneration but rewarding them at the end anyhow (usually not with money though, but rather in the form of a small gift such as a book).

You may now have a good idea of who to invite. If not, here are a few tips from practice:

• Create a site with a simple questionnaire for people who would like to help with website development and link to it from your site. In a few hours, you are sure to have perhaps hundreds of contacts for willing people.

• If there is a place where users come, e.g. the reception area of a newspaper office where they renew subscriptions, position yourself there and ask directly whether they have time to take the usability test. Another way to do this is for the person on the desk in reception to ask and take the interested users directly to your office.

• Place an ad in the classifieds or post it on your Facebook page.

If you don’t want to do any of the above? We all have friends and acquaintances who don’t even know where we work or have never visited our websites – use them. Just one rule to keep in mind: never, never invite the same person for tests twice, especially if it’s the same test!

Carrying out a usability test

You have a quiet room, you have a computer ready (you’ve tested the recording and mic functionality), you have people coming and you have the assignments ready for them. What now? It’s time to carry out the usability test.

At the beginning of the test, you have to tell people what awaits them. In order not to influence the results, it’s ideal to tell everyone the same thing – that is, read the same text to all of them. You don’t have to come up with the wording on your own; usability expert Steve Krug has done it for you already, so don’t worry. You can download the English version of it here.

The text describes what will happen during the usability test and also includes a printable form of agreement for making sound and video recording. So now you can smoothly begin your assignments.

Now comes the most important part. You could have done everything perfectly up to now, but this is the point when everything can go wrong – your communication during the test. The following are the three most fatal mistakes. Avoiding them and sticking to the following method can make usability testing a brilliant tool.

• Do not help. Do you remember how you deliberately didn’t include specific instructions about what to do? This is exactly what you shouldn’t do either while the test is in process. Let’s say a participant has a problem with completing your assignment (can’t find a link to a page or has forgotten her password). Do not help her under any circumstances. You can try a formulation such as, “Try thinking about it some more”, “What would you do if I were not here?”, or “Unfortunately I can’t help you but as soon as the test is over, I’ll answer any of your questions”. If the participant cannot complete the assignment for a long time, end it. It is difficult not to help a person who is obviously in trouble. I personally managed to “fail” a few times when I administered my first usability test. While it is difficult, it is very important because if you help the participant, you may as well throw the results away because you can’t be sure that you didn’t bring the user somewhere she wouldn’t otherwise have got to on her own. There of course are situations when you can help – for example if a strange window, not relevant to your website, pops up, or the mouse is disconnected, etc. – but don’t ever say, “And why don’t you try clicking here?”

• Ask what they’re thinking about – keep on prompting the participant to say what he’s thinking about, to verbalize what he’s trying to do as much as possible. The instruction to do so was already included in the introductory text you read out at the beginning, however, some participants will tend to forget – you should do all you can to make sure there isn’t silence in the room and the participant speaks as much as possible. That is the key to improving your website – every one of the participant’s thoughts will move you forward.

• Praise and eat humble pie – many people you invite will say, “I’m not smart enough for this” and when they can’t do something, they will repeat the phrase. Participants will constantly feel as if you are testing their abilities. Use every opportunity to disagree with them: “I’m not sure where to click?” “Thank you – you have just found a huge shortcoming in our website. Actually, you’re doing it right, and it’s us who developed the website who got it wrong because you can’t find what you’re looking for.” After the test is over, tell them again how much they have helped you. You can even show them the right solution to their task.

After the test

The first thing that shows you’ve done the usability test right is that the results are absolutely horrible. Users had problems finishing your tasks, you’ve noticed mistakes you made in planning the site, users do things in a very different way than you thought when building the site. These are the results you’re looking for. If you, the website developer, feel miserable, be assured that this is exactly the kind of feeling you needed to reach.

In order to transform the usability test results into a better website, do the following:

• Share the results with everyone – Do not keep, under any circumstance, the test results just to yourself. Let as many colleagues as possible see the video recorded during the test. Definitely show it to designers, project managers and programmers. Don’t be shy to show it to other people who worked on the development of the site. Practically anyone from the team can learn something from a usability test. Before you show them the video, give them a pen and paper and ask them to take notes with ideas on what to change on the web in order to prevent some of the problems the next time around. I’ll bet everyone will have a paper full of various thoughts. Are you all too busy to watch a 5- or 6-hour long recording? Cut the usability test video – leave out the parts where participants succeed in their tasks and pick out the “cherries on the top”.

• Have a discussion and make a list of changes. After watching the video, compare your lists and write the problems on a big board. A huge discussion is surely to arise. A long list will be created. Set a priority for each item – mark the issues according to how big they are.

You will notice as you go along how the view of the web changes in the team. At the beginning of the exercise, someone may say something like, “Are these users completely stupid?” And then, when they see that every test participant had a problem with the same thing, they will go quiet. You will notice the effect of a usability test on the work of the team in the future. A programmer will try to simplify everything as much as possible right in the beginning. A project manager may want to run a usability test during the design stage. The owner will approve a budget for usability tests.

The possibility of seeing users utilizing your site in real time is something that will completely change the way you work.

How to solve issues discovered

One of the things that happens to you after a usability test is that you suddenly get a feeling that you completely need to change your website. You’ll probably feel that it’s badly designed; programmers may arrive with an idea to re-program the whole site. One of the worst things you could do, though, would be succumbing to these temptations.

Any usability issue found can be resolved in two ways: the complex way (total redesign) or the dilettante/superficial way (do the absolute minimum in order to remove the problem). Based on a number of years of experience, I recommend choosing the second option.

If you don’t believe me, take a look at the following list:

Dilettante solution vs. complex solution

• Much less people will be affected by the issue / No one or almost no one will be affected by the error
• Easily implemented / Requires much effort to implement
• Finished in a few minutes, a day the most / Can take a month or two, and may actually never be finished

Complex solutions seem to be clean and thorough. In reality though, it means you are starting from zero. In the end, the fact that you have done it over again doesn’t mean that you have not made the mistakes again or haven’t created new ones. On the contrary, if you redo something completely, you will end up facing a dilemma at the end what to do with the new issues.

Usability expert Steve Krug introduced a rule he calls “Do as little as possible”. Always look for the fastest solution to fix an issue. Only when that doesn’t work, use something more complex.

Here is an overview of Krug’s points:

Do not redesign, tweak (Krug’s No. 1 rule of successful mistake correction)

• Everybody hates redesigns, so if you suggest a wholly new solution, they will complain anyway.
• You may solve your problem with a redesign, however, it’s almost certain you will make at least five other, perhaps more serious, mistakes.
• Do the smallest changes possible – tweak until the problem is solved. If that doesn’t help, revert to a more serious solution.
• It often means that you want to increase the font, change the background color of a message or something along those lines.

Do not add, removing something is much better (Krug’s No. 2 rule of successful mistake correction)

• People usually feel that adding something will work better, let’s say a new notice or phrase. Often, it’s much better to remove something. Most mistakes are actually caused by organization on the site being so complicated that it becomes chaotic. This is a very frequent reason for mistakes.
• Always consider the list of issues in relation to effectiveness. If there are small issues which affect only a few users and would take too long to fix, don’t bother with them. There are probably ten larger issues that can be addressed quickly. Focus on these first and then on the others.

How do you find out whether you have improved the product? Simple, run the same usability test again … with different participants, of course.

Now you know everything important about usability tests. If you’d like to know more, I recommend two of Steve Krug’s books: Don’t make me think and Rocket surgery made easy. They are short, funny and will tell you everything you need to know.

So what is A/B testing? Simply said, A/B testing is a method which allows you to measure the effect of changes to your website on the visitors without actually making them.

Imagine as an example a web page with an article to which you make a small change – let’s say you move a picture from its current place in the right column into the center. Now comes the trick: you display the new layout version to a very small portion of your visitors. What will happen? Will visitors stay on the page for a longer or shorter time? Will a smaller or a larger percentage of readers finish reading the article?

After a few thousand views of both versions, you will be able to neatly and relatively exactly compare and determine whether the change is – or is not – an improvement. If indeed it is, you know what to do – change the picture position for everyone. If not, no problem – you can test some more.

You can test any changes this way – always on a small percentage of visitors and find out whether something works even before widely implementing it. And what is important, you are not limited to testing just one change but for example five different options. Using the same example as above, you can test positioning the photo in various places (left column, right column, top, center…) and by trial, find out where its positioning works the best.

Simply speaking, A/B testing gives you a luxury that we would probably welcome in all parts of life – the possibility of not having to make a decision every time we want to do something and then wait and worry whether we have made the right decision.

Instead, thanks to A/B testing, you have the option to choose the best solution from the multiple possibilities before making it available to all your readers.

Why should we run A/B tests?

A/B testing is one of the most practical tools for improving your website, primarily for the following reasons:

• Low cost, quick start: one of the best things about A/B testing is that, unlike using questionnaires or usability tests, you can start immediately. And if you decide to use one of the free services that are available, you won’t even incur any costs. No complicated searches for participants or long preparation times. One test idea and an observation of certain rules discussed below and you can be on your way. You don’t have to hire an expert. You can easily do it just with this article.

• The right sample: With a survey or questionnaire, you will always face questions such as “do we have the right sample of people?” or whether the group of people surveyed had been “representative”. There are no such questions in A/B testing – you know 100 percent that you have the right sample. It’s the people that come to your website, exactly those that you are improving your site for. There is no need to look for anyone elsewhere.

• Very practical: Standard questionnaire surveys often bring theoretical findings, such as people are more interested in sports news or perhaps that they would welcome more videos in articles. But what should an ideal piece of sports news look like? Where exactly should the video be in the article? A/B testing, together with usability tests, offer very practical findings, which are – as a bonus – all immediately applicable. They not only provide you with information about what to do but also how exactly to do it.

Getting started with A/B testing

So how do we do A/B testing?

1. Hypothesis. At the beginning of each test, having a “hypothesis”, i.e. a question you want answered by the test, is very important. For example – “Would visitors click more on an article if it had a larger headline?” Any new function you would like to implement on the website is also a hypothesis – e.g. will the function bring a better reaction from the visitors? You probably already have a number of ideas as to what to test. You can also find some tips below.
2. Alternate versions. The second step is the proposal of changes with which we will verify the hypothesis – let’s say the creation of alternative versions of the original website with new functionality or design. At this point, it is important not to limit yourself to just one alternative to your current site. You can come up with as many versions as you would like. This way, you can be sure that you have not missed the best solution possible. For example, if you are testing a larger headline for your article, why not include various fonts, colors and/or size? The test will let you know the best solution.
3. Test variables. Often, defining the test variables is a much underestimated step. A test variable is anything you can measure and which allows you to objectively determine which page is better. It could be the number of clicks on a certain link, the time spent on a page or even whether a visitor had completed a certain operation (confirmation of a subscription payment). People frequently measure the number of clicks, but is that always the best? Always consider carefully what to measure and test more variables. That way you can see whether your new version won’t do more harm than good (e.g. the change forces more readers to click but they leave the page immediately because they hadn’t found what they were looking for).

Which tools to use for A/B testing?

Once you have done all three things listed above, only one thing remains – deploy the test. Let’s look closely at how to do it.

Ideally, you would use one of the ready-made solutions available. There are plenty of them and some are even for free. And it is certainly simpler than having the A/B testing tool custom programmed directly for your website – a solution which is possible, but definitely not simple.

When choosing, look mainly for simple implementation (do you have a programmer at hand or will the tests be administered by someone who knows programming?). Of course, the price is also crucial. Paid tools often offer free tutorials or assistance, so free does not always equal the best.

Here is a list of some of the best known A/B testing tools:

Google Analytics is among the best known A/B testing tools. One of its main advantages when compared to competing products is the fact that it’s free. And it works in a simple way: basically, you create and add all versions of your web page to the server – if the current page is at article.php, you create article1.php, article2.php, etc.

What follows is an easy step-by-step process of implementation:

If you do not have an account in Google Analytics, create one and after registering, click “Experiments” in the “Behavior” section of your website profile. Click on “Create experiment”.

Type in the URL of the website you want to test (in our example, it’s article.php) and select the measured variable – for example average visit duration, revenue, etc. Decide the percentage of your users you want to include in the experiment – determine the number according to the number of users that usually visit your site – if you have a lot of visitors, you can test the page on 1% of your visitors. If, however, you have few visitors, you could wait for the results from a 1% sample for a long time.

Then you just add the URLs of the changed pages to be tested. The last step is to add a special JavaScript code to each one. The code will determine which page is shown to which user.

And that’s all. You can find more details on how to do A/B testing with Google Analytics here.

Optimizely.com is one of the best known paid tools for A/B testing. Other paid tools mentioned below all work in a similar way. Compared to Google Analytics, they all have one significant advantage – you do not need a programmer to run the test – it’s enough to include in the page one very simple JavaScript code. Everything else can be “clicked” and selected without needing to know the source code. It’s almost as if you were editing text in Word or some other text editor – you can play around and change the text, the font size, anything. You choose the percentage of visitors to which you’ll display the tested version and select the measured variables. The tool saves the changes and, after starting the test, it will display your altered pages to random visitors.

Other popular A/B testing tools include Unbounce.com or VisualWebsiteOptimizer.com. Choose the one that suits you best. An overview of the best known tools is here.

Understanding the results

Most of the split testing tools offer results as you go along and even calculate which version has the highest chance of winning the test. At the same time, the tool helps to determine the right sample. At this point in the process, you don’t need to do anything else.

This, however, can become a double-edged sword. Despite the simplicity of A/B testing, it can be spoiled (just like anything in life). Watch out for these common mistakes:

• One variable, one hypothesis – one of the most common mistakes in A/B testing is that we want to test too many things at the same time. An example would be two completely different article visuals (each has a completely different layout, ordering, as well as elements). Remember: one A/B test, one hypothesis you are testing, one change. This is a way to avoid potential problems of not being able to understand what actually brought the change when reading the test result. Was it the background color? Or omission of a large ad space? Always change only one thing. Then test it.

• Attention – sample size too small. The second most common error is impatience. Frequently, one of the tested versions can have much better results very early on. Do not stop the test! Continue to the end of the test – it can often have very different results at the end than it seemed at the beginning. Contemporary A/B testing tools contain a function which automatically calculates the number of users you need to test in order to ensure the test is right. Here you can find a calculator which helps you to find out the approximate sample size even before running the test. If you are testing a large number of versions, it pays off to wait a little longer for results. Also, you may want to repeat the test one more time – test fewer versions the second time (e.g. original vs. winner and runner-up from the first test).

• Poorly chosen metrics. Even if you manage all of the above well, there is still a chance you are failing in the selection of the metrics with which you’ll be measuring the effect of your changes. Is the version that gets more clicks really better for you? Or are you more interested in whether visitors, after they click, subscribe to your publication? Number of clicks tends to be an alpha and omega for online media so pay attention to other criteria that may prove significant for you.

Testing tips

You surely have many ideas on what to test. A/B testing is practical any time you want to introduce something new and you discuss in your team how to proceed the best way. Don’t limit yourself to new functions you’d like to add to your site though. Think about things already present on your site, especially:

• Things that seem self-evident to you (if you hear someone say “people love xxx”, “the best way to do this is xxx” without having A/B test results to prove it, don’t just believe the statement – try it).

• Things on your site you think are done really well (this link has many clicks, the ad in the right corner works beautifully) – what if these things can be done even better? Don’t forget that every hundredth of a conversion counts!

• Things you think don’t matter (font color, font size, or moving an ad box 10 px more to the right). If you test a lot, you’ll find out that many of the results may not make much sense to you, yet they work. Do not rely on your rational criteria only and do not think that minute details don’t matter. They do – test everything.

Here are a few tips for tests you can run right now:

• A correct looking lead on the home page – is a smaller headline, full-width photo and no lead, or a larger headline, smaller photo and more text better? Or should it be altogether different? Come up with all different possibilities and find out what users prefer.

• Ad formats – you know that when there are too many, they don’t work; when there are only a few, conversion is higher but revenue smaller – so what is the right ratio? Try out different ad distribution on your page and select the best one.

• Homepage layout – there’s an everlasting fight within online media about the homepage. Each service, each journalist wants to see his or her article there, yet they cannot all be there. What is the right number of articles for visitors to still click? Which section is better suited for the right column and which will do better in the middle? The simplest way to find out is trying it.

• Test the wording of all buttons on your page. Compare “Enter the article discussion” to “Discuss the article now” for example. Test what works best and you may be surprised.

You can find lots of ideas and inspiration for A/B testing at www.abtests.com. It includes a number of case studies, so you can see the findings right away. Many of you may be surprised by them.

Still not enough? Suggested reading about A/B testing includes Always Be Testing: The Complete Guide to Google Website Optimizer. Although recommendations related to the implementation of Google Analytics experiments (the tool was formerly known as Website Optimizer) have become obsolete, the book offers hundreds of ideas and ways to improve your web site.
So what are you waiting for? Dive into A/B testing now!

Internet is often described as ‘the most measurable medium ever created’, thus real-time analytics is designed to monitor very precisely the current data on the behavior of visitors in order to help you optimize your website, content and overall online operations by delivering live and granular data. Through knowledge of what users are doing on your website right now, it is possible to understand the consequences of short-term changes and content updates, and respond to them immediately. It’s about moving with a greater speed towards previously unknown questions, defining new insights and reducing the time between what is going on on your website and your proper response or reaction within minutes.

Omnipresent generation of data in online environment is not just about transforming your content itself, it is, at the same time, also the transformation of workflow in news organizations and helping to develop effective content production strategies related to actual demand. New tools – like real-time tracking – are being deployed in newsrooms to provide insights around how your content is performing on site, in social networks, news aggregators and search engines. A new role in digital newsroom has been created: an audience editor is in charge to deliver and interpret real-time perceptiveness into the heart of the news workflow. This change in interaction with live audience improves how internal resources are deployed and what new perspective is explored in the newsroom.

Real-time analytics allows digital editors, content producers and analysts to easily inspect content viewing patterns on their own websites and to monitor visitor’s activity trends as it happens on site and react within minutes accordingly. The reports are updated continuously and each user’s activity is reported seconds after it occurs on your site. For example, you can track how many people are on your homepage right now, what is their geographic locations, the traffic sources that referred them to your site and which pages they’re interacting with after leaving your homepage. With real-time analytics, you can immediately and continuously monitor the effects that site changes and content updates have on your traffic, whether a new and changed content on your site is being viewed, what content is being consumed massively and monitor the immediate impact on traffic from a blog/social network posts or tweets. We can also see when you have stopped receiving visits from the social network, which helps you to realize when to reengage.

Better understanding of content consumption trends in progress leads your editors to more relevant, high-interest content creation. It helps to draw your attention to a particular information demand even on a kind of seemingly unimportant topics (you can also track and see, based on the traffic referrers, how people got to an article which led to rapidly growing reader’s interest trend). This way you may find that readers are significantly more attracted not by a present lead article and cover story on your homepage but greater information demand is focused to an article that is not listed on your homepage. Based on this finding, you can decide to place a link to the article on a location of your website through which people tend to visit your site. At the same time, you can instruct editors to supplement and enrich the article with related links, images, video or to add to the article a chart or infographics, and thus creating a compelling reader’s experience.

Based on continuous (a second-by-second basis) measurement of visitors’ behavior, it is easier to understand which various factors contribute to your visitors’ satisfaction (e.g. increase time spent on site), and how to convert a random visits to a loyal, returning visitor. More granular up-to-date data will generate fast yet informed decisions for your news desk. Soon, you will realize that just refining the title of an article can make a huge difference in terms of numbers of visitors accessing the article.

How to choose the right tool to fit my needs?

Before you start to compare the different real-time analytics tools, their features and the granularity of the reported data, think about what questions these tools should answer and solve for you. You must have a clear answer to questions like these: What criteria do you use to decide which articles to publish on your homepage? What are the types of things your editors disagree about internally and you have to decide intuitively? What everyday decisions have a major impact on the way your site looks and feels? According to what criteria do you control and assess the quality of content processing on your website? Do you know which time during the day is the most effective use of social networks? Who will be in charge of working with real-time analytics in the newsroom and what are his/her expectations? What would be needed to simplify and streamline the work?

Once you can answer these questions, you have got quite a clear understanding what features are a must for your daily use, so you can start comparing different real-time analytics tools.

Real-time analytics tools

There is a wide range of on-site measurement tools from different vendors that provide real-time analytics software and services. Data from your site is usually gathered via page tagging – an embedded tracking code, usually written in JavaScript or Java; increasingly Flash is used. (Of course, there are also other data collection methods, e.g. packet analyzer.)

Here you can find a very useful comparison chart of different available real-time analytics platforms: http://clicky.com/compare/. Another well worked-out comparison might be found in this Google Doc. Yet, other good up-to-date overviews of available web analytics tools are: Top 30 Web Analytics Tools and 53 Alternatives to Google Analytics.

Case studies

Reynolds Journalism Institute  hosted Journalytics Summit in September 2013. Following presentations are case studies given by representatives of real-time analytics company and their clients:

Outbrain (Visual Revenue) case study

Dennis R. Mortensen, Outbrain

Rhonda Prast, The Kansas City Star

Chartbeat case study

Joe Alicata, Chartbeat

Joy Mayer, Columbia Missourian

Parse.ly case study

Andrew Montalenti & Mike Sukmanowsky, Parse.ly
Adam Felder, Atlantic Media

Related article:

Real-time analytics can help you use scarce editorial resources more effectively

Every new website faces a daunting problem: “How do I get people to visit my site?” Most websites have only a handful of options. They can spend money to promote their brand and URL through advertising networks and other websites, bringing traffic to the site directly. They can invest in developing their content management system and their online reporting techniques to draw traffic from Google and other search engines, commonly called Search Engine Optimization (SEO). Or they can build a social media following on Twitter, Facebook or any of the dozens of other specialized social media platforms available today that refer traffic to the website. This last technique is often called Social Media Marketing. When planning a new website, each of these audience development techniques should be planned into all aspects of the site – content, software and hosting.

When Ujyaalo 90 Network in Nepal began planning their online presence in 2011, they faced a problem. Google, the dominant search engine in Nepal, was still relatively under-developed in the Nepali language, the primary language of Ujyaalo 90. They faced an environment in which depending on Google alone to drive traffic might not show immediate results. In addition, they faced a market that was moving online quickly, but predominantly through mobile phones. So after internal discussion and analysis, Ujyaalo Online developed a plan to build their Facebook fan base as their primary online marketing tool. Facebook gave them access to Nepal’s rapidly growing online audience and a familiar mobile platform to build on. SEO was also pursued but only as part of the development of the site’s content management system. The timeline below explains how Ujyaalo used Facebook to grow into one of Nepal’s largest social media and online media outlets.

The ad is a sign of the times for Nepal where owning a mobile phone has become common throughout the country, even on the slopes of Mt.  Everest.  The Nepal Telecommunications Authority reports that in August 2013 mobile phone penetration reached 72% of the population.  Mobile “smart phones” have also contributed to the rapid increase in Internet penetration in Nepal, growing from 19% in 2012 to almost 27% in August of 2013.

Newspaper publishers in Nepal have begun to experience the early stages of their audience’s shifting media habits.  Consequently, Nepal’s Centre for Investigative Journalism hosted “Doing Digital” a seminar for publishing executives trying to understand the challenges and opportunities presented by the digital transition.  MDIF presented “Journalism in the Digital World” a summary of the opportunities and the challenges presented by readers, listeners and viewers all merging to become the digital audience.  The digital transition is clearly a tale of the best of days and the worst of days, the good and the bad.

For publishers, the best of days is epitomized by the wealth of new tools and techniques.  Digital has made online story-telling a new narrative form combining the use of text narrative, audio, video, data and infographics.  Examples discussed included the New York Times’ Pulitzer Prize winning “Snowfall” as well as other examples from around the region.  The discussion highlighted two key points.  First, for these multimedia stories to be successful, journalism has to be combined with technology.  Second, the recognition that most publishers do not have the resources of the New York Times and that there are free or low cost tools like Timeline.js to help publishers tell multimedia stories.

Digital technology has also made data journalism a new opportunity for journalists and their audiences to find patterns and stories in the data.  Data journalism like Veja’s Rede de Escandalos provided tangible example of data journalism’s reporting power.   One of Brazil’s oldest news magazines, Veja used its own past reporting on scandals in Brazil to create a unique database of scandals, actors, government bodies involved in each scandal.

Finally, digital technology revolutionizes the way publishers, editors and reporters communicate with their audience.  Digital in many cases has turned audience communication into stories.  India’s website “I Paid A Bribe” website demonstrated how audience engagement and communication can create ongoing coverage of key themes, like corruption.

But publishers attending the seminar also focused on the business models needed to survive the transition. Unfortunately, the discussion of business models presented the challenge facing all digital publishers, where will online revenue come from.  The challenge is acute in countries like Nepal, where audience adoption of online has grown much faster than local advertisers’ transition to online.

During the seminar, MDIF discussed several examples of revenue streams that publishers should evaluate as they begin to actively publish online.  Since online advertising remains a very small revenue opportunity in Nepal, the discussion focused on enterprise and project social funding services, often called crowdfunding.  Crowdfunding services have expanded greatly with regional specialists, like Africa’s mobile fundraising platform M-changa or industry specialists like IndieVoices, which focuses on independent media and journalism projects. In addition to crowdfunding, syndication and content expense sharing partnerships were also discussed where two organizations partner to share the cost and potentially the revenue of a digital reporting project.  Finally, different subscription and paid content models were presented.  Though many of the Nepali publishers believed that the technology to easily collect revenue from their online audience was not yet available in Nepal.

Nepali media and media in any region undergoing a rapid digital transition face both opportunities and challenges.  Digital reporting and story telling tools have created a whole ways of communicating a story.  But, these new tools require training, technology and infrastructure support.  None of these are free.  Revenues from advertisers typically lag the audience’s move to digital platforms, creating a gap in digital’s ability to generate revenue.  This leaves publishers who move online with the challenge of how to generate some new revenue to support these new requirements.  CIJ Nepal’s seminar for regional publishers created a foundation for an active discussion and experimentation with both the opportunities and challenges facing Nepal’s traditional print media.

In late May 2013, the Financial Times launched a new, mobile-friendly news service called fastFT. It’s a mixture of traditional news wire and Twitter-like stream with an emphasis on speed and brevity, delivered both in a newsfeed on FT.com as well as on the standalone page. The article “6 product development lessons from how the Financial Times built fastFT” summarizes lessons to be learned from the product development.