2013-07-15

Does Anyone Want To Listen To The News Anymore?

We've been experimenting with adding audio layers to our news stories—think book-on-tape. Here's what we've learned about the web's least-used medium so far.



In June, we started embedding audio into a select group of Co.Labs articles as part of an experiment with the startup SpokenLayer. We theorized that adding audio to some articles would increase the time spent on site for those articles, as well as decrease the bounce rate as people spend time listening to articles rather than skimming through them. So far, we've added audio to four articles. Although it's a small number of posts, we received enough traffic on them to run some basic tests to see how effective having the audio has been. Moreover, as a publisher, we need to be able to move quickly to validate results so we can direct our limited resources efficiently. Fortunately, we bought a set number of SpokenLayer recordings, so we basically have to use them. That means we can use these results to refine, rather than stop or expand, the experiment. Here's what we found.

Some Basic Numbers

In the period from June 1 to today, the site received 482,681 views on articles (meaning non-homepage and taxonomy pages). Readers spent 8,839 total hours engaging with this content. Of those 482,681 views, 228,901 were bounces—that is, views where the visitor did not go on to visit another page on the site.

We ran four articles with SpokenLayer. Although we don't know the exact number of plays yet (I'll update this when we do), we do know thanks to Google Analytics In-Page that readers are clicking on the play button quite a bit when it's available. But it's actually not particularly important to our test. We're not only wondering if people stick around longer to listen to audio, we're interested in whether investing in audio sends a signal to a reader that the article is worth paying attention to.

To determine whether the differences in time spent and bounce rate were statistically significant, I compared them to the overall site numbers using a two-tailed t-test. Using a two-tailed test is important because I wanted to make sure that if it wasn't helping us, at least adding SpokenLayer wasn't hurting us. Here's what I found.

Why The Boom In Enterprise Tech Will Happen In New York

  • Views: 2863
  • Time Spent: 73.2 hours
  • Bounces: 1207

Time spent statistically significant?: YES
Bounces statistically significant?: YES

This Digital Telescope Will Blow Your Mind

  • Views: 10,035
  • Time Spent: 107 hours
  • Bounces: 753

Time spent statistically significant?: WORSE!
Bounces statistically significant?: YES

The Rise Of The DIY Data Scientist

  • Views: 2594
  • Time Spent: 66 hours
  • Bounces: 1068

Time spent statistically significant?: YES
Bounces statistically significant?: YES

Can Apple Heal Hollywood's Head-In-Ass Disorder?

  • Views: 1237
  • Time Spent: 26 hours
  • Bounces: 682

Time spent statistically significant?: NO
Bounces statistically significant?: WORSE

Caveats

Before discussing what these numbers mean, it's worthwhile to discuss a few caveats. Of course, lots of factors go into what causes a reader to stick around longer on an article or click into another article on the site, including the length of the article and attached audio (the ones we samples are around the same length in both), where traffic comes from (we see lots of bounces from search, less from social), and purely subjective measures like article "quality." In other words, it's hard to prove that SpokenLayer alone caused any of these differences in article performance. This is why it's important to stress once again the small sample size of four articles against a site that publishes almost twice that every day. As I said above, we're not looking for conclusions to the experiment from this data, we're looking to refine it based on what we've observed so far. In that light, here are a few observations.

Tentative Conclusions

So far, adding audio to our articles has had mixed results. Two out of the four articles saw statistically significant increases in hours spent on the article vs. the rest of the site. Three out of four saw bounce rates decrease significantly. However, one article each actually performed statistically worse on bounce rate and time on site. I'm going to focus on these.

One of these articles—Mike Grothaus's piece about a telescope in London that sees through weather—is particularly interesting because it has more pageviews than any of the articles and had fewer bounces, but was actually statistically worse in terms of time spent with the article. The piece is also unique among the four in that it is the only slideshow. Looking deeper into the metrics, I can see that the slideshow contributed to the increased number of pageviews (although it was also a good performer in unique views) and the bounce rate. Looking at other slideshows we've run on the site, I can see a similar trend of low bounce rates and less time spent with the article, so I wonder if any effect of adding audio to the piece wasn't nullified by including a slideshow, which in our site layout also appears significantly above the audio player. I'm going to hypothesize that having too many interactive multimedia elements on a page can overwhelm users and actually cause them to spend less time on the site. Going forward, we should test letting audio stand alone vs. combining it with other types of multimedia and see which performs better.

The other article I want to focus on is (not to pick on him) Grothaus' article about Apple and Hollywood. This is the newest article, which means it's also the smallest sample size. But it also did worse than any other article, with time spent not statistically better and bounces actually statistically worse. Looking at the traffic behind this article, I can see it got most of its traffic from a link on the aggregation site Real Clear Technology. As most publishers know, aggregators can send a ton of traffic your way, but it tends not to be good traffic—that is, readers who stick around. What's interesting to me is that having audio on the story didn't signal to these casual readers that the article was higher quality. I wonder if that has anything to do with the appearance of our audio widget, so I'm going to suggest that we A/B test different sizes, placements, and calls to action to see if any one performs better.

The Bottom Line

It's still early in this experiment, but we know what we can focus on: selecting stories that deserve audio (stories with lots of multimedia already probably aren't good choices) and testing different audio widget designs to see if we can better signal to readers that there's more value to this article than they would have initially assumed.


Previous Updates


The Hacked Newsroom: Why Co.Labs Exists

It may not be obvious yet, but publishing and software are converging. Code is increasingly its own type of content, and content is becoming inseparable from the platform used to create it. That's why we started Co.Labs. This website is more than just a technology vertical: In addition to publishing stories, we operate as an internal laboratory at Fast Company, running experiments that (we hope) will help shape the future of this magazine, and maybe even the industry at large.

This post is a tracker (one of our experiments) that will document every update to every experiment that we run, so that everyone can learn from what we're trying. If you'd like to participate, feel free to drop us a line on Twitter or comment on this post. We're glad to have you along for the ride.


All The Experiments

We waste a lot of time producing articles.

Before you see finished articles like this one, the raw materials live scattered across the hard drives of reporters, editors, videographers, photo editors, producers, and developers. Every time you start a big article, you feel like Sisyphus grabbing the boulder: You forage for info, you share things, you brainstorm—and you almost immediately start to lose track of your raw materials. Writebot is Co.Labs Editor Chris Dannen's attempt to solve this surprisingly complex problem.

Measures of Success:

  • Increase in articles produced
  • Time saved per article


Email has always been a bad place to push.

Online publishers have been offering email subscriptions to their content since the beginning of the web. At the time, it was the best way to "push" content out to readers. But email, which needs to be repeatedly checked and sorted through, was never very good as a push notification service. Moreover, as platforms like Facebook and Twitter continue to draw users away from the inbox, we need to do a better job of reaching out to our readers where they spend time the most.

Twitter, in particular, is much better suited for "pushing" content to individual users with its direct message and @-reply features. To take advantage of these features, Co.Labs technical intern Jiyhun Lee is building a Twitter application that will allow readers to subscribe to our newsletter and updates to individual tracking posts, receiving links to new material via @-reply. We think users will enjoy receiving these messages on Twitter because, frankly, it's less annoying than email. The benefit to us is that these messages are public and much easier to share via retweet than emails, and we can track clicks and engagement much more easily than with email.

There's one more feature we're building in for our own use. We're addicted to real-time analytics, a problem that manifests itself publicly in the "Trending Articles" section of the sidebar, which pulls data from Chartbeat's API. In order to create a better feedback loop between analytics and our writers, Jiyhun's bot will soon send authors public encouragement via an @-reply when one of their articles is trending. Our hope is that this becomes a non-invasive way to help writers understand how their work performs, in real time.

Measures of Success:

  • Number of subscriptions
  • Number of bot re-tweets
  • Number of bot replies
  • Number of bot favorites
  • Clicks on Twitter-bot links vs. clicks on email newsletter links
  • Return visitor rate increase


Our tracking posts experiment has another benefit: e-books.

Lots of publishers are trying to make money by reselling their content as e-books, and Fast Company is no exception. We recently signed an agreement with e-book publisher Vook where we can decide to produce and sell any content as an e-book as long as it's over 20,000 words long. When we heard this at Co.Labs, we immediately thought of our tracking posts.

On the one hand, these stories are good candidates for short e-books because they're focused on a single subject, but they're broader than a single blog post. Over time, they grow to be long enough— some are already approaching the 20,000-word mark—that reading them page-by-page is preferable to consuming them in the browser. Most importantly, these trackers contain lots of individual stories around the same theme that connect to each other in fascinating ways that don't make sense to explore in our "slow live-blog," but would work great as a book.

To help with the process, Co.Labs editor-at-large, Clay Andres (aka, Professor Walrus), a veteran of the technical publishing industry, is going through and cleaning up our stub posts: Formatting them properly, tying up loose ends, looking for connections between discrete updates that can be expanded into new chapters of the story.

On the other hand, considering these posts as material for e-books raises some thorny legal and ethical questions. Many of our trackers excerpt and link to articles on other sites not produced by Fast Company. Our guideline for citing other posts is that we never use them as the main text of the article. Instead, we use them as starting points for our own analysis, adding value and new insights to the original author's work. While we believe this qualifies as fair use on our website, the rules might be different when we start to package and sell the text as standalone works. We're working with lawyers to hammer out some of these details, because we believe that they would make excellent e-books.

Assuming we can figure out the legal issues, look for some of these collections by the end of the year.

Measures of Success:

  • Number of e-books created
  • Number of e-books sold
  • Revenues resulting from e-book sales
  • Profit on e-book sales


Audio is a powerful, but forgotten sense.

Online publishers know the value of images—people love to flip through those things!—and video (hello, engagement), but unless they have a dedicated podcasting division, most seem to have forgotten how much readers love simple audio recordings. There are numerous benefits to having an audio version of an article: You can experience it while in the car, on a device without a screen, or with your eyes closed, and we're willing to bet that we're missing out on a significant readership because we don't offer audio. That's why we're experimenting with adding audio versions of some of our most important articles using a startup called SpokenLayer. Our first attempt is this article about Writebot, another one of our experiments. Expect plenty more to come.

Measures of Success:

  • Audio plays
  • Time on site increase
  • Bounce rate decrease


We take code seriously.

We're writers and developers, so we care deeply about all our language. That's why our dev team helped us implement GitHub Gist embeds directly into our CMS. This means we can easily embed code in our posts with proper syntax highlighting for just about any programming language. So far, we've used this feature to display the hidden messages embedded in Bitcoin and demonstrate how parallax scrolling works in our CMS. Expect to see a lot more code from us in the future.

Measures of Success:

  • Number of Gists embedded
  • Number of Gists forked
  • Time on site increase
  • Bounce rate decrease


You'll be surprised what happens when you invest in quality.

In mid-April, we went live with a half dozen articles which we call "stubs." The idea here is to plant a flag in a story right away with a short post—a "stub"—and then build the article as the story develops over time, rather than just cranking out short, discrete posts every time something new breaks. One of our writers refers to this aptly as a "slow live blog." Here's what we learned.

Measures of Success:

  • Time on site increase
  • Bounce rate decrease
  • Return visitor increase
  • Increase in traffic from search referral


The browser is the new universal gaming device.

To celebrate Pac-Man's birthday, Fast Company developer Harry Guillermo built his own version using JavaScript. Just for fun, Fast Company's CTO, Matt Mankins, used Guillermo's work to build a social version of Pac-Man.

Measures of Success:

  • Increase in fun!
  • Time on site increase


You have two choices:

You can build an app that might save a day of work for everyone in your company but might also end up wasting more of your time than it saves. Or, you can or spend 3 hours doing mindless manual tasks and make everyone else who touches your work miserable. We chose to spend the time building a quick Chrome extension that helped save nearly a day of our judges' time on the Target/CoLabs Retail Accelerator.

Measures of Success:

  • Time saved


Our home page isn't for everyone.

It’s useful if you’re one of our core readers who knows who we are and wants to read every article. If you’re not one of those people (and the chances are pretty good you’re not), you may benefit more from something like the new Co.Labs Back Page.

Measures of Success:

  • Backpage click-through rate vs. home page click-through rate
  • Return visitor rate increase
  • Bounce rate decrease

[Image: Flickr user Mustafa Khayat]






Add New Comment

4 Comments

  • Anthony Reardon

    Pretty interesting Gabe,

    I couldn't help but think that you might be asking the wrong questions and perhaps arriving at the wrong conclusions here. However, the systematic approach and information you provided does make the issue constructively debatable. Nice job.

    Is audio going to influence greater time on site per article? Hmmm...that's a dubious measure to test. I think if you are studying the time spent as a factor of immersive quality, passively including audio media as a value-add would not influence statistical improvements. I would tend to agree with JoeS- subject matter and audience motivation need to be incorporated into your control, otherwise testing the inclusion of audio media player might be worthless.

    Thought the telescope article with slideshow was a novelty attraction. More people are inclined to look at some new tech gadget to get the idea, but fewer will want to know the entire development history and inner workings. Seems like you have a question on the views, but really needs to be unique there. Slides media was not a distractor, but actually worked very well with the audio. Personally, I'd look at such an article interested in the technology that can actually cut through the weather, and would have left as soon as I realized it was just a gimmick. You might do better to include more visual media with audio, and reserve written as a value -add.

    Same goes for bounce rate. Is audio going to influence people to continue exploring the site, as if it signaled some greater quality of experience. I think if it is a passive value-add, you are still going to get that superficial touch and go behavior... and it probably has more to do with readers not having a lot of time to stick around and see what else is going on. "Got to get back to work" so to speak.

    So I think you might do better to more assertively leverage the audio value. Perhaps come out with a special content category like "Grab Air" or something where busy people can grab audio articles and jack into their cars on the way to and from, for example. You might go with content specific to that user audience like "How to report an in-text-icated driver" or "How carpooling really is your ticket to the fast lane". 

    Needless to say, such an approach would call for a different line of questions, but I bet there are significant conclusions to be had for your bottom line by incorporating audio media.

    Best, Anthony 

     

  • Gabe Stein

    Thanks Anthony, I appreciate the thoughts. I'm not sure I agree that time on site shouldn't be impacted by audio. As Joe said, he reads a lot faster than people speak, so if people are listening to audio, they should stick around longer.

    The bigger question you've rightly hit on, however, is how to measure business impact. The reason we chose to look at time on site and bounce rate is because they are important metrics for us, as publishers, that we use to sell our site to advertisers. We also use these measures as a proxy for "engagement" -- that is, people interact with our content rather than merely reading it and going away.

    So the question is: how can we, as you say, "more assertively leverage the audio value" in a way that provides a direct benefit to us. Yes, opening up an audio section and advertising it might be a good idea. And yes, we need to make the audio more accessible on mobile devices in non-connected settings. But both of those things are time-consuming builds that would take a lot of work up-front to prove, and we just don't have the money or time to take those gambles. This is our way of testing it in a lighter way that doesn't require a lot of new infrastructure or advertising, which is really the only thing we can get away with right now.

    So if not time on site and bounce rate, what other variables can we be looking at, or how else can we test it in a lightweight way that might prove the value so we can convince ourselves and our management to let us devote more time to the bigger build-outs you mentioned?

    Thanks again!

  • Anthony Reardon

    You bet Gabe!

    That's just it. My theory is most users come here as a reader base and won't listen to the audio if it is not proposed for some special purpose. I thought your first series of tests sort of demonstrated that, but it would be good to look at how many Audio Plays per article you got during a given timeframe.

    Keeping it light, include specific quantifiable calls to action to like, share, and comment on the audio files themselves. Maybe a dedicated page for the (audio version) instead of just in article. The split could reduce bounce rate on articles by giving people one more thing they might like to check out onsite. Split again by offering a link through to a list of all audio versions on dedicated page. Now you articulate bounce rate on audio experience, perhaps boost of time on site with cleaner data, and you might end up with a justification for a category build/ advertiser solution along the way.

    I like that you responded to my feedback and engaged me in the challenge. One alternative would be to get your community involved via hackathon or contest like you did with the Target project. Perhaps that way you can get a heavyweight effort without necessarily so much of your own time and money. You know, straight out ask people to share with someone that might like to listen during a commute, invite feedback, get them to actively participate in the experiment you've got going on this split/ dedicated page.

    So, instead of passively including an audio version, some relatively minor adjustments along the line of your long-form recombinant article experiments.

    Best, Anthony 

  • JoeS

    I would suggest that subject matter of articles is a larger driver of stats than play button.  I read faster than people speak, by a large factor.