2013-11-18

Co.Labs

How Journalists At The Guardian Built That Epic NSA Story

A behind-the-scenes look at how the paper’s HTML5 mega-feature was conceived and built.



The Guardian knew it had a massive story on its hands. Its NSA surveillance scoop was one of the biggest national news stories in years. But the team quickly realized that the story would spiral out of control because of its complexity—new leads were emerging daily as details emerged. To overcome the story's complexity, the paper assembled a team of journalists, developers, and designers to come up with a new, stunning way to tell it.

When it went live on November 1, the finished product—a six-part multimedia extravaganza entitled NSA Files Decoded: What the revelations mean for you—was met with resounding praise among the type of online journalism geeks that go nuts for this kind of thing. Weaving together long form text, documentary-quality video interviews, embedded documents, and interactive graphics, the story instantly generated a level of buzz not seen about a single piece of journalism since the New York Times published its Snow Fall piece last year.

But it wasn't just the Nieman Lab junkies and future-of-news enthusiasts that ate the whole thing up. Within four days, the story had pulled in hundreds of thousands of readers, many of whom spent as much as 30 minutes reading the piece and consuming its interactive elements scroll-by-scroll. All told, the project took about two months and three people working full-time (plus about another dozen others chipping in) to complete.

From Brainstorming to Mock-ups: How The NSA Feature Came About

The idea to produce an interactive feature on the NSA spying scandal came out of a meeting held this summer at the U.K.-based paper's newly opened New York City headquarters. Gathered in the office of editor-in-chief Janine Gibson was a small team of editors and reporters who were brainstorming ways to present the larger NSA surveillance story in a broad, but digestible format.

"As they were reported, the stories are often necessarily technical, both legally and technologically," says interactive editor Gabriel Dance. "Because of this, and because of the way we've needed to report the story as we've gone through the documents, we wanted to create a piece that explained to readers what this meant specifically for them."

The problem, as Dance and his colleagues saw it, was that an ongoing story of this magnitude was hard to effectively explain in a series of traditional news articles published over the course of many weeks. While that day-to-day reporting is crucial to the larger narrative (and, one hopes, informing whatever result may come out of the NSA spying scandal), readers could use a breather from the deluge and an opportunity to sit back and re-digest the story from a broader vantage point.

Sure, they could whip up a 5,000-word explainer and hit "publish" (as others have done and will undoubtedly continue to do) but today's digital news ecosystem calls for something more immersive and engaging. To achieve that, they would need to blend a written narrative with interactive elements and video clips and package it all in an eye-catchingly beautiful layout. Like the New York Times's Snow Fall and Pitchfork's interactive feature on Daft Punk, NSA Files Decoded needed to feel like an experience rather than a news article.

After brainstorming the possibilities, the team set out to write the story and produce the multimedia elements that would go along with it. These processes unfolded in tandem with one another, often overlapping as the text informed the content of the interactive elements and vice versa. Meanwhile, Dance sat down with designer Greg Chen to sketch out the project from a visual standpoint.

"We had a design style in mind, as well as some behaviors," says Dance, who made his name in interactive journalism during a four-and-a-half-year stint at the New York Times. "We knew that we wanted to do several video interviews, but that these interviews would be used as talking heads that would complement a part of the writing. We also knew that we wanted to include documents and that we needed to create something that would immediately engage readers and pique their curiosity."

The pair sketched wireframes of their design concepts on paper, from which Chen then created mock-ups in Adobe Photoshop and Illustrator, as well as animated prototypes in After Affects.

"With different elements responding to scrolling events on the page, it can be difficult to explain functionality," Dance explains. "We've developed a technique where we block out our designs and interactions using motion graphics so that when we build it, it's easier to understand how the look and feel should function."

Pulling It All Together: The Tech Behind The Layout

The mockups and prototypes were eventually handed off to Feilding Cage, a journalist and coder who played an integral role in producing the story. In addition to contributing some written copy and coding various elements of the package, Cage built a framework that would dynamically merge the piece's many moving parts into a single HTML document. The framework, a combination of JavaScript, CSS (compiled with LESS.js), and HTML, served as a structural outline that could be triggered with a Ruby script that instantly snaps everything together in the browser.

"It was modular in such a way that we could pull it all together and view it as a single project," says Cage. "People were able to work on their piece of the interactive separately and we could preview it and test it separately." This distributed workflow made the whole thing significantly easier to produce.

Leaning heavily on the modern capabilities of HTML5, CSS, and various JavaScript libraries (especially jQuery), Cage and other members of the team transformed ideas and mock-ups into a functional, almost alive-seeming web layout that smartly responds to the reader's behavior. As you scroll down the page, video players and interactive elements are progressively loaded (and then unloaded when they're no longer needed), making the page easier for browsers to digest without choking.

The interview video clips, which are embedded using the open-source HTML5 video library Video.js, were filmed in front of a green screen so they could be seamlessly integrated into the page's design. Each clip starts playing automatically when the user scrolls over it and stops (saving their spot in the video) when they keep scrolling. Ordinarily, videos that auto-start are an unforgivable UX faux pas. And indeed, it takes a minute to get used to being interrupted by talking heads as you're trying to read an article. But in this case, the voices belong to people—NSA officials, congressmen, and activists—who are uniquely well positioned to help tell this story.

Like many of today's multimedia journalism projects, this one utilizes the D3.js framework to power its interactive data visualizations. D3, which recently became the first JavaScript library to win a journalism award, is used to create or modify elements on the page based on user input. In this case, the framework was well suited to power things like an interactive chart breaking down congressional votes on NSA-related legislation and a visualization detailing who sits on the FISA Court, broken down by demographics and political party.

Another visualization creates a simplified bubble chart illustrating the types of data the NSA can access from different sorts of online activity: web browsing, Google searches, Facebook, email, and so forth. Toggling the activities on (by clicking icons) expands the color-coded bubbles into a larger and larger sea of data—illustrating for the reader just how expansive this data-corralling dragnet really is.

It's details like these that could be worked into the written narrative, but never in a way that expresses them quite as effectively as data visualizations can.

Building A Responsive Story For the Modern Web

Building a web layout this fluid and interactive without Flash or some other proprietary plug-in would have been unthinkable just a few years ago. Thankfully, the major browsers have advanced far enough that standards like HTML5, CSS3, and all the latest whiz-bang JavaScript frameworks can be used to replicate what only clunky add-on technologies could handle before.

But it's not just browsers that have evolved—consumers have too. They're now accessing the web from a variety of devices throughout the day, and any serious web design project needs to reflect that. This becomes especially tricky when you've created complex, interactive graphics, which may or may not be easily translated to small screens.

"The cable map is really beautiful piece, but it's really intensive even for a desktop comp to draw and update," says Cage. "We didn't think that it could be optimized for mobile, so we included the text that went with it and let the user know that there's more to this on larger screens or on an iPad."

One by one, the team looked at each non-textual element on the page and decided whether or not it could be displayed to users on smartphones and tablets. Some were axed, while others were trimmed down. The embedded documents, as simple as they may seem, don't translate well to the screen of a phone, so they didn't make the cut in the mobile version.

Throughout the process, they tested each component of the piece individually—and then as a whole—on a variety of browsers and devices, reflecting the technology used by Guardian readers in the wild.

"Testing was done on actual devices and computers instead of a virtual machine or simulator," says Cage. "Because of the complexity of this project, we really wanted to test in a way that was as close to the real experience for our users, including downloading over Wi-Fi and a variety of connections."

Analytics Say: This Is The Future Of Storytelling

Within a few days of going live, NSA Files Decoded was yielding some pretty impressive metrics: Hundreds of thousands of readers flocked to the story, sharing it on social networks at a rapid clip. Perhaps most importantly—albeit unsurprisingly—the story pushed the much-coveted "time on site" metric above five minutes for the entire Guardian site.

"The time on site has been fantastic," says Dance. "There are more than 23,000 visits that lasted over 30 minutes. That's more than 11,500 hours, or 479 days spent on the story. And that's just a slice of the users. That blows my mind."

Helping to fuel these numbers is a smart social strategy. Like just about any online news story, this one comes with persistent sharing buttons for Facebook and Twitter sitting atop the page. But what really drove the social buzz was the decision to include Twitter sharing and deep-linking icons next to every paragraph. This allowed readers to share very specific passages, documents, or graphics with friends, catering to various interests and allowing previously unknown details to bubble up on the social web. "The deep linking was a huge asset," Cage says.

For the Guardian, the payoff of all this hard work isn't a directly financial one: There's not a single advertisement on the seemingly bottomless page that houses the story. Instead, the mega-feature has served to boost its site-wide numbers, making it a more attractive buy for advertisers. Meanwhile, a sudden burst of positive exposure helps buoy an already respected media brand at a time when the attention of readers—not to mention revenue—is more fractured than ever.

"The most challenging bit was orchestrating all of the different aspects of the piece into a cohesive story: visually, narratively, and behaviorally," says Dance. "I'm sure we'll learn some lessons from this project and incorporate them into future projects. There'll definitely be more to come."






Add New Comment

3 Comments

  • happycamper

    Great article. Any insights into the backend BI systems used to analyse the data and generate the data visualizations? Tableau?

  • Bill Owen

    This development is wonderful and effective. Yet one more virtual nail in the coffin of text printed on pulp and those things called "books".

  • Bradford

    I agree. It doesn't sound like the Tenzin I know, either. And he should only be shown in color, IMHO.