2014-01-22

Co.Labs

Under The Hood Of The New NYTimes.com

To keep up with changing reader demands, the Times's tech team had to rewire nearly everything.



For media companies, the pace of technological change can be unforgiving. The New York Times, well-staffed as it may be, is no exception. To meet rapidly changing business goals and reader expectations, the tech team there recently pulled off a redesign that required overhauling the backend of one of the biggest news sites on the planet.

As you might imagine, things got complicated.

By the time the project—code-named "NYT 5" internally—kicked off two years ago, the problems were clear. As web standards, mobile device adoption, and reader expectations all barreled forward, the NYTimes.com architecture had been gathering dust, resulting in hacked templates, messy code, and pages that didn't always play nice with the latest devices.

Even worse, the old system wasn't allowing the Times's developers and interactive journalists to iterate quickly, making it harder to deliver badly needed updates to impatient readers. These gripes, combined with business needs like increasing ad inventory, informed the first set of redesign mockups that starting make the rounds internally six months later.

"As a company we had all agreed that the fixed width layout that we were using from 2006 was no longer cutting it with plethora of different devices that are available today," says Mike Finkel, Executive Director of Web Products Technology at the New York Times.

"We were doing a lot of hacking to make our content work on our iPad app, iPhone app, and Android app. We all agreed that this was the time to do something new."

As editorial, design, and business goals all came into sharper focus, the true breadth of the project revealed itself. By the time NYT 5 launched earlier this month, it wouldn't just include the freshly polished exterior—everything under the hood would be new, too.

Modern Publishing (Without Hacking The CMS)

If you're one of the many people whose jaws hit the floor upon seeing the publication's Snow Fall story last year, consider yourself somewhat lucky. At the Times, this type of multimedia-rich layout has historically been born in spite of their publishing system, not thanks to it.

Indeed, like many of the immersive web stories that followed, Snow Fall was built alongside the Times's content management system, independent of their standard online editorial workflow.

It's not just the interactive blockbusters that demand these kinds of arduous technical backflips. Smaller multimedia projects have historically employed a variety of template hacks and workarounds too. By overhauling the site's infrastructure, the tech team took the first and biggest step toward changing all that.

"Every time the newsroom wanted to do something that fell outside of the realm of our standard layouts and publishing mechanisms, it was just a huge hack-fest to do it," says Finkel. "A lot of times they would have to publish their interactive pieces into a standard article template, but the published item was just a ton of JavaScript, HTML, and CSS coded to completely rewrite the DOM. They've been doing that for a several years now. The technical debt that amounts when you're continually hacking away at something is something that's really hard to overcome without starting from scratch."

At the same time, even regular news articles were published in proprietary format, static HTML documents which lived in the Times's data centers. This less-than-dynamic system made it hard to maintain a consistent look and feel across templates over time.

As development manager Reed Emmons explained in a recent post on the New York Times Open blog:

"Since the article’s HTML was fixed but the scripts and stylesheets inside it were dynamic, our front end team needed to maintain every template version that had ever been created. This is why an article published two years ago is a bit different than an article published a few weeks ago. Soft launching backward compatible CSS and waiting for it to uncache in every browser before launching a JavaScript change became common practice."

By comparison, the new infrastructure is much more dynamic. The team completely rebuilt the PHP rendering engine that pulls every page together as users navigate the site. The new framework is cleaner, more efficient, and—crucially—architected in such a way that building on top of it in the future won't create the sort of technical debt that has caused so many headaches over the years. It will also make building new features and apps easier, allowing developers to glue existing components together rather than starting from scratch.

Rather than a brand new CMS built from the ground up, the editorial side was given a host of new features and capabilities, including a new publishing template and better control over multimedia assets. Part of that newfound freedom, Emmons explains, is automation that frees them from thinking too much about how the content is formatted. Among these time-savers is "an algorithm that paints images down the page in a way that makes sense contextually."

For the editorial side, this is just phase one. Over time, publishing interactive mega-features like Snow Fall on-the-fly will become much easier.

Building A More Responsive Layout

Never in human history has a technology been as quickly adopted as the smartphone. Indeed, by the time the NYT 5 project kicked off, "responsive design" was already maturing from buzzword to best practice. The Boston Globe launched its own cross-device-friendly design in 2011, with many other major publishers leaping onto the bandwagon.

For the Times, the first step toward catching up was an overhaul of the front-end architecture. To keep things structured, efficient, and easy to update, Emmons and his team cobbled together some of the most beloved JavaScript frameworks, including Backbone.js and RequireJS for structure, as well as web development standbys like jQuery, Modernizr, Underscore.js, Hammer.js, and SockJS.

"We leveraged tools that have been very successful already and sort of glued them together," says Emmons. "There was a fair bit of tailoring to meet the needs for our organization, but all of the core principals of it are existing libraries that already exist and are well supported throughout the industry."

On top of this skeleton of scripts, Emmons and his team built the next biggest piece of the NYT 5 puzzle: A homegrown, responsive JavaScript framework that would allow the Times to more effectively serve content to readers on whatever newfangled device they happen to be staring into on the subway.

"There's probably a good hundred different ways that our articles can be rendered," says Emmons. "To serve all these interesting scenarios we needed a more custom solution. That's why we rolled our own responsive framework to get all these different breakpoints starting at 768 with the iPad portrait mode all the way up to 2030."

The mission of the Times's new responsive engine goes beyond the obvious need to deliver web-based news to iPads and Samsung Galaxy devices. It also allows the site to make intelligent decisions about rendering content based on the type of media elements displayed, the orientation of the device, and even the monetary value of the advertisements on the page.

Speeding Things Up For Readers And Developers Alike

Like any modern web design project, the NYT 5 initiative placed a heavy emphasis on speeding up the site. Things like consolidating the site's CSS stylesheets with Less and reducing the number of JavaScript calls can go a long way toward improving page load times. And, of course, rebuilding the entire PHP rendering engine from the ground up doesn't hurt either.

Of all the tools utilized throughout the project, few were quite as handy to Emmons and Finkel as Varnish, a front-end caching layer that helps with speed and scalability.

"Varnish is a reverse proxy cache with a lot of customizability as far as like the types of things it caches, how quickly, how long it caches these things for, and whether we incrementally purge those cache entries," explains Finkel. "It's one of the biggest reasons we were able to make a lot of the more creative decisions on the backend. We knew we had this fantastic front-end cache layer to rely on."

Between all this front-end streamlining and the rebuilding of how the server-side architecture works its magic, the New York Times site should, in theory, be noticeably faster to readers. Just as important, though, is the efficiency with which developers can now code new features atop a freshly rebuilt, modern platform.

A More Future-Proof News Site

Like the server-side overhaul, the reinvention of the site's front-end had as much to do with making developers happy as it did with wooing readers. Using industry standard JavaScript frameworks and streamlining the way new features are coded, for instance, makes future maintenance much, much easier.

"We standardized everything so there's one way to code and it all fits into our framework," explains Emmons. "You could be a junior developer and understand the structure. It's about solving the product and business issues at hand rather than trying to figure out what data structure to use. We want to allow them to be creative with their solution without necessarily having to think about how to do it."

This is a huge deal. The reason other publications were so easily able to beat the Times to the punch with things like responsive design is because its rusting architecture made it harder to push new iterations out in response to evolving reader habits. The expectations of those readers will continue to change at an unrelenting pace. The difference is that now, the code and developers behind NYTimes.com will be in a far better position to adapt.






Add New Comment

1 Comments

  • They have done a lot of standard FEO work but the homepage still has 17 CSS objects and 45 JavaScript objects which adds a lot of overhead to site download speed. Setting longer expiry for static objects will give repeat visitors a better experience.