2013-04-16

How Facebook Did UX Testing For Facebook Home (With Fewer Than 60 People)

As we famously learned this month, Facebook doesn’t subject its designers to data-driven mandates. So we were curious: When did the UX testing for Home begin and what effect did it have on the finished product? The details--including a counterintuitive pool of testers--might surprise you.



Facebook Home is a project so monumental, it could actually redefine Facebook as a corporation--assuming it succeeds. Facebook's effort to own your home screen is also a remarkable milestone in the death of the file system and changes our definition of what makes an OS. To find out how the Facebook team tested Home before releasing it into the wild, we spoke to Marco De Sa, Facebook's UX researcher assigned to Home.

What did you think when you first encountered the prototype of Facebook Home?

I got involved around four or five months ago, shortly after I joined Facebook. The first time I saw the product I was amazed--I tried to come up with a research plan that would address everything that we believed was interesting to look at. But as time went by, we started seeing that some of the approaches that we were using were more adequate and provided really interesting feedback on certain things--such as the diary study for learning how people would see content over time. That was something that we couldn't do with just an interview or an observation of 45 minutes. Some of the things I knew right away that, okay, this is the best way to do this; some of the other things you learn throughout the process, trying to adjust our methods and techniques based on the type of data that we're getting.

How finalized was the Facebook Home product before you began UX testing?

The biggest parts of the product were kind of defined. We were just trying to improve them as much as we could. Some things actually changed along the way, but for the most part, cover feed and Chat Heads and the notifications were things that we knew that we wanted to include; and then we tried work on showing it to users and getting feedback and trying to improve it as much as we could. Some of the main things were already built, and the idea was to test some of the experiences that have been added to interact with notifications.

What were the big questions you were tasked with answering?

Was this the type of content that we should be showing? How frequently should we refresh? Since the beginning, those were the main questions, and then we used a bunch of approaches to address each of those specific issues. For instance, how to interact with Chat Heads. How would people react to the content? What kind of content were they expecting to see? What would they like to see on the cover feed? Things like that.

Were the touch gestures already defined by this point?

We started looking at how people would understand the interactions if we changed the gestures. Certain gestures were preferable to others, because they were easier to understand, or because they were more effective in navigating the content. Our focus is not necessarily on the performance of the gestures, but more on the experience they provide whether they're clear or not--whether people felt comfortable using them, whether they conveyed the right action or whether they were associated to the right action, and whether words were used in general or not.

In the big picture, did any of your research findings have major effects on how the product turned out?

One of the things that we did as research was running a bunch of diary studies; so we had people providing feedback over time, quite frequently, every week. I'd interview or get feedback through surveys, questionnaires by a bunch of people that were using the product, and that helped us understand how people would get interested more in some types of content or less over time. It also helped fine-tune the algorithm that actually provided levels of content that shows up on cover feed. The product actually changed a lot over the last few months.

What were your testing priorities?

I worked quite a bit on the new user experience. We tried to understand: What were the gestures that people got right away, gestures that really didn't need any explanation? What were the things that weren't as clear, where we have to assist people for the first experience so that they could start using it right away?

Was Facebook Home UX research unique in any way, compared to testing that has gone into other Facebook products or features?

One of the most interesting things about this research was the fact that we actually used so many different techniques: the diary study, interview, and observation, user surveys. We used logging for different types of events. We used focus groups. We had internal conversations with users to get feedback as well. I think the combination of those different approaches allowed us to see some patterns emerge and focus on the right things. Each of those individually was really usable for certain types of interactions, to cover certain types of issues that the product has. The combination was something that was really interesting and I think really paid off in the end.

How did you organize the lines of inquiry that would be your research?

We didn't actually divide just based on gestures and content. We divided mostly on approaches. Each approach that we used would yield results targeting something specific, because that type of research is more adequate for certain types of information. For instance, for the content, just understanding how people would react to the content over time was very interesting, because it's not something that you can understand from a 20-minute interview.

Why aren’t in-person interviews good for something like assessing content?

Because people are absolutely fascinated the first time they see cover feed; and it's just a surprising experience. We were trying to understand how would that change over time after the first experience; so for that, we used a diary study.

How did you test the first-time user experience?

For things like the new user experience, as later mentioned, I intercepted people walking around [Facebook] campus. I showed them and got their first impressions on looking at Facebook Home, how they'd react, what they expected that they could do over the content that they were seeing, and that was the way we got to understand how people would learn, what were the difficulties that they would find trying to use it for the first time. So, different techniques were used for different things, but the interesting thing is by using those different techniques, we actually saw some patterns emerge and some of those things coming across over different approaches kind of were the ones that we had to prioritize.

So all these studies were with Facebook employees? Is it always a good idea to test on your most expert users?

Oh, we definitely try to test with external users as much as we can and for Facebook Home, at the beginning of the study, we actually showed some external users; but for some of these interactions Facebook employees are users just like any other. Trying to see how they use gestures and things like that, or how they learn or discover new gestures is something that you can actually do with Facebook employees as you would do with another user. Of course, Facebook employees seem to be more proficient with using Facebook, but again these interactions aren't necessarily something that is Facebook dependent. They're just navigating content and things like that.

Is there really adequate diversity of opinion and experience within the Facebook corporation for testing something that will be used by people all over?

We try to talk to Facebook employees who were not involved in design, were not engineers, we try to get a broad sample of people with different levels of expertise, recent employees, older employees. So, we try to recruit considering all of those differences. Yes, we're designing for a billion people.

How many people are we talking about, total?

I interviewed around 60 people with the diary study. The amount of data that we required, the amount of feedback, and the level of interaction that we had with each of those participants. For the first batch, we had around 12 people doing the diary study and for the second one we had around 10 people doing the diary study over the course of four or five weeks. Each of these users was interviewed several times. They provide feedback four or five times a week, so it's a pretty intense study and we couldn't do it with a lot of people. Usually, this type of approach is used with fewer people, actually, but because of the importance of this project and the type of experience that we're providing, so intense. It's interesting and different from what people are used to.

How did you translate what was in those diaries into some kind of meaningful aggregate information?

When we do intercepts in the middle of the courtyard, whether we're in the lab or just going to peoples' desks, we like to stream whatever we're doing to the engineers and to the designers. We film everything that we do, and we pipe this [to the Home team] so they can actually see what's happening live. They can send me questions and we can shift the focus of the interview based on what people are working on.

Were there asynchronous methods for conveying this information too?

Every week, I'd send an email to the whole team with the highlights of research, the main findings, the high priorities, and all the details which follow, targeting with both different methods and the different issues that we were looking at. We had designers and engineers come to me and ask me questions and we talked several times a week. At the end of the week I'd create a summary of the research that was done during that week.

When you went off campus, what equipment did you bring?

We had a webcam connected to a laptop, streaming to whoever [on the Home dev and design teams] might be interested in watching.

Which way was the camera facing: user or device?

You do both depending on the type of test. We actually have a camera facing the person who's using the device, and for this test in particular, another camera pointing at the device.

If you work at smaller company, is it a good idea to do the campus interviews or would you take a different approach?

I think [on-campus testing] is always a good idea. We do a lot of quantitative work, which requires, sometimes, adding more scale and approaching more users, but I do feel like that the quantitative work that we do can be done by pretty much any company. We just need to reach out to users, focus on the user; and we used a bunch of techniques. Sometimes we just have techniques just to make it work. Again, going outside on the campus, just approaching people with a camera attached to phone isn't the most scientific way, but it was what we made work and, actually, gave out really interesting results.

What sort of “quantitative” approach do you take? Do you get granular about each step in a given task flow, or try to just come up with holistic insights?

Both, actually. If it's a test-based exercise, we look at how many steps the person took to get there. How long did it take? We also look at the facial expression where the person was, like, trying to think what was the next step, whether it was just something that came out really clearly and you need to get to; but we try to combine as much as we can to different approaches. We sometimes look at it from a listening perspective and try to understand, as an experience, how it's working. But we also divide certain tasks and use a quantitative approach when we look at number of steps, number of clicks, or even time taken to do them.  


Need more advice on creating a killer first-time user experience for your web or mobile app? We’re tracking the best ideas in this ongoing story.


[Image: Flickr user Casey Muir-Taylor]