Remote Research

Nate Bolt and Tony Tulathimutte

• Publisher: Ingram CS-Plus
• Year: 2010
• ISBN:
  9781933820446
• Edition: 0
• Topic: Design
• Subtopic: UI/UX Design

Chapter 1

Why Remote Research?

The Appeal of Lab Research

Is Lab Research Dead?

What’s Remote Research GoodFor?

When to Go Remote

Moderated vs. Automated

When to Use Which Remote Method

Chapter Summary

In-person lab research used to be the only game in town, and as with most industry practices, its procedures were developed, refined, and standardized, and then became entrenched in the corporate R&D product development cycle. Practically everything gets tested in a lab nowadays: commercial Web sites, professional and consumer software, even video games (see Figure 1.1).

Photo courtesy of Danny Hope

Figure 1.1

http://www.flickr.com/photos/rosenfeldmedia/4218821411/
Brighton University’s usability lab, from behind the traditional two-waymirror.

The Appeal of Lab Research

Part of the appeal of lab-based user research was that it provided a seemingly scientific basis for making decisions by using observational data, instead of someone’s error-prone gut instincts. Stakeholders appreciated the firm protocol and apparent reliability of properly managed lab research. Lots of user research practitioners continue to perform lab research just because it’s what people have been doing for a long time.

Market Research vs. User Experience Research

Let’s make something clear. Focus groups are practically synonymous with user research in most people’s minds, and focus groups belong to the world of market research. But there’s a huge difference between market research and user experience (UX) research. Market research is much more common and comprises the lion’s share of research spending; UX research comprises just a fraction of that (see Figure 1.2).

Figure 1.2

http://www.flickr.com/photos/rosenfeldmedia/
4218822967/
The relationship between market research and UX research. The two fields seem similar, but they have different goals and take different forms.

However, this book is about user experience research, not market research. The main difference between the two fields is that market research focuses on opinions and preferences, whereas UX research focuses on behaviors. The distinction can be confusing, especially since a lot of online consumer research companies try to convince you they give you insight into “what your customers are doing on your Web site,” when they’re really just providing opinions.

A market research study might have goals like these:

• “Determine how users respond to our branding.”
• “Identify different segments’ color preferences for the homepage.”
• “See if users like our new mascot.”
• “Determine what users enjoy most and least about our site.”

While the goals of a UX study, on the other hand, would sound more like these:

• “Can anyone actually use my interface?”
• “Determine where users make errors in completing a purchase.”
• “See whether users can successfully create a playlist.”
• “Understand why users aren’t logging in.”
• “See how users mentally organize different product categories.”

It’s important to keep in mind that market research is pretty useless over small sample sizes. Opinions can vary widely across demographics and location, are very sensitive to the phrasing of the research questions, and can change fairly quickly. Behavior, on the other hand, is fairly consistent across demographics and location for many tasks, and most usability flaws in a given software interface can be uncovered in a moderated study by a much smaller number of users.

How small? That’s a contentious question. UX luminary Jakob Nielsen (in)famously claimed that having five users was enough to uncover 80% of usability flaws in an interface, but others like Jared Spool insist that the number depends on factors such as user segmentation, risks associated with the errors, task complexity, and so on. At any rate, our point is that a moderated UX study usually requires much fewer people than a market research study.

Put it this way: ask 10 people what they think about how well a door is designed, and their comments might not overlap at all. One blames the condition of the hinges, another talks about the weight of the door, another complains about the color of the doorframe, and so on. But if you observe 10 people walking through the door, and the first two accidentally try to push when they ought to pull, then you’ve found your design flaw right there.

So, to put it all together: whether you go with market research or UX research depends on what you’re trying to find out. This book is about UX research, so it’s focused on user behaviors rather than opinions.

Further reading about the sample size question

Turner, C. W., Lewis, J. R., and Nielsen, J. (2006). Determining usability test sample size. In W. Karwowski (ed.), International Encyclopedia of Ergonomics and Human Factors (pp. 3084–3088). Boca Raton, FL: CRC Press.

Lewis, J. (2001). Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. The International Journal of Human–Computer Interaction 13(4), 445–479.

Lindgaard, G., and Chattratichart, J. (2007). Usability testing: What have we overlooked? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, CA, USA, April 28–May 03, 2007). CHI ’07. ACM, New York, NY, 1415–142

Is Lab Research Dead?

Heck no. Lab and remote research share the same broad purpose: to understand how people interact and behave with the thing you’ve made (from here on, let’s just call it “the interface”). There’s no need to set up a false opposition between the two approaches—one isn’t inherently better than the other. Despite the versatility of remote research, there are lots of reasons you might want to conduct an in-person study instead, most of which have to do with security, equipment, or the type of interaction you want to have with your research participants. More generally, lab research is appropriate when you need a high degree of control over some aspect of the session, such as the following situations.

Info security. Security is often a concern for institutions like banks and hospitals, which deal in sensitive information, or companies concerned with guarding certain types of intellectual property. If you’re testing a top-secret prototype, you obviously don’t want to let people access something from their home computer, where it could be saved or screen-captured. On the other hand, you might also be doing a study on users who would be secretive about sharing what’s on their screen—government employees, doctors, or lab technicians, for instance. Either way, you’ll want to test users in a controlled lab environment to keep things confidential, especially if what you’re testing is so hush-hush that you must have your users sign a nondisclosure form.

Inability to use screen sharing. You might also want to use a lab if your users are unable to share their screen over the Internet, for whatever reason. Some studies (of rural users, cybercafe patrons, etc.) may require you to talk to users who don’t have reliable high-speed Internet connections, who own computers too slow or unstable to use screen sharing services effectively, or who have operating systems incompatible with the screen sharing tools you’re using. These restrictions apply only to moderated studies, for which you need to see what’s on your users’ screens.

The need for special equipment. Depending on the interface you’re testing, you may require certain special software or physical equipment to run the study properly, which is most often the case with software that’s still under development. Getting users to install and configure tools to run elaborate software can be a pain (though that’s not unheard of), and requiring users to have certain equipment can make recruiting needlessly difficult.

The importance of seeing the user’s body. Some kinds of research will require you to study certain things about the user that are difficult to gather remotely. UX research has recently begun using eye-tracking studies, and for that kind of study, you’d need to bring the users to the eye-tracking device. Other studies might require you to attend to the participants’ physical movements, which may be difficult to capture with a stationary webcam. And then there are multiuser testing sessions, in which a single research moderator facilitates many participants at once; screen sharing is currently not well suited to sharing multiple desktops at once, though some tools (e.g., GoToMeeting) make it relatively painless to switch from one desktop to another. We want to emphasize, however, that for most studies, seeing the user at all is not actually important; we explain why in Chapters 5, “Moderating,” (see “Ain’t Nothing Wrong with Using the Phone”) and 10, “The Challenges of Remote Testing.”

Although these situations are all compelling reasons to conduct in-person research, part of what we want to demonstrate in this book is that remote research is very broad and adaptable, and even if a study is conducted in a lab, elements of remote methods can be adapted and incorporated to enhance in-person research methods. We’ll get to that in Chapter9, “New Approaches to User Research.”

Why I Went Remote

by Brian Beaver

Old habits die hard, but for any number of reasons—cost, convenience, international testing—a handful of former lab researchers have switched to remote methods and never looked back. Brian Beaver, award-winning creative director at Sony, explains why he decided to go with remote methods after many years of lab testing.

On Going...

Table of contents

• How to Use This Book
• Frequently Asked Questions
• Foreword
• Chapter 1
• Chapter 2
• Chapter 3
• Chapter 4
• Chapter 5
• Chapter 6
• Chapter 7
• Chapter 8
• Chapter 9
• Chapter 10
• Conclusion