Online Panel Research

A Data Quality Perspective

Mario Callegaro, Reginald P. Baker, Jelke Bethlehem, Anja S. Göritz, Jon A. Krosnick, Paul J. Lavrakas, Mario Callegaro, Reginald P. Baker, Jelke Bethlehem, Anja S. Göritz, Jon A. Krosnick, Paul J. Lavrakas

• Publisher: Wiley
• Year: 2014
• ISBN:
  9781118763513
• Edition: 1
• Topic: Mathematics
• Subtopic: Probability & Statistics
• Index: Mathematics

Provides new insights into the accuracy and value of online panels for completing surveys

Over the last decade, there has been a major global shift in survey and market research towards data collection, using samples selected from online panels. Yet despite their widespread use, remarkably little is known about the quality of the resulting data.

This edited volume is one of the first attempts to carefully examine the quality of the survey data being generated by online samples. It describes some of the best empirically-based research on what has become a very important yet controversial method of collecting data. Online Panel Research presents 19 chapters of previously unpublished work addressing a wide range of topics, including coverage bias, nonresponse, measurement error, adjustment techniques, the relationship between nonresponse and measurement error, impact of smartphone adoption on data collection, Internet rating panels, and operational issues.

The datasets used to prepare the analyses reported in the chapters are available on the accompanying website: www.wiley.com/go/online_panel

• Covers controversial topics such as professional respondents, speeders, and respondent validation.
• Addresses cutting-edge topics such as the challenge of smartphone survey completion, software to manage online panels, and Internet and mobile ratings panels.
• Discusses and provides examples of comparison studies between online panels and other surveys or benchmarks.
• Describes adjustment techniques to improve sample representativeness.
• Addresses coverage, nonresponse, attrition, and the relationship between nonresponse and measurement error with examples using data from the United States and Europe.
• Addresses practical questions such as motivations for joining an online panel and best practices for managing communications with panelists.
• Presents a meta-analysis of determinants of response quantity.
• Features contributions from 50 international authors with a wide variety of backgrounds and expertise.

This book will be an invaluable resource for opinion and market researchers, academic researchers relying on web-based data collection, governmental researchers, statisticians, psychologists, sociologists, and other research practitioners.

Chapter 1
Online panel research
History, concepts, applications and a look at the future¹

Mario Callegaro^a, Reg Baker^b, Jelke Bethlehem^c, Anja S. Göritz^d, Jon A. Krosnick^e, and Paul J. Lavrakas^f

^a Google UK

^b Market Strategies International, USA

^c Statistics Netherlands, The Netherlands

^d University of Freiburg, Germany

^e Stanford University, USA

^f Independent Research Psychologist/Research Methodologist, USA

1.1 Introduction

Online panels have become a prominent way to collect survey data. They are used in many fields including market research (Comley, 2007; Göritz, 2010; Postoaca, 2006), social research (Tortora, 2008), psychological research (Göritz, 2007), election studies (Clarke, Sanders, Stewart, & Whiteley, 2008), and medical research (Couper, 2007).

Panel-based online survey research has grown steadily over the last decade. ESOMAR has estimated that global expenditures on online research as a percentage of total expenditures on quantitative research grew from 19% in 2006 to 35% in 2012.¹ Inside Research (2012), using data from 37 market research companies and their subsidiaries, estimated that clients spent about $1.5 billion on online research in the United States during 2006 versus more than $2 billion during 2012. In Europe, expenditures were $490 million in 2006 and a little more than $1 billion in 2012.

1.2 Internet penetration and online panels

In principle, one might expect the validity of research using online panels is a function of the Internet penetration in the country being studied. The higher the household Internet penetration, the greater the chance that a panel may reflect the socio-demographic characteristics of the entire population. When Internet penetration is less than 100%, people who cannot access the Internet are presumably not missing at random. It seems likely that such individuals have socio-demographic, attitudinal, and other characteristics that distinguish them from the population of individuals who do have Internet access (Ragnedda & Muschert, 2013). In the United States, Internet access is positively associated with being male, being younger, having a higher income and more education, being Caucasian or Asian, and not being Hispanic (Horrigan, 2010). Similar patterns but with different race and ethnicity features were found in the Netherlands (Bethlehem & Biffignandi, 2012).

In the mid-1990s in the United States, Internet access at the household level was about 20% and increased to 50% by 2001 (U.S. Department of Commerce & National Telecommunications and Information Administration, 2010). By end of 2011, it had reached 75%². As Comley (2007) has pointed out, the excitement over cost savings and fast turnaround in the United States was stronger than were concerns about sample representativeness and the ability to generalize the results to a larger population. In Europe, the average individuals Internet penetration among 28 countries in 2013 was 75% (Seybert and Reinecke 2013).³ However, increasing Internet penetration does not necessarily mean that coverage bias has been decreasing. As the non-Internet population becomes smaller, it may become more different from the Internet population than when it was larger. So these figures do not necessarily assure increasing accuracy of online surveys. Further, measuring Internet penetration has become more challenging (Horrigan, 2010).⁴

1.3 Definitions and terminology

The meaning of the word “panel” in “online panel” is different from the traditional meaning of that word in the survey research world (Göritz, Reinhold, & Batinic, 2002). According to the traditional definition, “panel surveys measure the same variables with identical individuals at several points in time” (Hansen, 2008, p. 330). The main goal of a panel in this usage is to study change over time in what would be called a “longitudinal panel.” In contrast, an online panel is a form of access panel, defined in the international standard, ISO 20252 “Market, opinion and social research – Vocabulary and Service Requirements,” as “a sample database of potential respondents who declare that they will cooperate for future data collection if selected” (International Organization for Standardization, 2012, p. 1). These panels sometimes include a very large number of people (often one million or more) who are sampled on numerous occasions and asked to complete a questionnaire for a myriad of generally unrelated studies. Originally, these panels were called discontinuous access panels whose “prescreened respondents report over time on a variety of topics” (Sudman & Wansink, 2002, p. 2).⁵ Panel members can be re-sampled (and routinely are) to take part in another study with varying levels of frequency.

1.3.1 Types of online panels

There are a number of different types of online panels. The most important distinction is between probability and nonprobability panels (described below). In the latter type, there is considerable variation in how panels are recruited, how panel members are sampled, how they are interviewed, the types of people on the panel, and the kinds of data typically collected.

Some panel companies “sell” potential respondent samples to researchers but do not host surveys. In these cases, panel members selected for a study receive a survey invitation from the panel company directing them to another website where the survey is hosted. Through the use of links built into the survey questionnaire, the panel provider can track which members started the survey, which were screened out, which aborted the survey, and which completed the survey. In this model, a panelist's experience with a survey is different every time in terms of the questionnaire's look and feel.

In contrast, other panel companies host and program all the questionnaires. Panel members therefore complete questionnaires that are consistent in terms of their layout, look and feel.

Finally, some panel companies use both approaches, depending on the preference of each client.

1.3.2 Panel composition

Panels also differ in terms of the types of members. In terms of membership, there are generally four types of online panels (see also Baker et al., 2010, p. 8):

• general population panel;
• specialty panel;
• proprietary panel;
• election panel.

General population panels are the most common. These panels tend to be very large and are recruited to include the diversity of the general population, sometimes including people in hard-to-reach subpopulations. A general population panel typically is used as a frame, from which samples are drawn based on the requirements of individual studies. Examples of studies using general population panels are found in Chapters 5, 6, 8, 10, and 11 in this volume.

Specialty panels are designed to permit study of subpopulations defined by demographics and/or behavioral characteristics. Examples include Hispanics, car owners, small business owners, and medical professionals. One form of a specialty panel is a B2B panel, the goal of which is to include diverse professionals working in specific companies. Individuals are selected based on their roles in the company and the company's firmographics, meaning characteristics such as size, industry, number of employees, and annual revenues (Vaygelt, 2006). Specialty panels typically are built through a combination of recruiting from sources believed to have large numbers of people who fit the panel's specification and by screening a general population panel.

Proprietary panels are a subclass of specialty panels in which the members participate in research for a particular company. These panels are also called client panels (Poynter, 2010, p. 9), community panels, and, more recently, insight communities (Visioncritical University, 2013). They provide the opportunity for a company to establish a long-term relationship with a group of consumers – typically customers of products or services offered by the company – in a setting that allows for a mix of qualitative and quantitative research, of which surveying panels' members is just one method of research.

In election panels, people eligible to vote are recruited, and then the panel is subsampled during the months before (and perhaps after) an election to study attitude formation and change (Clarke et al., 2008). These panels resemble more traditional longitudinal panels, because each member is surveyed at each wave before and after the election. An example of an election panel is described in Chapter 4, and election panels are studied in Chapters 14 and 15.

Finally, some online panels rely on passive data collection rather than surveys. Internet audience ratings panels (Napoli, 2011) track a panelist's browsing behavior via software installed on the panelist's computer or by using other technologies, such as a router,⁶ to record the sites he or she visits, the amount of time spent on each site, and the actions taken on that site. This type of panel is discussed in detail in Chapter 17.

1.4 A brief history of online panels

1.4.1 Early days of online panels

As described above, online panels are essentially access panels moved online. Göritz et al. defined an online panel as a “pool of registered persons who have agreed to take part in online studies on a regular basis” (Göritz, Reinhold, & Batinic, 2002, p. 27). They are the natural evolution of the consumer panels (Delmas & Levy, 1998) in market research, used for decades as a sample source for mail, phone, and face-to-face surveys (Sudman & Wansink, 2002).

The attraction of online panels is threefold: (1) fast data collection; (2) promised lower cost per interview than most other methods; and (3) sampling efficiency due to ext...

Table of contents

• Cover
• Wiley Series in Survey Methodology
• Title Page
• Copyright
• Preface
• Acknowledgments
• About the Editors
• About the Contributors
• Chapter 1: Online panel research: History, concepts, applications and a look at the future
• Chapter 2: A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples
• Part I: Coverage
• Part II: Nonresponse
• Part III: Measurement Error
• Part IV: Weighting Adjustments
• Part V: Nonresponse and Measurement Error
• Part VI: Special Domains
• Part VII: Operational Issues in Online Panels
• Index
• WILEY SERIES IN SURVEY METHODOLOGY
• End User License Agreement