Interviewer Effects from a Total Survey Error Perspective

Kristen Olson, Jolene D. Smyth, Jennifer Dykema, Allyson L. Holbrook, Frauke Kreuter, Brady T. West, Kristen Olson, Jolene D. Smyth, Jennifer Dykema, Allyson L. Holbrook, Frauke Kreuter, Brady T. West

• Publisher: Chapman and Hall/CRC
• Year: 2020
• ISBN:
  9781000064476
• Edition: 1
• Topic: Mathematics
• Subtopic: Probability & Statistics
• Index: Mathematics

Interviewer Effects from a Total Survey Error Perspective presents a comprehensive collection of state-of-the-art research on interviewer-administered survey data collection. Interviewers play an essential role in the collection of the high-quality survey data used to learn about our society and improve the human condition. Although many surveys are conducted using self-administered modes, interviewer-administered modes continue to be optimal for surveys that require high levels of participation, include difficult-to-survey populations, and collect biophysical data. Survey interviewing is complex, multifaceted, and challenging. Interviewers are responsible for locating sampled units, contacting sampled individuals and convincing them to cooperate, asking questions on a variety of topics, collecting other kinds of data, and providing data about respondents and the interview environment. Careful attention to the methodology that underlies survey interviewing is essential for interviewer-administered data collections to succeed.

In 2019, survey methodologists, survey practitioners, and survey operations specialists participated in an international workshop at the University of Nebraska-Lincoln to identify best practices for surveys employing interviewers and outline an agenda for future methodological research. This book features 23 chapters on survey interviewing by these worldwide leaders in the theory and practice of survey interviewing. Chapters include:

• The legacy of Dr. Charles F. Cannell's groundbreaking research on training survey interviewers and the theory of survey interviewing
• Best practices for training survey interviewers
• Interviewer management and monitoring during data collection
• The complex effects of interviewers on survey nonresponse
• Collecting survey measures and survey paradata in different modes
• Designing studies to estimate and evaluate interviewer effects
• Best practices for analyzing interviewer effects
• Key gaps in the research literature, including an agenda for future methodological research
• Chapter appendices available to download from https://digitalcommons.unl.edu/sociw/

Written for managers of survey interviewers, survey methodologists, and students interested in the survey data collection process, this unique reference uses the Total Survey Error framework to examine optimal approaches to survey interviewing, presenting state-of-the-art methodological research on all stages of the survey process involving interviewers. Acknowledging the important history of survey interviewing while looking to the future, this one-of-a-kind reference provides researchers and practitioners with a roadmap for maximizing data quality in interviewer-administered surveys.

Section VII

Estimating Interviewer Effects

21

Modeling Interviewer Effects in the National Health Interview Survey

James Dahlhamer, Benjamin Zablotsky, Carla Zelaya, and Aaron Maitland

CONTENTS

21.1 Introduction

21.2 Methods

21.2.1 Data

21.2.2 Measures

21.2.3 Data Structure

21.2.4 Statistical Analyses

21.3 Results

21.3.1 IICs by Question Characteristics

21.3.2 Interviewer IICs by Interviewer Characteristics

21.4 Discussion

References

21.1 Introduction

Through the administration of survey questions, interviewers have the potential to influence the amount of measurement error in the resulting estimates from these questions. Interviewers can have either a positive or a negative effect on the data. On the one hand, interviewers can clarify concepts and probe responses to collect accurate data (Conrad and Schober 2000; Schober and Conrad 1997). On the other hand, differences in how interviewers administer survey questions and probe inadequate responses can lead to systematic deviations or biases in the level of a statistic, such as a mean, across interviewers and increase the overall amount of variance associated with that statistic. For example, some interviewers may elicit more affirmative responses to a question (say, doctor-diagnosed hypertension) relative to other interviewers. While we assume that the systematic biases in doctor-diagnosed hypertension across interviewers cancel out so that they have a mean of zero, the variability caused by the systematic biases remains. This increase in variance due to interviewers is often called the interviewer effect and is measured by a statistic called the interviewer intraclass correlation coefficient, or IIC (Kish 1962). As the IIC increases, so does the variance of the sample statistic. Hence, the increase in variance due to interviewers reduces the ability to detect true differences between groups of respondents.

Interviewer effects have many potential causes, two of which include the behavior of the interviewer and the characteristics of the questions administered by the interviewer. Standardized interviewing relies on all interviewers administering questions in essentially the same manner to all survey respondents. In reality, interviewer behavior may vary within and across interviewers in ways that lead to interviewer effects. For example, interviews become shorter as the field period progresses, suggesting that interviewers may read questions faster as they conduct more interviews (Olson and Peytchev 2007; Holbrook, et al., Chapter 17, this volume; Garbarski, et al., Chapter 18, this volume). Successful interviewers are also required to master at least two often contradictory sets of skills: one being the ability to improvise in doorstep communication to recruit households that are reluctant to participate, and a second being the ability to read a script exactly as worded once the interview has started (Groves and Couper 1998). It is possible that interviewers who are better at tailoring communications at the doorstep are less likely to read questions as worded in the interview given the diverse nature of these skills. Hence, interviewers with higher cooperation rates may be prone to larger interviewer effects (Brunton-Smith, Sturgis, and Williams 2012).

Questions that are more complex, lengthy, or have higher reading levels can be more difficult to administer and harder for respondents to answer, leading to interviewer effects (Mangione, Fowler, and Louis 1992; Pickery and Loosveldt 2001). For example, longer questions include more text for interviewers to read and require respondents to hold more information in working memory, which may lead to more requests for clarification and probing. Sensitive questions that may be embarrassing for interviewers are also prone to interviewer effects (Mangione, Fowler, and Louis 1992; Schnell and Kreuter 2005). Several studies have also found that attitudinal or subjective questions appear to be more prone to interviewer effects relative to factual questions (see West and Blom 2017 for a review). In addition, questions with optional text may be administered inconsistently, with different interviewers applying different rules for when the text should be read.

An ideal design for measuring interviewer effects is an interpenetrated design, where sample units are randomly assigned to interviewers (Mahalanobis 1946). An interpenetrated design ensures that the interviewers’ workload assignments are similar in terms of the expected responses to the survey questions. Interpenetrated designs are rare in most large and ongoing in-person surveys that utilize geographic clustering to control sampling costs, because it is often infeasible and cost-prohibitive to randomly assign sample units to interviewers across a large geographic area. It is more realistic to achieve partial interpenetration within a smaller geographic area. In recent years, it has become...

Table of contents

• Cover
• Half-Title
• Series
• Title
• Copyright
• Dedication
• Contents
• Preface
• Contributors
• About the Editors
• Section I History and Overview
• Section II Training Interviewers
• Section III Managing and Monitoring Interviewers and the Survey Process
• Section IV Interviewer Effects and Interview Context and Mode
• Section V Interviewers and Nonresponse
• Section VI Interview Pace and Behaviors
• Section VII Estimating Interviewer Effects
• Index