Population-Based Survey Experiments

Diana C. Mutz

• Publisher: Princeton University Press
• Year: 2011
• ISBN:
  9781400840489
• Topic: Scienze sociali
• Subtopic: Metodologia e ricerca nelle scienze sociali

Population-based survey experiments have become an invaluable tool for social scientists struggling to generalize laboratory-based results, and for survey researchers besieged by uncertainties about causality. Thanks to technological advances in recent years, experiments can now be administered to random samples of the population to which a theory applies. Yet until now, there was no self-contained resource for social scientists seeking a concise and accessible overview of this methodology, its strengths and weaknesses, and the unique challenges it poses for implementation and analysis.Drawing on examples from across the social sciences, this book covers everything you need to know to plan, implement, and analyze the results of population-based survey experiments. But it is more than just a "how to" manual. This lively book challenges conventional wisdom about internal and external validity, showing why strong causal claims need not come at the expense of external validity, and how it is now possible to execute experiments remotely using large-scale population samples.Designed for social scientists across the disciplines, Population-Based Survey Experiments provides the first complete introduction to this methodology.

• Offers the most comprehensive treatment of the subject
• Features a wealth of examples and practical advice
• Reexamines issues of internal and external validity
• Can be used in conjunction with downloadable data from ExperimentCentral.org for design and analysis exercises in the classroom

CHAPTER ONE

Population-Based Survey Experiments

A HYBRID METHODOLOGY FOR THE SOCIAL SCIENCES

Approaches to scientific knowledge are a bit like rabid sports rivals; often they cannot say anything nice about their own team without simultaneously disparaging the other side. At some level, they know these intense rivalries would not exist if the other team were not a worthy contender, but the positive aspects of the other side are seldom acknowledged.

Likewise, empirical social scientists tend to develop expertise either in large-scale observational methods such as survey research, or in laboratory-based experimental approaches. They then spend the rest of their careers defending their choice of that particular approach in virtually everything they publish. Each time we submit a journal article, we rehearse all of the advantages of our own methodological choice, briefly mention its weaknesses, and make the case in no uncertain terms that what we have spent our time on is worthwhile. Go team! Competition among methodological approaches is certainly implied, even if it is not explicitly stated. We do our best to defend our own in group by stressing the importance of internal validity if we have produced an experiment, or external validity if we have completed an observational study.

Fortunately, this caricature is gradually becoming less accurate, both in terms of its characterization of researchers—an increasing number of whom are trained in multiple methods—and in terms of how methodologists are focusing their attention. Although there are still many survey researchers working on improving their particular method, and many experimentalists focused on developing innovative experimental techniques, there are also methodologists paying specific attention to the problem of integrating results from experimental and observational studies. For the most part, these approaches involve applying complex statistical models to estimates of convenience sample-based experimental treatment effects in order to estimate what they might be in the population as a whole.¹ The goal of population-based experiments is to address this problem through research design rather than analyses, combining the best aspects of both approaches, capitalizing on their strengths and eliminating many of their weaknesses. The purpose of this volume is to introduce scholars and students in the social sciences to the possibilities of this approach.

Defined in the most rudimentary terms, a population-based survey experiment is an experiment that is administered to a representative population sample. Another common term for this approach is simply “survey-experiment,” but this abbreviated form can be misleading because it is not always clear what the term “survey” is meant to convey. The use of survey methods does not distinguish this approach from other combinations of survey and experimental methods. After all, many experiments already involve survey methods at least in administering pre-test and post-test questionnaires, but that is not what is meant here. Population-based survey experiments are not defined by their use of survey interview techniques—whether written or oral—nor by their location in a setting other than a laboratory. Instead, a population-based experiment² uses survey sampling methods to produce a collection of experimental subjects that is representative of the target population of interest for a particular theory, whether that population is a country, a state, an ethnic group, or some other subgroup. The population represented by the sample should be representative of the population to which the researcher intends to extend his or her findings.

In population-based survey experiments, experimental subjects are randomly assigned to conditions by the researcher, and treatments are administered as in any other experiment. But the participants are not generally required to show up in a laboratory in order to participate. Theoretically I suppose they could,³ but population-based experiments are infinitely more practical when the representative samples are not required to show up in a single location.

To clarify further, for purposes of this volume, when I use the term “experiment” in the context of population-based survey experiments, I am referring to studies in which the researcher controls the random assignment of participants to variations of the independent variable in order to observe their effects on a dependent variable. Importantly, the term “experiment” is often used far more broadly than this particular definition. For example, many classic “experiments” such as galileo’s observation of gravitational acceleration do not involve random assignment to conditions. And in the social sciences, Milgram’s famous demonstration of obedience to authority initially lacked any second group or source of comparison, although he later added these to his design.

So while there are many important experiments that do not meet this definition, I exclude these types of studies from my definition of population-based survey experiments for two reasons. First, in order to be able to make clear statements about the contribution of population-based experiments to internal and external validity, I must limit discussion to experiments for which these two ends are indeed primary goals. Establishing causality and generalizing to a defined target population are not always the goals of research, but they are central to the majority of social scientific work. In addition, the type of experimentation I circumscribe is where population-based survey experiments have the most to offer. Other kinds of experimental studies undoubtedly could benefit from more diverse subject populations as well, but given that experiments that fall outside of this definition are focused on other purposes, this methodological development is less important to these types of studies. However, when scholars want to be certain that a given relationship involves cause and effect, and that their theory may be generalized beyond a narrow pool of subjects, then this is precisely the context in which population-based survey experiments can make their biggest contribution.

Strictly speaking, population-based survey experiments are more experiment than survey. By design, population-based experiments are experimental studies drawing on the power of random assignment to establish unbiased causal inferences. They are also administered to randomly selected, representative samples of the target population of interest, just as a survey would be. However, population-based experiments need not (and often have not) relied on nationally representative population samples. The population of interest might be members of a particular ethnic group, parents of children under the age of 18, those who watch television news, or some other group, but the key is that convenience samples are abandoned in favor of samples representing the target population of interest.

The advantage of population-based survey experiments is that theories can be tested on samples that are representative of the populations to which they are said to apply. The downside of this trade-off is that most researchers have little experience in administering experimental treatments outside of a laboratory setting, so new techniques and considerations come into play, as described at length in this volume.

WHY NOW?

In one sense, population-based survey experiments are not new at all; simplified versions of them have been around at least since the early years of survey research in the United States. However, technological developments in survey research, combined with the development of innovative techniques in experimental design, have made highly complex and methodologically sophisticated population-based experiments increasingly accessible to social scientists across many disciplines. Unfortunately, aside from a few journal articles that have been contributed by early adopters of this technique,⁴ there has been no book to date addressing this topic in a comprehensive and accessible fashion.

Population-based experiments are neither fish nor fowl. As a result, the guidelines available in textbooks for each of these individual methods—for example, the considerations related to internal and external validity, the design advice, and so forth—do not address concerns specific to population-based experiments. The purpose of this volume is to fill this niche, and thus to encourage wider and more informed use of this technique across the social sciences.

Why is the population-based experimental approach emerging as a distinct methodological option only now? Two technological innovations have brought about the emergence of this method. The first was the development of technology for computer-assisted telephone interviewing (CATI). Until the development of CATI, there were rigid constraints on experimental designs executed in the context of large population samples. The classic “split-ballot” experiment allowed for variation of a single facet, whereas current technologies allow for multiple variations of multiple factors. It has become unnecessary to produce many different versions of a paper questionnaire because the software simply does this for you, with the appropriate variation of the experimental stimulus automatically popping up on the interviewer’s computer screen. This advance has allowed researchers to execute extremely complex experimental designs on large and diverse subject pools via telephone surveys.

In addition, the development of the Internet has further expanded the possibilities for population-based experiments. Although Internet-based interviewing of representative population samples is still in its infancy at this point, it is already possible to provide pictorial stimuli as well as video footage to random samples of respondents. The ability to exploit such dynamic data collection instruments has expanded the methodological repertoire and the inferential range of social scientists in many fields. Although population-based survey experiments were done by telephone or face to face long before Internet-based interviewing emerged, the Internet has greatly increased their potential.

The many advances in interviewing technology present social science with the potential to introduce some of its most important hypotheses to virtual laboratories scattered nationwide. Whether they are evaluating theoretical hypotheses, examining the robustness of laboratory findings, or testing empirical hypotheses of other varieties, scientists’ abilities to experiment on large and diverse subject pools now enable them to address important social and behavioral phenomena with greater effectiveness and efficiency.

WHO USES POPULATION-BASED EXPERIMENTS?

Population-based experiments can and have been used by social scientists in sociology, political science, psychology, economics, cognitive science, law, public health, communication, and public policy, to name just a few of the major fields that find this approach appealing. But the list does not end there. Population-based experiments have been utilized in more than twenty disciplines including psychiatry, anthropology, business, demography, African American studies, medicine, computer science, Middle Eastern studies, education, history, and even aviation studies. So long as the perceptions, behaviors, or attitudes of human beings are of interest, and the researcher’s goal is to test a causal proposition of some kind, population-based survey experiments are likely to be valuable. But they are particularly so when the study is one that would benefit from combining the internal validity of experiments with the external validity of representative population samples.

My personal interest in population-based experiments stems in part from my experiences as an avid user of this method in my own research. In graduate school I was nominally trained in both survey and experimental methods, but these were conceived of as alternative rather than synthesizable approaches. The extent to which experiments were integrated with survey work was limited to tests of alternative question wording, the kind of study that was focused on minor methodological advances rather than substantively focused survey or experimental research. Given that I was not particularly interested in survey measurement issues, this did not seem like an exciting approach to me at the time. But just a few years later, I became aware of the potential this method offered for examining substantive research hypotheses and began incorporating it regularly into my own research.

Beginning in 2001, Arthur (Skip) Lupia and I served as the original principal investigators involved in Time-sharing Experiments for the Social Sciences (TESS), a large-scale infrastructure project supported by the National Science Foundation which had as its mission to promote methodological innovation through the use of population-based survey experiments. Our inspiration for this program came from its intellectual forerunner, The Multi-Investigator Study, which was spearheaded by Paul Sniderman of Stanford University. Paul originally gathered a group of scholars within the field of political science to share time on a single telephone survey. Each team of investigators was allotted a small amount of time on the survey, and all shared the core demographic questions. The theme that tied these studies together was methodological rather than substantive. Individually, the studies would make contributions to their respective fields and subfields. But collectively, by all using experimental designs, they would demonstrate novel ways to establish causality within the context of diverse population samples.

Skip Lupia and I were fortunate to be two of the young scholars who were invited to put experiments on the Multi-Investigator Study. This platform gave us an opportunity to test our hypotheses in a new experimental context and advanced our research agendas substantially. This relatively simple, but powerful idea demonstrated the tremendous benefits of combining separately conceived and jointly implemented original studies. There were efficiencies of both time and money in this combined effort that meant that more researchers could engage in original data collection. TESS took this idea a step further by establishing an ongoing cross-disciplinary platform for research employing population-based survey experiments.

Our desire to provide this opportunity to social science writ large was the origin of the plan for Time-sharing Experiments for the Social Sciences. Population-based survey experiments could be found here and there across the social sciences even before its inception in 2001, but with TESS, we took the spirit and success of the Multi-Investigator Studies and extended them to serve a greater number of researchers across a larger number of disciplines on an ongoing basis.

The advantages of TESS are straightforward from the perspective of users: it requires a minimum investment of investigators’ time to propose a study, provides a quick turnaround time, and is free of charge as a result of generous support from the Na...

Table of contents

• Cover
• Title
• Copyright
• Contents
• List of Tables
• Preface
• Chapter One: Population-Based Survey Experiments A Hybrid Methodology for the Social Sciences
• Part I: Treatments for Population-Based Experimental Designs
• Part II: Execution and Analysis
• Part III: Situating Population-Based Survey Experiments
• Bibliography
• Index