Improving Public Opinion Surveys

Interdisciplinary Innovation and the American National Election Studies

John H. Aldrich, Kathleen M. McGraw, John H. Aldrich, Kathleen M. McGraw, John Aldrich, Kathleen McGraw

• Publisher: Princeton University Press
• Year: 2011
• ISBN:
  9781400840298
• Topic: Politics & International Relations
• Subtopic: Political Campaigns & Elections
• Index: Politics & International Relations

The American National Election Studies (ANES) is the premier social science survey program devoted to voting and elections. Conducted during the presidential election years and midterm Congressional elections, the survey is based on interviews with voters and delves into why they make certain choices. In this edited volume, John Aldrich and Kathleen McGraw bring together a group of leading social scientists that developed and tested new measures that might be added to the ANES, with the ultimate goal of extending scholarly understanding of the causes and consequences of electoral outcomes.
The contributors--leading experts from several disciplines in the fields of polling, public opinion, survey methodology, and elections and voting behavior--illuminate some of the most important questions and results from the ANES 2006 pilot study. They look at such varied topics as self-monitoring in the expression of political attitudes, personal values and political orientations, alternate measures of political trust, perceptions of similarity and disagreement in partisan groups, measuring ambivalence about government, gender preferences in politics, and the political issues of abortion, crime, and taxes.
Testing new ideas in the study of politics and the political psychology of voting choices and turnout, this collection is an invaluable resource for all students and scholars working to understand the American electorate.

PART 1

The American National Election Studies: The “Gold Standard” for Survey research

CHAPTER 1

Introduction to the Volume

John H. Aldrich and Kathleen M. McGraw

PUBLIC OPINION SURVEYS provide insights into a very large range of social, economic, and political phenomena. In this book, we look at the survey itself as the means by which the scholarly community seeks to understand those phenomena. A survey can be understood as a collection of ideas at the forefront both of what the scholarly community believes to be important for understanding opinion and behavior and of how that community thinks those ideas may best be measured and evaluated. In this volume, we have the opportunity to report the results of testing a large variety of the best new ideas that the scholarly community thinks should be considered for inclusion in such a survey. These ideas were proposed for addition to the American National Election Study (ANES) and tested in a pilot study conducted in 2006.

Why should the reader be interested in such reports? The answer is that these reports are a major part of why the ANES is considered the gold standard among public opinion surveys. These reports constitute The perhaps the most intellectually stimulating part of the gold standard, since they include the justification for the ideas that constitute the survey and, thus, the basis for the intellectual advances that the survey will offer. In addition, they reflect a return to a deep commitment to intellectual openness and an interdisciplinary search for the best new ideas and their translation into measurable projects. Survey research is in the midst of a particularly exciting time as new technologies are being added to these new ideas to create major ferment in the means and methods of survey research. Tat the ANES remains the gold standard is not inevitable, except by virtue of its commitment and resolve to be so. Let us begin by indicating how and why the ANES achieved its status and how this project seeks to extend that standard.

THE ANES

First used in 1948, the ANES has been in the field in every presidential election and nearly every congressional election since. It is supported by the National Science Foundation as one of its three “big social science” projects (the other two are the General Social Survey [GSS] and the Panel Study of Income Dynamics [PSID]). Each project is a national resource for the scholarly community, and each covers a large range of topics in the social and behavioral sciences. Each is, in its area, the gold standard for survey-based research.

The ANES has a special place in even this exalted category. Its longevity alone is a remarkable strength, especially when coupled with the serious attention given to maintaining as much continuity in measurement as the science makes possible. Sixty years of measuring public opinion and voting behavior has made possible the compilation of time-series analyses that are now starting to show real insights into, and to change how we view, campaigns and elections. The relatively recent decision by the NSF that the surveys will continue to be conducted face-to-face was based on its decision, reached after careful consultation with survey experts from across and even beyond the nation, that such a survey, expensive though it may be, is in fact the gold standard (a phrase that came out of this evaluation process) for doing a survey as well as it can be done (for transcripts of a 2003 workshop at the NSF that helped underpin these decisions, see http://www.nsf.gov/sbe/ses/polisci/reports/aebtrans21503.pdf).

There is one other reason why the ANES is an especially appropriate place to evaluate new ideas. The study of public opinion and presidential elections itself has several virtues. Voting is the one decision that the entire nation is asked to reach by the same day. All citizens are at least potentially exposed to the same phenomena—the same events of the last four years, the same candidates and parties standing for the presidency, many of the same campaigns, the same ads, the same media coverage, etc. Not only is the presidential election a common decision point, but presidential election campaigns are sufficiently complex and diverse enough to provide a rich tapestry on which to examine attitude formation, evaluation, decision making and choice, and many other topics of interest to the social and behavioral sciences. In this sense, the ANES survey instrument is genuinely the product of, and contributor to, interdisciplinary scholarship. The scholars contributing to this book come from nearly every social and behavioral science discipline and beyond themselves.

These strengths of the ANES survey instrument impose severe constraints as well. The instrument is limited by time, that is, the time that respondents are willing to give to responding to a survey. The survey's longevity and its importance to the scholarly community have made continuity of the sixty-year time series valuable, but have also limited the time available for innovations. And the centrality of national elections means that ideas that have relevance for understanding turnout and vote The choice are given priority over other equally interesting but less immediately relevant ideas.

The high value of space and time on the survey requires high hurdles for access. The ANES became a national resource supported by the NSF in the early 1970s. While the project's principal investigators (PIs) necessarily retain final authority, the Board of Overseers provides the basis for peer review of proposals for inclusion. From the 1970s through the 1990s, the board selected a series of topics and solicited the user community for proposals. Successful proposals were then included in pilot studies—the ANES website reports on ten such studies conducted from 1979 to 1997 (see http://www.electionstudies.org/overview/rdd_tab1.htm). The board and PIs then selected new instrumentation based in part on the results of such proposals and their pilot-study testing. As part of the process for opening up the 2008 study to innovations, the NSF provided funding for a new mechanism for the input of ideas to test in a pilot study in 2006. The chapters that follow result from that new mechanism and provide the concrete evidence of the ANES as gold standard.

THE GOLD STANDARD IN SURVEY RESEARCH

What might being the gold standard in survey research entail? We (like others) have numerous criteria in mind. First, the basic notion of a survey is that it has high external validity. In particular, the advantage of surveys over other modes of empirical research is that one can make inferences about a (well-defined) universe from the observations were drawn. Tat means that the first great strength of a survey is that it is based on a probability sampling process. It is because of this grounding that one can make statistical inferences back to the universe one wants to study. This presumes, for example, that no matter what the other advantages of many online surveys may be, if they are not drawn from a universe with known probability, no matter how carefully one designs weights, it is extremely difficult for them to have a strong basis for inference to the population of interest.

A second great virtue of the procedures followed by the ANES is that they foster a high response rate. Many of the time-dependent surveys, such as those done by the media to provide a basis for their reporting of elections, have very low response rates (less than 20%). Indeed, declining response rates have become a major concern for all survey research, whether it is done by the government, the academy, the media, or campaigns. The ANES works very hard, innovatively, and indeed expensively to maintain a high response rate. Without a sufficiently high response rate, the advantages of probability-based sampling are lost.

The third component of the gold standard is to balance the openness of the instrument to the appropriate intellectual community with the maintenance of high standards of quality. This pairing often requires a balancing of objectives. The “wiki” approach, associated with the Internet, has the virtue of great openness, but as a result, it places a great premium on methods to maintain standards, even of truthfulness (something that Wikipedia has famously had to confront, often after embarrassing revelations). The ANES, when conducted exclusively by and through the trailblazing efforts of Angus Campbell and others at the University of Michigan, attracted some criticism for appearing to be closed. It was indeed the genius of their collaboration with the NSF to create the Board of Overseers, which, combined with regular NSF funding, led to the achievement of intellectual openness and high quality standards. The idea was to openly solicit ideas from the broadest possible audience while subjecting the resulting proposals to that central strength of scientific legitimacy, peer review. Among the early results of this collaboration with the NSF was the development of the 1978 study as, for the first time, a true congressional election study (involving both a change of sampling base and of content). The first pilot study was conducted in 1979. Tat survey grew from a call for proposals, peer review of those proposals, conferences to discuss those ideas, and then a pilot study to test the proposed new instrumentation. Only then would such instrumentation be considered for inclusion in the regular, or time series, studies. Pilot studies were conducted ten times from 1979 to 2000. Funding difficulties ended both the regular midterm election studies and the pilot studies. In preparation for the 2008 ANES Time Series Study, the NSF recompeted the project and awarded the winning proposal (submitted by Jon Krosnick and Arthur Lupia), among other things, the opportunity to conduct another pilot study.

Technological advances made possible new means of securing the newest ideas and conducting peer review. The PIs created the “Online Commons,” an Internet-based location for the presentation and discussion of ideas, where many scholars presented their proposals. The PIs also made a concerted effort to solicit ideas from an interdisciplinary base. The Board of Overseers (on which we served) included an unusually large and diverse interdisciplinary set of members. The PIs also worked with the Social Science Research Institute at Duke University and the American Psychological Association to hold a conference where leading psychologists discussed recent advances in psychological research and their potential for integration into election surveys. These forums for the creation of new ideas ultimately led to a set of proposals for inclusion of instrumentation in the 2006 Pilot Study, which were peer-reviewed by the ANES Board of Overseers after the larger intellectual community had discussed them on the Online Commons.

That the ANES (and the GSS and the PSID) have achieved gold standard status is due precisely to the resource-intensive complex of procedures they employ, including the holding of very expensive face-to-face interviews, enabling the continuation of probability-based sampling with high response rates, and maintaining a commitment to intellectual openness and the highest standard of peer review.

THE ANES, THE GOLD STANDARD, AND THE FUTURE OF SURVEY RESEARCH

The studies that we have compiled here are the product of these complex sets of procedures, which were designed to elicit as many exciting new ideas as possible and to submit them to rigorous peer review. Those ideas that survived this process then entered the pilot study process, which transformed them from ideas on paper to new instrumentation in the pilot survey. This, of course, was embedded in a process designed to test the efficacy of the new instrumentation. The results from that process serve as the basis for the reports presented here.

The future of surveys is uncertain. Changing technologies, on the one hand, present new opportunities for conducting surveys. On the other hand, they also present challenges, such as ensuring external validity, while the expense of conducting traditional, face-to-face surveys escalates. Just how the future of survey research will unfold is, thus, quite uncertain.

What does remain certain is that features of the gold standard for survey research need to be maintained. Surveys remain viable to the extent that they retain their ability to genera The inferences about populations of interest—actual and likely voters, for example. Procedures that retain response rates sufficiently high to maintain the surveys’ inferential basis will continue to be essential— and likely increasingly difficult to attain.

The aspect of the gold standard that forms the basis of this book, however, will remain perhaps the most innovative component of that standard, and perhaps the one most vital to maintain. Namely, a survey will have a claim for setting the standard for survey research in the academy only so long as it retains openness to new ideas combined with peer review to maintain the highest of intellectual standards.

We turn now, therefore, to see just where this dimension of the gold standard has led the academy today. We begin by having the PIs for the 2008 ANES, Jon Krosnick and Arthur Lupia, describe the logic underlying, and the strategy for implementing, their innovations for the incorporation of new ideas into the ANES study and into the scholarly community more generally. We conclude by having the PIs return to the stage to discuss how they decided which among these ideas would be included in the pre-post time series survey of 2008.

The heart of the book consists of seventeen chapters in which scholars from a variety of social and behavioral science disciplines define and justify their ideas that collectively made up a large portion of the 2006 Pilot Study and present their analyses and conclusions about how those ideas fared in that study.¹ These chapters are organized along substantive themes, and we provide a brief introduction to each new section of the book to set the stage for the contributions that follow. As is appropriate for a pilot study, all the contributions revolve around new instrumentation that, in theory, could improve our understanding of voting and turnout. In many cases, the authors make a convincing argument that the new instrumentation advances our understanding of political outcomes. In other cases, the authors conclude that the current ANES instrumentation is preferable to the proposed new measures. In either case, the research advances our understanding of best practices in the measurement of core constructs and advances our understanding of the dynamics of voting and turnout. Finally, we should note The that the 2006 Pilot Study, like all ANES surveys, is freely available for further analyses. As you read the following chapters, questions may arise that you would like to analyze. Further, the studies reported here and available online have been analyzed only in their component module forms. We look forward to additional analyses of these data that cut across these modular ideas.

1 While these seventeen empirical analyses cover a wide range of topics and proposed new instrumentation, they do not cover the full range of ideas examined in the 2006 Pilot Study. These additional topics include the following: a single question tapping individuals’ beliefs about the stability or malleability of character (Module 1); a five-question Need for Closure scale (Module 3); a single question to measure individuals’ beliefs in a just world (Module 4); three questions about ability to borrow money, aimed at capturing individuals’ levels of financial security (Module 8); a battery of questions measuring pessimism and optimism about the individual's and the nation's future (Module 12); a social networks battery (Module 13); a battery of questions assessing beliefs about the efficacy of the president and Congress (Module 16); tolerance of Muslims (Module 21); two questions about death and “end times” beliefs (Module 29). In addition, many ideas that were proposed were unable to be included in the pilot study itself. Interested readers can examine the archives of the ANES Online Commons (http://www.electionstudies.org/onlinecommons/overview.htm) for more information.

CHAPTER 2

The American National Election Studies and the Importance of New Ideas

Jon A. Krosnick and Arthur Lupia

MILLIONS OF AMERICANS participate in elections for the president and Congress of the United States. These elections draw great attention from every corner of the nation. In the months leading up to each Election Day, news and opinions about candidates are the focus of numerous newspaper headlines, television programs, and websites. Questions about many election-related topics are brought to the fore. Some topics remain focal for the duration of the campaign. Others fade into obscurity. The influence of all such news extends beyond the media and into many workplace discussions and dinner-table conversations. After the votes are cast and the confetti falls from the ceilings of partisan gatherings in large cities and small towns, the winners celebrate, the losers ruminate, and debates about the meaning of it all reach a fever pitch. In every case, the results of our federal elections are recorded as high-water marks in the evolving history of our nation.

Why do people act as they do in federal elections? Why do some people vote, while others choose not to do so? Of those who vote, why do they vote The for the candidates that they choose? There is great interest in the answers to these questions. How myriad observers understand what happened during each U.S. presidential and congressional election fuels many assessments of the health of the world's most visible democracy. Observations about stability and change in how different kinds of Americans feel about political candidates, political parties, and major issues provide a clearer understanding of the continuing relationship between citizens and government. Such clarity can reveal which aspects of policy voters appreciate and which aspects make them angry, anxious, or ambivalent. In popular and individual perceptions of, and feelings toward, election-related phenomena lies the legitimacy of a country's democratic experience.

It is critical for a nation that cherishes its democratic legitimacy to seek credible measures of how citizens think, feel, and act in electoral contexts. The reason is that following every election, we see many attempts to convince broad audiences that the Election Day actions of the American people are evidence of the political, moral, or ethical position that the author wishes to advance. After every federal election, inward-oriented political interests have strong incentives to try to spin these electoral moments toward fictional narratives that support their preferred worldviews. If credible measures of citizen attitudes and opinions, and analyses of them, are not available, there will be no strong evidentiary basis for differentiating propagandistic tall tales from empirically defensible and logically coherent readings of electoral history. To the extent that a nation benefits from having its history constrained by evidence and logic, the availability of credible data about voter attitudes and opinions is a public good of significant value.

During recent election cycles, many polls were described as reflecting the public's views. Many of these surveys, however, had severe limitations that limited their potential to reflect public opinion accurately. For example, nearly all public polls involve very short interviews. They typically focus on just a few issues that are prominent in public discussion at the time of the poll. Moreover, findings from these polls are reported in relatively simple, descriptive ways. While it is common for reporters or pollsters to claim that these surveys clarify processes underlying the latest horse-race numbers, the surveys themselves are typically insufficient to support the kinds of statistical analyses necessary to validate such causal claims. Indeed, the vast majority of polls reported by the news media are conduct...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• Contents
• Part 1 - The American National Election Studies: The “Gold Standard” for Survey Research
• Part 2 - Individual Predispositions
• Part 3 - Political Orientations and the Media
• Part 4 - Perceptions of Political Institutions and Groups
• Part 5 - Political Issues
• Part 6 - Concluding Thoughts and Future Directions
• List of Contributors