Asking Questions

The Definitive Guide to Questionnaire Design -- For Market Research, Political Polls, and Social and Health Questionnaires

Norman M. Bradburn, Seymour Sudman, Brian Wansink

• Publisher: Jossey-Bass
• Year: 2015
• ISBN:
  9781119214762
• Edition: 2
• Topic: Negocios y empresa
• Subtopic: Investigación de mercados

Since it was first published more than twenty-five years ago, Asking Questions has become a classic guide for designing questionnaires¾the most widely used method for collecting information about people?s attitudes and behavior. An essential tool for market researchers advertisers, pollsters, and social scientists, this thoroughly updated and definitive work combines time-proven techniques with the most current research, findings, and methods. The book presents a cognitive approach to questionnaire design and includes timely information on the Internet and electronic resources. Comprehensive and concise, Asking Questions can be used to design questionnaires for any subject area, whether administered by telephone, online, mail, in groups, or face-to-face. The book describes the design process from start to finish and is filled with illustrative examples from actual surveys.

Part One
Strategies for Asking Questions

Chapter One
The Social Context of Question Asking

The precise wording of questions plays a vital role in determining the answers given by respondents. This fact is not appreciated as fully as it should be, even in ordinary conversation. For example, a colleague mentioned that he needed to pick out granite for a kitchen countertop. The only day he could make the trip was the Saturday before Labor Day. Although he called on Friday to make certain the store was open, he arrived at the store on Saturday only to find a sign on the door that said “Closed Labor Day Weekend.” When asked if he remembered what question he had asked the clerk at the store, he said, “I asked him what hours he was open on Saturday, and he replied ‘Nine to five.’”

This story illustrates the basic challenge for those who engage in the business of asking questions. It illustrates not only the importance of the golden rule for asking questions—Ask what you want to know, not something else—but also, more important, the ambiguities of language and the powerful force of context in interpreting the meaning of questions and answers. Our colleague had unwittingly asked a perfectly ambiguous question. Did the question refer to Saturdays in general or the next Saturday specifically? The clerk obviously interpreted the question as referring to Saturdays in general. Our colleague meant the next Saturday and did not think his question could mean anything else until he arrived at the store and found it closed.

In everyday life, these types of miscommunications happen all the time. Most of the time they are corrected by further conversation or by direct questions that clarify their meaning. Sometimes they only get corrected when expected behavior does not occur, as was the case when the store turned out to be closed. But the stylized form of question asking used in surveys does not often provide feedback about ambiguities or miscommunications. We must depend on pretesting to weed out ambiguities and to help reformulate questions as clearly as possible—to ask about what we want to know, not something else.

The thesis of this book is that question wording is a crucial element in surveys. The importance of the precise ordering of words in a question can be illustrated by another example.

Two priests, a Dominican and a Jesuit, are discussing whether it is a sin to smoke and pray at the same time. After failing to reach a conclusion, each goes off to consult his respective superior. The next week they meet again. The Dominican says, “Well, what did your superior say?”

The Jesuit responds, “He said it was all right.”

“That’s funny,” the Dominican replies. “My superior said it was a sin.”

The Jesuit says, “What did you ask him?”

The Dominican replies, “I asked him if it was all right to smoke while praying.”

“Oh,” says the Jesuit. “I asked my superior if it was all right to pray while smoking.”

Small Wording Changes that Made Big Differences

The fact that seemingly small changes in wording can cause large differences in responses has been well known to survey practitioners since the early days of surveys. Yet, typically, formulating the questionnaire is thought to be the easiest part of survey research and often receives too little effort. Because no codified rules for question asking exist, it might appear that few, if any, basic principles exist to differentiate good from bad questions. We believe, however, that many such principles do exist. This book provides principles that novices and experienced practitioners can use to ask better questions. In addition, throughout the book we present examples of both good and bad questions to illustrate that question wording and the question’s social context make a difference.

Loaded Words Produce Loaded Results

Suppose a person wanted to know whether workers believed they were fairly compensated for their work. Asking “Are you fairly compensated for your work?” is likely to elicit a very different answer than asking “Does your employer or his representative resort to trickery in order to defraud you of part of your earnings?” One would not be surprised to find that an advocate for improving the situation of workers asked the second question. Clearly the uses of words like “trickery” and “defraud” signal that the author of the question does not have a high opinion of employers. Indeed, this was a question asked by Karl Marx on an early survey of workers.

Questionnaires from lobbying groups are often perceived to be biased. A questionnaire received by one of the authors contained the following question: “The so-called ‘targeted tax cuts’ are a maze of special interest credits for narrow, favored groups. Experts agree the complex, loophole-ridden tax code makes it easy for Big Government liberals to raise taxes without the people even realizing it. Do you feel a simpler tax system—such as a single flat rate or a national sales tax with no income tax—would make it easier for you to tell when politicians try to raise your taxes?”

Even an inexperienced researcher can see that this question is heavily loaded with emotionally charged words, such as “so-called,” “loophole-ridden,” and “Big Government liberal.” The authors of this questionnaire are clearly interested in obtaining responses that support their position. Although the example here is extreme, it does illustrate how a questionnaire writer can consciously or unconsciously word a question to obtain a desired answer. Perhaps not surprisingly, the questionnaire was accompanied by a request for a contribution to help defray the cost of compiling and publicizing the survey. Surveys of this type, sometimes called frugging (fundraising under the guise) surveys, are often primarily intended to raise funds rather than to collect survey information. The American Association for Public Opinion Research has labeled fundraising surveys deceptive and unethical, but they are unfortunately not illegal.

Wording questions to obtain a desired answer is not the only type of problem that besets survey authors. Sometimes questions are simply complex and difficult to understand. Consider this example from a British Royal Commission appointed to study problems of population (cited in Moser and Kalton, 1972): “Has it happened to you that over a long period of time, when you neither practiced abstinence, nor used birth control, you did not conceive?” This question is very difficult to understand, and it is not clear what the investigators were trying to find out.

The Nuances of Politically Charged Issues

Yet even when there are no deliberate efforts to bias the question, it is often difficult to write good questions because the words to describe the phenomenon being studied may be politically charged. The terms used to describe the area of concern may be so politically sensitive that using different terms changes the response percentages considerably. A question asking about welfare and assistance to the poor from the 1998 General Social Survey (Davis, Smith, and Marsden, 2000) produced quite different opinions.

We are faced with many problems in this country, none of which can be solved easily or inexpensively. I am going to name some of these problems, and for each one I’d like you to tell me whether you think we’re spending too much money on it, too little money, or about the right amount. Are we spending too much money, too little money or about the right amount on …

	“Welfare” (N = 1,317)	“Assistance to the Poor” (N = 1,390)
Too little	17%	62%
About right	38%	26%
Too much	45%	12%
Total	100%	100%

Not all wording changes cause changes in response distributions. For example, even though two old examples of questions about government responsibility to the unemployed were worded differently, 69 percent of respondents answered “yes.” Perhaps this is because the questions were fairly general. One question, from a June 1939 Roper survey, asked, “Do you think our government should or should not provide for all people who have no other means of subsistence?” (Hastings and Southwick, 1974, p. 118).

A differently worded question, this one from a Gallup poll of January 1938, asked, “Do you think it is the government’s responsibility to pay the living expenses of needy people who are out of work?” (Gallup, 1972, p. 26).

Respondents are less likely to agree as questions become more specific, as illustrated by three Gallup questions from May to June 1945:

Do you think the government should give money to workers who are unemployed for a limited length of time until they can find another job? (Yes 63%)

It has been proposed that unemployed workers with dependents be given up to $25 per week by the government for as many as 26 weeks during one year while they are out of work and looking for a job. Do you favor or oppose this plan? (Favor 46%)

Would you be willing to pay higher taxes to give people up to $25 a week for 26 weeks if they fail to find satisfactory jobs? (Yes 34%)

Note that introducing more details—such as specifying actual dollars, specifying the length of the support, and reminding respondents that unemployment benefits might have to be paid for with increased taxes—changed the meaning of the question and produced a corresponding change in responses. In later chapters we will discuss in more detail how wording affects responses, and we will make specific recommendations for constructing better questionnaires.

Questioning as a Social Process

A survey interview and an ordinary social conversation have many similarities. Indeed, Bingham and Moore (1959) defined the research interview as a “conversation with a purpose.” The opportunity to meet and talk with a variety of people appears to be a key attraction for many professional interviewers. By the same token, a key attraction for many respondents appears to be the opportunity to talk about a number of topics with a sympathetic listener. We do not know a great deal about the precise motivations of people who participate in surveys, but the tenor of the evidence suggests that most people enjoy the experience. Those who refuse to participate do not refuse because they have already participated in too many surveys and are tired; characteristically, they are people who do not like surveys at all and consistently refuse to participate in them or have experienced bad surveys.

Viewing Respondents as Volunteer Conversationalists

Unlike witnesses in court, survey respondents are under no compulsion to answer our questions. They must be persuaded to participate in the interview, and their interest (or at least patience) must be maintained throughout. If questions are demeaning, embarrassing, or upsetting, respondents may terminate the interview or falsify their answers. Unlike the job applicant or the patient answering a doctor’s questions, respondents have nothing tangible to gain from the interview. Their only reward is some measure of psychic gratification—such as the opportunity to state their opinions or relate their experiences to a sympathetic and nonjudgmental listener, the chance to contribute to public or scientific knowledge, or even the positive feeling that they have helped the interviewer. The willingness of the public to participate in surveys has been declining in recent years for many reasons, one of which is the tremendous number of poor and misleading surveys that are conducted. It is therefore doubly important for the survey researcher to make sure that the ...

Table of contents

• Cover
• Table of Contents
• Title
• Copyright
• Dedication
• Preface
• The Authors
• Part One: Strategies for Asking Questions
• Part Two: Tactics for Asking Questions
• Part Three: Drafting and Crafting the Questionnaire
• Bibliography and Recommended Readings
• Glossary
• Index
• Appendix A: List of Academic and Not-for-Profit Survey Research Organizations
• Appendix B: Illinois Liquor Control Commission: College Student Survey
• Appendix C: Faculty Retention Survey
• Appendix D: Kinko’s: Open-ended Service Satisfaction Survey
• End User License Agreement