Doing Survey Research

A Guide to Quantitative Methods

Peter M. Nardi

• Publisher: Routledge
• Year: 2018
• ISBN:
  9781351697248
• Edition: 4
• Topic: Sozialwissenschaften
• Subtopic: Soziologie

Each day we are faced with continuing claims made by media pundits, politicians, teachers, and friends, often quoting research. Consider also the numerous comments and posts on Internet blogs, Twitter, and Facebook that illustrate the confusion between opinion and factual data. How do we learn to interpret the research we hear about and read, to distinguish opinions from scientific facts, and to use this knowledge to conduct our own studies to answer the questions faced in everyday situations?

Understanding the components that go into scientific research and learning how to do research, make decisions about which statistics to use, and analyze statistical findings are goals for everyone in today's research-oriented world. Questions about the reliability and validity of data from a study or public opinion poll come up routinely and need critical review. This book contributes to achieving these objectives.

Doing Survey Research is intended for people who want to learn how to conduct quantitative studies for a project in an undergraduate course, a graduate-level thesis, or a survey that an employer may want completed. This brief, practical textbook prepares beginners to conduct their own survey research and write up the results, as well as read and interpret other people's research. It combines survey design with data analysis and interpretation.

And it is for those who need to understand and critically interpret survey research found in scholarly journals, reports distributed in the workplace, and social scientific findings presented online in the media, on a blog, or in social media postings.

Essential new updates to this edition include coverage of Big Data, Meta-Analysis, and A/B testing methodology—methods used by scholars as well as businesses like Netflix and Amazon.

New to this Fourth Edition

• Each chapter and its exercises feature updated data and illustrations from current academic and popular articles relevant to today's web-oriented students, including studies focused on topics related to social media.
• Update web site http://doingsurveyresearch.wordpress.com/
• New Coverage of Big Data (used by popular web sites like Amazon and Netflix) and the ethical issues which emerge not only about privacy, but also how it relates to the methods discussed in this book about sampling, probability, and research design.
• New coverage of meta-data, and the increasingly popular method in many professional and other settings.

1
Why We do Research

Research is formalized curiosity. It is poking and prying with a purpose.

—Zora Neale Hurston, writer

Learning Goals

In this chapter, the differences between everyday thinking and scientific thinking are discussed. An argument is made about the advantages of doing survey research and understanding various kinds of research: exploratory, descriptive, explanatory, and evaluation. The chapter concludes with a comparison of quantitative and qualitative research methods. By the conclusion, you should be able to give examples of everyday thinking, discuss the components of scientific reasoning, and describe the different types of research methods.

Remember the time you asked one of your parents to buy a particular pair of expensive running shoes, let you go to a party, or do something just because “all the kids” were? Well, the astute parent would question this and wonder if it were really true that all your friends were actually involved. “Why not collect some data first and then report back to me what percentage of your friends own those clothes or are going to that concert?” That’s what parents trained to do survey research might say!

Too often we make conclusions about entire groups of people on the basis of observations of only those we know or see around us. Or to put it more scientifically: Too often we generalize about an entire population on the basis of a nonsystematic method of collecting data from a biased sample. This is one of the major differences between everyday experience, pseudoscience, and scientific thinking. What follows is an argument for doing things more systematically and scientifically than usual, and a justification for doing quantitative survey research as one of many approaches to understanding how and why humans think and act the way we do.

Everyday Thinking

How we see the world around us is shaped by a variety of forces that include the books we read, the television and movies we see, the culture’s rules and guidelines for good behavior we repeatedly hear, and the teachings from the religious organizations and schools we attend. It is also highly influenced by our friends—the peer groups we spend a good deal of time in and from whom we learn. But so many of these social forces are linked to each other in ways that tend to reinforce our already held values and beliefs. Conclusions based on networks of friends and family members are hardly useful, then, in trying to understand how most other people think or behave.

The trick of socialization, to paraphrase writer Carlos Castaneda, is to convince us that the way we see the world is the only reality, one supported by social consensus. By experiencing more diverse cultures and meeting a wider range of people, we come to understand that to make conclusions based simply on the way our friends and family live, what they believe in, or how we were socialized, is limiting. Too often those with whom we associate are similar to ourselves in values and beliefs; we end up selectively sampling like-minded people and erroneously concluding that “everyone” experiences reality in the same way we do.

Although we seem to manage fairly well on a daily basis, our everyday experiences are often based on methods that can lead to problematic decisions with outcomes that can seriously affect our lives. Imagine if some friends told us that they were able to stop flu symptoms and prevent a recurrence of a cold simply by eating only the white filling in a chocolate cookie every day for a month. Would we rush out to buy bags of these wonderful cookies and use them to ward off the flu instead of getting a vaccine? It certainly would be more fun (and fattening) but not likely to keep us from getting the flu. What if they told us to take vitamin C or echinacea? Would we run to the store to get some, or would we critically inquire about the research that tests this general statement?

Every day we make conclusions and act on them with similarly limited information. Recall how sometimes you get a message from someone you suspected would be getting in touch: “I had a feeling you were going to text me.” Never mind that you don’t remember the number of times you did not guess accurately or that you normally contact that person around the same time every day.

Or recall when you ran to the nearest movie theater to catch a film your friends exclaimed to be the best ever and some critic called “the best film of the year,” even though it was only March. Although it would be impossible and crazy to survey a large number of people every time we need to make a decision to do something, be aware of how the process of making a choice is often not much more systematic than believing the person with the secret cure for the common cold. Yet we make such decisions because we know from experience (“empirical data”) that this particular friend’s recommendations have almost always been reliable.

What we do in everyday life is typically the result of some less scientific thinking than the procedures we would expect policy makers, neurosurgeons, or airline pilots to follow when they are in control of our well-being. In other words, what we need to consider is how ordinary thinking differs from the systematic methods needed to understand complex social behavior and attitudes. By doing so, we begin our research journey on the correct foot.

The characteristics of ordinary, everyday thinking and inquiry include

• Biased questions
• Limited sampling
• Selective attention, perception, and retention
• Pseudoscience.

Biased Questions

Did anyone ever ask, “Do you really want to go to that Justin Bieber concert?” and say it with an air of disdain? It would be difficult to answer, “Yes, I really do want to hear him sing” since the questioner was implying that the performer was so bad no one in his or her right mind could really want to go. And remember the time someone wanted to know the reasons why everyone was so dissatisfied with the workplace cafeteria, thereby assuming that “everyone” was unhappy and all we needed to do now was find those reasons? Consider the research question usually worded this way: “Why is it that I study more than the other students yet they get better grades?” and the implications already formed by the phrasing. Rhetorical questions are posed to make a point rather than to seek an answer and are certainly not suitable for scientific data gathering.

In each of these three cases, everyday nonscientific thinking leads us to ask questions that are already biased or slanted in a particular direction. The first is already providing the answer in the way it is worded. It does not allow for a full range of possible answers, so the only conclusion would be that everyone who was asked the question in this way in our survey does not want to hear Bieber sing. We’d be making an accurate statement of what was uncovered, but the answers are only as good as the way the question was framed and applicable just to the limited sample. The tendency, of course, is to generalize to all people, that is, to make conclusions about people not surveyed using the information obtained from those that were.

Similarly, the second question proceeds from a selective assumption that may not be accurate, so the only answers we get from a survey of employees’ experiences are negatively biased ones about the organization. How many times do we really ask, “Why is this workplace such a happy one?” with the same enthusiasm and inquisitiveness as we asked why people are dissatisfied with work? Or, more accurately, how often do we ask what they think about their workplace, without qualifying it as good or awful when we inquire?

With the third question about study habits, we are talking ourselves into believing something that may not be true, but asking it in this form just the same. Rarely do we, in everyday talk or thought, put it the following way: “I wonder if there is a relationship between the amount of time people study and their grades.” We often begin with a particular viewpoint and proceed to ask questions from that selective position, sometimes not even aware that we are doing so. Thus, the answers we get tend to verify what we already believe (sometimes referred to as “confirmation bias”) and give us a false sense of having found some objective and honest answers (see Box 1.1). We now have the reasons why our limited sample of employees hate the cafeteria food and falsely generalize that everyone hates the food because we never found out how many liked it in the first place.

Box 1.1 Everyday Biases

A great source for studying how people selectively see the world, make use of limited samples, generalize to people not studied, and ask questions in biased ways is to read the Letters to the Editor page in newspapers or the Comments section of online blogs and responses to controversial Facebook posts. Soon after a major shooting occurred in a suburban high school, a series of comments appeared in an online story, each with its own explanation for what happened and each likely representing the writer’s own personal values and biases. These opinions included the following possible reasons:

1. Life in white monotonous suburbia and the alienation that results
2. Republican congressmen who refuse to pass legislation controlling guns
3. Parents who neglect their kids’ depression and isolation
4. Schools that allow hate-filled speech and all sorts of taunting and bullying of kids who are different
5. Legal abortion and the theory of evolution, both of which lead to a devaluation of life.

Take a look at your local paper’s letters to the editor or the comments for an online column or Facebook post and see how many examples you can find of everyday, nonscientific reasoning.

Our daily interactions with friends, family members, and the people we encounter on a routine basis rarely require anything more scientific. Ordinary discourse contains many such poorly worded questions and biased assumptions, and the world manages to keep going. However, imagine the consequences of asking questions in these ways when trying to understand more complex and important behaviors and opinions or when attempting to develop public policies that can seriously affect people’s lives. No one should think that an educational institution, for example, can create rules to require a certain amount of study hours per week simply based on the findings of a survey that asks only why those who study less get better grades. Should the Human Resources office modify an entire organization on the basis of the results of a survey focused solely on the dissatisfaction of the employees who took the time to fill it out?

Everyday thinking, in other words, typically employs biased phrases and other nonscientific styles of framing questions when making sense of the world we live in. The results of these queries may help us decide which movie to watch, but they are not useful when it comes to making choices that affect social policies or arriving at conclusions fairly and consistently. How we can improve the way we ask questions and minimize the biases that derive from faulty wording is discussed in more detail in Chapter 4.

Limited Sampling

Most of the time, we make sense of reality by reflecting on the experiences we’ve had. It is not too difficult to figure out how our friends feel about various rules, political leaders, music, and television programs. We are generally good at assessing the climate of opinion about controversial topics among our peers and those whom we encounter regularly in our living spaces. Unfortunately, there is also the tendency to take these limited experiences and then assign them to larger groups of people.

“Everyone I know hates the food in the cafeteria”—so it must be bad. “I talked with others in the class who have good grades and they’re not reading the assignments”—so it doesn’t matter how much one studies. “No one I know liked that movie”—so it must be failing at the box office. While it may sound reasonable to make such concluding statements, the problem, of course, is that we have simply taken the opinions or measured the behaviors of those we already know and then made the assumption that they are somehow representative of “all” people: “But, Mom, everyone is going to be driving to school this year.” But if they are our friends, they are likely to be people who share our tastes and values.

Confirmation bias is this tendency to seek information and sources that support our already held opinions. We tend to avoid sampling websites, opinion articles, and news sources that contradict or challenge our values and views.

Selective Perception

People forget that most of the time our lives are constrained and limited by the social spaces we inhabit. Ethnicity, social class, gender, sexual orientation, and religion are just some of the many characteristics that provide different experiences and push us int...