Survey Bias

7 Key excerpts on "Survey Bias"

• 

  eBook - ePub

  Polling America

  An Encyclopedia of Public Opinion [2 volumes]

  • Richard L. Clark, Kelly N. Foster, Samuel J. Best, Benjamin Radcliff, Richard L. Clark, Kelly N. Foster, Samuel J. Best, Benjamin Radcliff, Richard L. Clark, Kelly N. Foster, Samuel J. Best, Benjamin Radcliff(Authors)
  • 2020(Publication Date)
  • Greenwood

    (Publisher)

  For example, if exactly 50 percent all voters will vote for Candidate X but we employ a survey method that consistently and systematically excludes 10 percent of Candidate X’s supporters while excluding none of those who do not support X, then surveys using that method would produce biased results. Random variation would cause different administrations of a survey with this biased methodology to produce different results; random variation even makes it possible that a survey with this method would produce an estimate that was exactly correct. However, on average, surveys with this biased method would estimate Candidate X receiving 45 percent of the vote, or 10 percent less than the true value of this statistic. Although there will still be variability in the results of surveys, the variability would center on a biased value of 45 percent of the vote, rather than the accurate value of 50 percent.

  Bias in surveys, as well as variation, can come from a number of sources. Survey methodologists typically think of survey error in two ways: as total survey error and as mean squared error (MSE). Both of these concepts consider the commonly discussed sampling error and myriad other possible sources of error introduced into survey estimates through the survey research process. Total survey error is the difference between the estimate for a statistic measured in a particular survey and the true value of that statistic. The MSE considers not only error associated with the variability of survey statistics but also systematic bias.

  The MSE is a concept and measure commonly used to encompass both the systematic error or bias in a survey and the variable error or variance in a survey. The MSE is typically measured as the sum of the squared bias in a survey and the variance. Thus, in cases where there is no bias, the MSE is equal to the variance. The MSE applies to any specific statistic measured in a survey and might be different for different survey questions contained in the same public opinion poll. Since survey researchers can rarely know the true value of a statistic, and rarely know the bias involved in a given method, the MSE cannot typically be calculated or reported.

  A variety of sources can influence both bias and variance in a survey. Typically, these include both sampling error and nonsampling error. Biemer and Lyberg delineate different potential sources of survey error and their potential impact on both variable error and systematic error. Specification error, frame error, and nonresponse error all have a relatively small impact on the measured variability of survey results but have a potentially high risk for producing systematic bias. In contrast, sampling error has minimal risk of bias but has a potentially high influence on variability. Measurement error and data processing error have a potentially high impact on both variable error and systematic bias. Under this formulation, the MSE formula can be expanded to read MSE = (BSPEC + BNR + BMEAS + BDP )2 + VARSAMP + VARMEAS + VARDP

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Survey Methodology

  • Robert M. Groves, Floyd J. Fowler, Mick P. Couper, James M. Lepkowski, Eleanor Singer, Roger Tourangeau(Authors)
  • 2011(Publication Date)
  • Wiley

    (Publisher)

  Thus, we again use the notion of a “trial” to denote the single application of a measurement. If response deviations described above are systematic, that is, if there is a consistent direction of the response deviations over trials, then “response bias” might result. “Bias” is the difference between the expected value (over all conceptual trials) and the true value being estimated. Bias is a systematic distortion of a response process. There are two examples of response biases from our example surveys. In the NSDUH, independent estimates of the rates of use of many substances asked about, including cigarettes and illegal drugs, suggest that the reporting is somewhat biased; that is, that people on average tend to underreport how much they use various substances. Part of the explanation is that some people are concerned about how use of these substances would reflect on how they are viewed. It also has been found that the rates at which people are victims of crimes are somewhat underestimated from survey reports. One likely explanation is that some individual victimizations, particularly those crimes that have little lasting impact on victims, are forgotten in a fairly short period of time. Whatever the origins, research has shown that survey estimates of the use of some substances and of victimization tend to underestimate the actual rates. Answers to the survey questions are systematically lower than the true scores; in short, they are biased. In statistical notation, we note the average or expected value of the response over trials as E t (Y it), where, as before, t denotes a particular trial (or application) of the measurement. Response bias occurs when In addition to systematic underreporting or overreporting that can produce biased reports, there can be an instability in the response behavior of a person, producing another kind of response error

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Survey Development

  A Theory-Driven Mixed-Method Approach

  • Tony Chiu Ming Lam, Kathy Ellen Green(Authors)
  • 2023(Publication Date)
  • Routledge

    (Publisher)

  minimizing errors in participant recruitment and in solicitation of responses. This understanding is imperative for unraveling the roots of good practices and errors consciously or unconsciously introduced by the researchers and participants; consequently, more precise and effective strategies can be devised and implemented to minimize survey errors. When designing and implementing a survey, researchers need to consider which and if any of these theories apply in their specific study so that they can devise strategies to entice participation of individuals and to enhance their willingness to respond not only completely but truthfully as well.

  In the measurement literature, a plethora of biases in self-reporting has been identified. Each bias is a construct supported by theory, e.g., item order effect, satisficing, and socially desirable response. So, when we discuss theories of survey errors, we are in fact discussing response biases that are due to the measurement instrument, participant, or context. For example, the Stereotype Threat Theory (STT; Keller, 2007 ; Steele, 1997 ) has been proposed to explain and predict the impact of stereotype bias or negative stereotypic expectations on test performance of individuals who are stereotyped. In this chapter, we have included well-known biases or errors that are germane to survey research.

  Now that we have some understanding of the sources of survey errors and the theories of response and survey errors, we are ready to present a detailed account of our framework of sources of errors that can accommodate errors specific to survey research. Such a framework should help researchers conceptualize survey errors, inquire about what kind of specific errors might flourish in their research process, and develop strategies to combat these errors.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Handbook of Web Surveys

  • Silvia Biffignandi, Jelke Bethlehem(Authors)
  • 2021(Publication Date)
  • Wiley

    (Publisher)

  Chapter Five Errors in Web Surveys

  5.1 Introduction

  Survey researchers have control over several different aspects of a survey. With the proper choice of a sampling frame, a sampling design, and an estimation procedure, they can obtain precise estimators of population characteristics. Unfortunately, not everything is under control. Survey researchers have to confront with various phenomena that may have a negative impact on the quality and therefore the reliability of the survey outcomes. Some of these disturbances are almost impossible to prevent. Efforts will then have to aim at reducing their impact as much as possible. Nevertheless, notwithstanding all efforts to eliminate or reduce problems, final estimates of population parameters are biased. Estimates differ from the true value. This difference is the total error of the estimate.

  Errors may occur in surveys whatever the mode of data collection, but some errors are more likely to occur in some types of surveys. For an example, it makes a difference whether interviewers conduct interviews or the respondents complete the questionnaires themselves. The focus in this chapter is on errors in web surveys. Sometimes the impact of a specific type of error in a web survey is compared with other types of surveys.

  Sources of error increase the uncertainty with respect to the correctness of estimates. This uncertainty can manifest itself in the distribution of an estimator in two ways: (1) it can lead to a systematic deviation (bias) from the true population value, or (2) it can increase the variation around the true value of the population parameter.

  Let be an estimator for the population mean . Chapter 4 discusses the properties of a good estimator. One is that an estimator must be unbiased. This means its average value over all possible outcomes must be equal to the population mean to be estimated:

  (5.1)

  An estimator may be biased due to survey errors. Suppose one of the objectives of a survey is to estimate the average amount of time per day people spend on the Internet. If a web survey is conducted for this, people without Internet access will not be in the sample. Since these people do not spend time on the Internet, the estimate will be too high. The estimator has an upward bias. Denote this bias of the estimator

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Research Training for Social Scientists

  A Handbook for Postgraduate Researchers

  • Dawn Burton, Dawn Burton(Authors)
  • 2000(Publication Date)
  • SAGE Publications Ltd

    (Publisher)

  Non-response errors These are generated by inadequacies in the research process. Non-response errors can result from subjects who refuse to provide data, those who are asked to provide data but who are unable to undertake the required task and, finally, those whom the data collection procedures did not reach; therefore not providing individuals with an opportunity to respond. High levels of non-response can have serious implications for inference; the extent to which findings from the sample can be generalized to a wider population. For example, if a survey yields response rates of 10–25 per cent of a sample, the final sample bears little relationship to the original sample as those responding are self-selected. Whereas if a response rate of 95 per cent is achieved in a probability sample, the final sample is still very similar to the population as a whole. Fowler (1993) suggests that there are different types of bias associated with different methods of data collection. For example, mail questionnaires are more likely to be returned by individuals who are particularly interested in the project. As a result low response rates can be significantly biased in ways that are directly related to the purpose of the research. In addition, better educated people tend to send back mail questionnaires than those who are less well educated. Bias resulting from telephone and personal surveys tends to focus on availability. For example, if a survey was undertaken on weekdays between 9am and 5pm the likely scenario is that individuals without jobs (unemployed, retired and housewives) would be over-represented in the sample.

  There also tend to be lower response rates in cities than in suburbs and rural areas, which is largely a result of the disproportionate number of single people living in cities who can be difficult to contact; the nature of housing, specifically the greater number of individuals who live in flats, which makes access difficult; and the fact that some areas in the centre of cities are undesirable places to conduct fieldwork, especially at night. Researchers cannot assume that non-response is unbiased and it is not always obvious in what ways non-response is biased, which makes it difficult to counteract. The ideal solution is to reduce the non-response by amending data collection techniques such as varying methods and undertaking systematic follow-ups.

  Processing errors These are coding and data entry errors. Constructing a comprehensive coding book and having in place quality control procedures should reduce such errors.

  Sampling errors

  Sampling errors directly relate to the sample selection process. Sampling errors can be divided into two main groups: systematic error (or bias ), and random error

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Sampling Essentials

  Practical Guidelines for Making Sampling Choices

  • Johnnie N. Daniel(Author)
  • 2011(Publication Date)
  • SAGE Publications, Inc

    (Publisher)

  Another approach to minimize the effects of clustered frame bias is to include all the elements within a selected clustered listing. The probability of the selection of a cluster would apply to all elements in it. This will probably create nonresponse as not every element in the selected unit is likely to agree to participate in the study. Moreover, the responses of the elements who are first to participate may influence the responses of the elements who participate later. If the clusters are of different sizes, it may be difficult to control the total sample size. Moreover, this solution would increase the effective sample size and thereby have an impact on the budget for the study.

  Selection Bias

  What Is Selection Bias?

  Selection bias is bias due to systematic differences in the characteristics of population elements that are selected to be included in the study and population elements that are not selected. There is no selection bias when one takes a census. On the other hand, selection bias is very much a part of sampling. How much will depend upon the specific sampling procedures that are used. Non-probability sampling (e.g., availability sampling and volunteer sampling) is likely to have a great deal of selection bias. Probability sampling will also have selection bias when elements are selected with unequal probabilities and when interviewers improperly select a person to be interviewed from a household that has more than one person in the target population. If an interviewer fails to make the selection a random selection either by mistake or on purpose, selection bias will have occurred. Moreover, an interviewer creates selection bias if a randomly selected person who is not at home is substituted with someone in the household who is willing and able to respond.

  How Can Selection Bias Be Minimized?

  Choosing to take a census will minimize selection bias. It may also be minimized via using probability sampling with equal probability selection, effective training and supervision of data collectors, and the implementation of comprehensive quality control procedures.

  BIAS IN COLLECTING DATA

  Data collection bias is disparity between the true characteristics of the elements selected to be included in a study and the study’s observation or measurement of those characteristics. There are two major forms of data collection bias: nonresponse bias and response bias. Both types of bias are important in deciding whether to take a census or a sample.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Handbook of Web Surveys

  • Jelke Bethlehem, Silvia Biffignandi(Authors)
  • 2011(Publication Date)
  • Wiley

    (Publisher)

  Chapter Four Errors in Web Surveys 4.1 Introduction

  Survey researchers have control over many different aspects of a survey. With the proper choice of a sampling frame, a sampling design, and an estimation procedure, they can obtain precise estimators of population characteristics. Unfortunately not everything is under control. Survey researchers may be confronted with various phenomena that may have a negative impact on the quality and, therefore, the reliability of the survey outcomes. Some of these disturbances are almost impossible to prevent. Efforts will then have to be aimed at reducing their impact as much as possible. Nevertheless, notwithstanding all efforts to eliminate or reduce problems, final estimates of population parameters may be distorted. Estimates differ from the true value to be estimated. This difference is called the total error of the estimate.

  Errors may occur in surveys whatever the mode of data collection, but some errors are more likely to occur in some types of surveys. For an example, it makes a difference whether interviewers conduct interviews or the respondents complete the questionnaires themselves. Focus in this chapter is on errors in web surveys. Sometimes the impact of a specific type of error in a web survey is compared with other types of surveys.

  Sources of error will, if present, increase the uncertainty with respect to the correctness of estimates. This uncertainty can manifest itself in the distribution of an estimator in two ways: (1) it can lead to a systematic deviation (bias) from the true population value, or (2) it can increase the variation around the true value of the population parameter.

  Let be an estimator for the population mean . Chapter 3 discusses the properties of a good estimator. One is that an estimator must be unbiased . This means its average value over all possible outcomes must be equal to the population mean to be estimated:

  (4.1)

  An estimator may be biased as a result of survey errors. Suppose one objective of a survey is to estimate the average amount of time per day people spend on the Internet. If a web survey is conducted for this, people without Internet access will not be in the sample. As these people do not spend time on the Internet, the estimate will be too high. The estimator has an upward bias. This bias of the estimator

  Sign up to read

  Learn more about book