Statistical Analysis for Education and Psychology Researchers

Tools for researchers in education and psychology

Ian Peers

• Publisher: Routledge
• Year: 2006
• ISBN:
  9781135717513
• Edition: 1
• Topic: Education
• Subtopic: Education General

Basic statistical concepts such as probability, estimation and inference, and their role in research design and analysis are presented in this volume. The author demonstrates which statistical test to use in given circumstances and how to use it, drawing on data from psychology and education.; Written for those without a strong mathematical background, the book's examples can be worked using a pocket calculator. "Real life" data are analyzed using statistical software (SAS), output is interpreted, and a decision chart is presented which summarizes considerations when choosing a statistical test.

Chapter 1
Statistics and Research Design

In everyday use the word statistics has two meanings. In one sense it refers to the way we collect, analyze and interpret numerical facts which are termed data. Second, statistics are in themselves the raw numerical data resulting from observations or measurements, or are the results of calculations derived from such data. The term statistical analysis is often used and in general this may refer to the descriptive use of statistics to present and summarize data, or it may mean the way in which these statistics are used to make statistical inferences. This is the process whereby statistical information obtained from a sample is used to draw conclusions which have wider applicability than solely to the sample of observations or measurements obtained and is referred to as the inferential use of statistics.

Statistical inferences are described in terms of probability; the likelihood or chance in the long-run of occurrence of an event. In statistical jargon an event means any outcome(s) from among a set of all possible outcomes. The concept of probability is used in statistical modelling which compares patterns of variability and frequency of outcomes in real data with the idealized frequency of outcomes that would be expected from a statistical probability model. We can think of such a probability model as a mathematical representation to describe and predict phenomena such as patterns and relationships in data and outcomes of experiments. Interpretation of the fit or match of data to a particular statistical model can provide insight into the phenomena being investigated.

Data, however, do not interpret themselves and may be meaningless unless descriptive statistics are used to arrange numbers into a coherent and more meaningful summary of information. Inferential statistical techniques are the tools which can then be used to analyze data, answer specific questions, draw trustworthy conclusions and thereby gain information which might not be apparent from an initial scrutiny of data.

In this chapter we begin by emphasizing the importance of statistical thinking in research design and then go on to examine the role which statistics plays in the planning and data collection stages of a study. Next, we review the general principles and distinguishing features of survey and experimental designs and then present statistical guidelines which can be referred to during the design stage of a study or which can be used in the assessment of research papers, reports and manuscripts.

1.1 Why Consider Research Design in a Text About Statistics?

It is important to appreciate that statistics is much more than a collection of techniques for data analysis. Statistical ideas can and should be used to advantage in the design and data collection stages of a study. A well designed study that has generated reliable data but which has been poorly analyzed can be rescued by appropriate reanalysis. A poorly designed study, however, that has generated data of dubious quality, is beyond redemption, no matter how sophisticated the statistical analysis.

Use of Statistical Ideas in Research Planning

Sample size and statistical power

When planning a survey or an experiment a common problem for researchers is the determination of sample size or number of subjects in experimental groups. It is possible to estimate the number of subjects required either in a sample survey or in experimental design so that sample or treatment differences would be detected at a specified significance level. The significance level of a statistical test is the likelihood of concluding there is a difference (rejecting a hypothesis of no difference) when in fact there is a difference (the hypothesis of no difference is refuted). The estimation of sample size is achieved through statistical power analysis. Given certain assumptions, a statistical test is said to be powerful if it is able to detect a statistically significant difference should one exist (statistical power analysis is considered in Chapter 5). The point of doing a power analysis for a research plan based on a particular sample size is that if the design turns out to have insufficient power, that is one is unable to detect any statistically significant difference, then the researcher can revise the plan. One option would be to increase the sample size. There are other options, including, at the extreme, dropping the proposed study. Clearly, as little can be done after data has been collected, consideration of sample size and statistical power is crucial at the planning stage.

It should also be emphasized that statistical significance does not always imply educational significance. For example, a small gain in maths scores after an experimental maths programme may be statistically significant but may be considered to be of no educational importance. The researcher in planning an evaluation of this maths programme would have to determine what maths gain score would be considered a significant educational gain and design a study to be able to detect this magnitude of treatment effect.

Validity and reliability of measurement

Attention should be given to the construction of measuring instruments like questionnaires and sociometric indices. A common problem encountered with self-completion questionnaires is missing responses, often referred to as ‘missing data’. The best answer to this particular problem is to have none. If you do have missing data, this often tells you as much about the design of your questionnaire as the knowledge, opinions or thoughts of the respondent. The pattern of missing responses is also informative. Descriptive analysis of pilot study data may reveal selective non-response or indeed returned blank questionnaires for certain individuals. It is therefore sensible to spend time at the planning stage considering strategies to ensure complete responses and subsequently to complete a pilot study.

If there are problems with the specific method that generates the data, such as, ambiguous questions, then the data will not be valid. That is, the data will not be trustworthy because we have not measured what we think we have measured. In this case the questionnaire is said to have poor construct validity. Messick (1989) suggests that construct validity encompasses three other forms of validity often referred to in the measurement literature as content, concurrent and predictive validity. A questionnaire survey that has dubious construct validity is also likely to yield erroneous conclusions about differences that appear in the sample data. Researchers refer to the issue of drawing false conclusions from statistical tests of differences or relationships as a problem of statistical conclusion validity. Cook and Campbell (1979) suggest it is appropriate to establish whether differences or relationships exist before considering the magnitude or strength of any effects. Another aspect of statistical conclusion validity is the reliability of measurement, the idea that consistent results will be given by a measurement instrument when a subject is measured repeatedly under near identical conditions. Lack of reliability increases the amount of observed variation which has the effect of making it more difficult to detect significant covariation among variables. Larger sample sizes can, to some extent, compensate for this increase in variability of measures. However, as Henry, (1990) comments, ‘to compensate for the inflation of the variance [variability of observations] due to the lack of reliability of the instrument, it must be recognized and accounted for early in the design process’ p. 13.

Procedures for data collection

Data generated in a quantitative investigation should be the product of a research design, which is a plan specifying what information will be collected, how this will be done and how it should be analyzed. Quantitative studies such as surveys and experiments, if systematically planned, should make use of the idea of chance when data is collected because the role that chance plays in data generation influences the trustworthiness of any statements we make about research findings. For example, chance is involved when selecting subjects for a survey or allocating subjects to an experimental group. If data are collected in a systematic rather than in a haphazard way then knowing the role that chance plays in generating the data allows valid conclusions to be drawn about your results—or the results of others.

A random sampling procedure is often used in survey design. This means choosing subjects at random from a defined population. When random sampling is used each member of the target population has a known chance of being selected for the sample. In experimental research the principle of randomization is used as a means of assigning subjects to treatment groups on the basis of chance. Random assignment, which should not be confused with random sampling, is intended to produce experimental groups that are similar in all respects prior to any treatment. The randomization process, which does not mean a haphazard one, uses the laws of chance to assign subjects to treatment groups in a way which eliminates any systematic differences that might exist among subjects.

Whereas survey methods vary, for example, postal self-completion, class administered questionnaires, telephone, interview and observational surveys, the one feature most are likely to share is the need to obtain a sample. A survey sample is usually intended to represent the population from which it is drawn. If the sample is faulty or is not designed to be representative, then it is not reasonable to generalize beyond the achieved sample. This presents the researcher with a problem of external validity or generalizability. The ability to generalize findings relates to the definition of the population of interest, the sampling method used and the validity and reliability of measurement.

Sources of variability

A good sample design will minimize the amount of variability in observations or measurements to an acceptable level given the purpose and required precision of a survey or experiment. Variability is inherent in measurements on subjects. For example, consider a teacher who selects a sample of school children and measures for each child his or her assertiveness behaviour using the School Motivation Analysis Test (SMAT instrument described by Boyle and Houndoulesi, 1993) and then estimates an average assertiveness score. If on the following day the teacher repeats the testing, it is very likely that the two measurements for each child will vary as will the two averages of the first and second set of measurements. Such variability may be due to random variation in measurement, a real change in the children’s assertiveness (a dynamic trait) or a combination of both.

In designing a survey to estimate teenagers’ assertiveness, for example, consideration should be given to three potential sources of variability or error. The full inventory has 190 items, and some students may become bored or tired and not answer all items. This would introduce a measurement error which is an example of nonsampling bias. If the teacher had selected a non-probability sample that is a non-random sample, then the sample may not be representative of all teenagers (not every teenager would have an equal or known chance of being selected). This would introduce a selection bias and is an example of sampling bias. Finally any sample is subject to sampling error or sampling variability. Put simply this means that any particular sample average, given certain assumptions, will vary from another independent sample average. This idea of a distribution of sample averages and how it relates to sampling error will be explored in Chapter 4. The implication for planning is that the precision of a sample statistic such as an average assertiveness score is related to sampling error. In fact the precision of a sample statistic decreases as sampling error increases. Sampling error is influenced by sample size and variability of what is being measured in the population, in this case assertive-ness. In this example, the smaller the population variability in assertiveness (more homogeneous) the smaller will be the sampling error; this will provide a more precise average assertiveness score. Larger sample sizes also reduce sampling error, which is one reason why larger samples are preferable to smaller ones. ‘Small’ in survey and experimental research is generally taken to mean less than 30 subjects, but this is only a guideline. Generally for both experiments and surveys more subjects are better than fewer subjects up to a point. The more subjects that participate in an experiment, the more it is likely that randomization to treatment groups will be effective. Consequently, on average groups will be similar because any individual differences among subjects will be averaged out by the random allocation of subjects to groups.

Choice of statistical tests

After data collection, descriptive statistics are used to summarize data. The research questions addressed and the nature of the data will influence the choice of summary statistics. For example, if more scores are at one extreme rather than in the middle of a distribution, the median may be a more appropriate measure of central ten...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• List of Figures
• List of Tables
• Appendix—List of Figures and Tables
• List of Statistical Symbols
• List of Formula
• Acknowledgments
• Preface
• Chapter 1: Statistics and Research Design
• Chapter 2: Measurement Issues
• Chapter 3: Initial Data Analysis
• Chapter 4: Probability and Inference
• Chapter 5: Choosing a Statistical Test
• Chapter 6: Inferences Involving Binomial and Nominal Count Data
• Chapter 7: Inferences Involving Rank Data
• Chapter 8: Inferences Involving Continuous Data
• Appendix
• References