Experimental Design and Statistics

Steve Miller

• Publisher: Routledge
• Year: 2005
• ISBN:
  9781134954629
• Edition: 2
• Topic: Psychology
• Subtopic: History & Theory in Psychology
• Index: Psychology

The distinguishing feature of experimental psychology is not so much the nature of its theories as the methods used to test their validity. The first edition of Experimental Design and Statistics provided a clear and lucid introduction to these methods and the statistical techniques which support them. For this new edition the text has been revised, the coverage of two-sample tests has been extended, and new sections have been added introducing one-sample tests, linear regression and the product-moment correlation coefficient.
Problems associated with the applications of experimental design and how to use observations of behaviour in research are key questions for all introductory students of psychology. This new and expanded edition provides them with an invaluable text and source.

1
Designing an experiment

The scientific approach to behaviour

Psychologists do not have an exclusive claim on the study of human behaviour. Many philosophers, writers, poets, theologians, and even salesmen are interested in the nature of human thought and action. And indeed, there is a sense in which we all possess some sort of personal ‘theory’ of behaviour, a theory which is used to guide our relationships with other people and to help us predict each other’s reactions and sensitivities.

Now each of these different approaches can sometimes produce perfectly respectable psychological theories. For example, the popular belief, ‘Once bitten, twice shy’, has its counterpart in the psychological literature on punishment. And some theological views, such as the belief that suffering induces more tolerant attitudes, could easily be the starting-point for a programme of psychological research. So we cannot claim that there is a basic difference between the theories put forward by psychologists and those devised by other students and observers of human nature. The distinguishing feature of experimental psychology is not so much the nature of its theories as the methods used to test their validity. The approach adopted is essentially scientific; a psychological theory has to fit the facts of behaviour as derived from systematic observations taken in carefully controlled conditions. If a theory does not fit the facts it is discarded or revised, no matter how long its history, how appealing its logic, or how convenient its implications. As we shall see, this, emphasis on objectivity and rigorous control narrows the range of behaviour that can feasibly be studied, but in return it produces more solid and reliable conclusions.

Thus we have defined experimental psychology mainly in terms of its methodology, i.e. the procedures used to evaluate theories. But we have placed no limits on the way in which these theories are first devised. They may be based on years of exploratory research, on a vague hunch, or even on a political or religious viewpoint. It doesn’t really matter so long as the theories give rise to predictions which can be tested against the behaviour of human beings in controlled settings. This is the basis of scientific psychology.

Predictions in psychology

What does a prediction from a psychological theory actually look like? Here are some examples:

1. Highly aroused subjects respond more rapidly to visual stimuli than sleepy subjects.
2. Reaction time to sounds is faster than reaction time to visual stimuli.
3. Daydreaming occurs more frequently during simple tasks than complex tasks.
4. The amount of saliva produced by a dog when a bell is rung depends on how frequently the sound of the bell was previously associated with feeding.
5. Two letters can be classified as having the same name more rapidly if they are identical (e.g. AA) than if they differ in case (e.g. Aa).
6. Driving skill varies as a function of the level of alcohol in the blood.

These predictions (or experimental hypotheses) differ in a number of ways. Some accord very much with common sense whereas others do not. Some are based on rather complex theoretical notions whereas others seem to reflect a very simple model of behaviour. Some are clearly related to everyday behaviour whereas others deal with seemingly obscure psychological tasks. What is common to them all, however, is their general format. In effect each prediction is saying that as one thing changes there will be consequential change in something else: as arousal increases response time decreases, as task complexity increases so frequency of daydreaming decreases, as alcohol level changes so driving skill changes, and so on. Each of the above predictions has the same basic logical form although the sentence structure varies slightly from one case to another.

A prediction, then, is a statement that a change in one thing (the independent variable or IV) will produce a change in another thing (the dependent variable or DV). Thus a change in arousal level (the IV) is said to produce a change in response time (the DV). Task complexity (the IV) influences frequency of daydreaming (the DV). And variations in alcohol level (the IV) cause changes in driving skill (the DV). In general the independent variable will relate to a change in the conditions governing behaviour and the dependent variable will correspond to some measure of the subject’s behaviour or performance under those conditions.

Testing a prediction: the role of experimental design and statistics

How do we go about testing a psychological prediction? Clearly we need some sort of plan for collecting information or data about the relationship between the independent and dependent variables. The formulation of a plan for collecting relevant data is known as research design.

Consider, for example, the prediction that driving skill is affected by the level of blood alcohol. One possible plan for data collection would be to ask a sample of drivers whether they had experienced any change in their driving skill after drinking. This wouldn’t be a very good research design for obvious reasons. Drivers are probably very poor at estimating changes in their own skill even when they are sober. But we would be asking them to remember changes which took place while in various states of intoxication. Even if they could recall such information it is doubtful whether they would willingly report their own reckless or anti-social behaviour just because we asked them. So this approach would lead to unreliable and possibly biased results.

An alternative plan would be to collect our data from various official records of accident rates. We would be interested, for example, in the proportion of accidents involving drunken drivers. We could also compare accident rates before and after the introduction of stricter legislation on drink and driving. Or we could look at changes in accident rates as a function of the times at which public houses are open for the sale of drinks. Data of this kind would lead to rather tentative conclusions about the effects of alcohol level on driving skill. Although this sort of evidence is reasonably objective, it is none the less circumstantial.

Scientists are generally agreed that the most effective means of testing a prediction is deliberately to manipulate the independent variable and then to observe the consequential changes in the dependent variable. It is only this method of collecting data—the experimental method—which has the power to reveal cause-and-effect relationships in an unambiguous way. Thus the best way of testing whether alcohol influences driving skill is to actually administer different quantities of alcohol to groups of volunteers and to compare their performance in a subsequent driving task. This direct, experimental approach will produce more definite results than the methods based on the observation of natural events, such as a survey of accident rates in relation to the availability of drinks.

We are committed in this book to the use of experimental methods in testing psychological predictions. In this first chapter we shall discuss the different ways in which data can be collected in an experiment—this is the topic of experimental design. Our main interest is to decide how subjects should be divided between the various conditions of the experiment. Such questions of layout should not be decided arbitrarily. The idea of experimental design is to ensure that the data emerging at the end of an experiment are relevant to the prediction being tested and not contaminated by outside factors. In other words, an experiment has to be designed in such a way that its results will logically confirm or refute the predicted effects of the independent variable. It is surprisingly easy to sabotage these aims by carelessness at the design stage of an experiment.

Let us assume, for the moment, that an experiment has been adequately designed and carried out. The next step is to interpret the results to see if they support our prediction. Now it rarely happens—even with the best laid designs— that the results are completely clear cut. One would be pleasantly surprised if all the subjects in one condition produced dramatically different behaviour from all the subjects in another condition. We normally get some overlap between the performance of the two groups, some blurring of the effects of the independent variable. This is where statistics come in—to tell us whether we can draw any general conclusions from the data, or whether there is too much blur to say anything. We shall return to this problem and the various techniques needed to solve it in the coming chapters. But before we do this we must establish the basic logic of an experiment and the principles underlying its design.

What exactly is an experiment?

In formal terms an experiment is a means of collecting evidence to show the effect of one variable upon another. In the ideal case the experimenter manipulates the IV, holds all other variables constant, and then observes the changes in the DV. In this hypothetical, perfect experiment any changes in the DV must be caused by the manipulation of the IV.

Suppose, for example, we have a theory which predicts that sleep deprivation causes an increase in reaction time to visual signals. Given the co-operation of a group of subjects we could test this prediction experimentally. Half the subjects would be deprived of sleep for one night while the remaining half were allowed to sleep normally. The next morning we would measure the reaction time of each subject and see whether the sleep deprived group had noticeably longer reaction times. If they had, and provided that the two groups were similar in all other respects, we would be justified in concluding that sleep deprivation causes a slowing of reactions to visual stimuli. This procedure qualifies as an experiment because we have actually manipulated the independent variable (amount of sleep) and observed the consequential changes in the dependent variable (reaction time). This is the simplest type of experiment—one in which the independent variable takes on only two values or levels (no sleep—normal sleep). We shall focus on this basic design for most of the book, but the principles will apply equally to more complex experiments in which we compare the effects of three or more levels of the independent variable (e.g. no sleep—two hours’ sleep— normal sleep).

Now let us consider a second example. This time we have a theory which predicts a relationship between, say, intelligence and reaction time: intelligent subjects are expected to react faster than less intelligent subjects. Again we might divide our group of willing volunteers into two sub-groups, this time according to their IQs. Thus we would form a more intelligent group and a less intelligent group. If we found the predicted difference in reaction times we might be tempted to conclude that intelligence determines the speed of a subject’s reactions, just as sleep deprivation did in the previous example. The analogy is misleading, however, because intelligence cannot be deliberately manipulated in the way that sleep deprivation can. Subjects bring their own level of intelligence with them to the laboratory, and all we can do is to observe the relationship between intelligence and reaction time. But we cannot go on to infer a cause-and-effect relationship between the two variables because both of them may be influenced by a third, uncontrolled variable. For example, it may be that intelligent subjects tend to be healthier than less intelligent subjects, or younger, or more highly motivated, or more attentive...; any one of a number of such variables might be the real cause of the variation in reaction time. We might be mistaken if we ascribed the causal role to intelligence rather than one of the other variables which happens to be related to intelligence. This example highlights the major weakness of all non-experimental research; we can never be certain whether the independent variable we measure is actually the one that produces changes in the dependent variable.

Unfortunately many of the variables we would like to investigate cannot be brought under experimental control. We cannot manipulate a subject’s personal characteristics: his age, sex, social status, intelligence, personality, religious beliefs, social attitudes, and so on. Nor would we want to interfere with critical aspects of a subject’s physiological or emotional state, even though this might be possible in principle. In such cases the researcher has to fall back on observing natural variations in the variables of interest. He compares the performance of old and young subjects, or males and females, or, as in the above example, more and less intelligent subjects. He then has to resort to statistical procedures to rule out the possible influences of uncontrolled factors which may be changing together with the variable under study. The procedures which are used in this type of study are discussed in more detail in other texts under the heading of correlational designs (see, for example, Chatfield and Collins, 1980; Miller, forthcoming). They are of critical importance in such subjects as social psychology, sociology and economics where experimental control of the independent variables is usually impossible or unrealistic. But the experimental psychologist can avoid the complications of correlational studies if he is prepared to limit his research to variables which can be manipulated experimentally. This means, in effect, that he is going to be more concerned with the influence of external conditions on performance than with the effect of the characteristics of his subjects. It also means he will be less plagued by the problems of psychological measurement. The manipulation of independent variables like the brightness of a stimulus or the length of a word is much more straightforward than the measurement of independent variables like intelligence or neuroticism.

Irrelevant variables

So far the reader may have gained the impression that there are no unwanted variables in an experiment—that all the variation in the subjects’ performance will be caused by changes in the independent variable. This might be true of a scientific paradise, but in the real world it is impossible to hold constant all those variables which might influence the subjects’ behaviour in an experiment. Take, for example, our investigation of the effects of sleep deprivation on reaction time. How could we ensure that all our subjects are equally attentive and well motivated, or that they have equally acute vision? Can we be certain that the apparatus will remain equally sensitive throughout the experiment, or that the background noises and distractions will be the same for each subject? And how can we eliminate any changes in the experimenter’s tone of voice or general attitude to the subjects? Even changes in the room temperature or time of day might affect a subject’s reaction time. There are a large number of variables that might, in principle, affect the dependent variable, although we are only interested in the effects of sleep deprivation. For the purposes of our experiment, then, all the other factors may be thought of as irrelevant variables.

The effect of irrelevant variables

It would be useful if we could hold all of these irrelevant variables constant and just manipulate the independent variable. Then we would get a perfectly clear picture of its effect on the subject’s behaviour. But such complete control over all irrelevant variables can never be achieved. It is either physically impossible (e.g. how can one hold the sensitivity of the a...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• Table of statistical tests included in this book
• 1 Designing an experiment
• 2 Data presentation and the normal distribution
• 3 Basic concepts of statistical testing
• 4 Independent two-sample tests
• 5 Related two-sample tests
• 6 One-sample tests
• 7 Tests for trend and spread
• 8 Measuring the relationship between two variables
• 9 Predicting one variable from another
• 10 Experimental design and beyond
• Appendix 1: calculating the mean, median and mode from a frequency distribution
• Appendix 2: calculating the variance and standard deviation from a frequency distribution
• Appendix 3: statistical tables
• References and name index