Single-case and Small-n Experimental Designs

A Practical Guide To Randomization Tests, Second Edition

Pat Dugard, Portia File, Jonathan Todman

• Publisher: Routledge
• Year: 2012
• ISBN:
  9781136588471
• Edition: 2
• Topic: Psychology
• Subtopic: Research & Methodology in Psychology
• Index: Psychology

This practical guide explains the use of randomization tests and provides example designs and macros for implementation in IBM SPSS and Excel. It reviews the theory and practice of single-case and small- n designs so readers can draw valid causal inferences from small-scale clinical studies. The macros and example data are provided on the book's website so that users can run analyses of the text data as well as data from their own studies.

The new edition features:

• More explanation as to why randomization tests are useful and how to apply them.
• More varied and expanded examples that demonstrate the use of these tests in education, clinical work and psychology.
• A website with the macros and datasets for all of the text examples in IBM SPSS and Excel.
• Exercises at the end of most chapters that help readers test their understanding of the material.
• A new glossary that defines the key words that appear in italics when they are first introduced.
• A new appendix that reviews the basic skills needed to do randomization tests.
• New appendices that provide annotated SPSS and Excel macros to help readers write their own or tinker with the ones provided in the book.

The book opens with an overview of single case and small n designs -- why they are needed and how they differ from descriptive case studies. Chapter 2 focuses on the basic concepts of randoization tests. Next how to choose and implement a randomization design is reviewed including material on how to perform the randomizations, how to select the number of observations, and how to record the data. Chapter 5 focuses on how to analyze the data including how to use the macros and understand the results. Chapter 6 shows how randomization tests fit into the body of statistical inference. Chapter 7 discusses size and power. The book concludes with a demonstration of how to edit or modify the macros or use parts of them to write your own.

Ideal as a text for courses on single-case, small n design, and/or randomization tests taught at the graduate level in psychology (especially clinical, counseling, educational, and school), education, human development, nursing, and other social and health sciences, this inexpensive book also serves as a supplement in statistics or research methods courses. Practitioners and researchers with an applied clinical focus also appreciate this book's accessible approach. An introduction to basic statistics, SPSS, and Excel is assumed.

chapter one

Single-case and small-n designs in context

Introduction

This book is intended to help researchers, students, and teachers in clinical, educational, and other areas involving human participants. Research in these areas is often aimed at testing the efficacy of a treatment or intervention in improving some measure of health or well-being. Psychologists seeking to understand human behavior also often test the effect of an intervention, treatment, or other experimental condition. There are wellestablished procedures for designing experiments where treatments or interventions can be randomly allocated to large numbers of participants who are representative of a well-defined population. We refer to such designs as large-n or large-group designs. Tests such as t tests and analysis of variance (ANOVA), known collectively as parametric tests, generally provide valid analyses of such designs provided some assumptions about the data are approximately met.

However, in much research involving people, the availability of individuals within specific categories is limited, making large-n studies impractical. It is no solution to increase the numbers available by defining the category broadly, because then the large differences among individuals within the broad category are likely to reduce drastically the power of the tests to detect any treatment effect. This is not the only reason why single-case and small-n designs may be used. They are also needed in exploratory research, for studying change over time, and to give a finegrained focus on effectiveness in particular circumstances. In fact, singlecase and small-n studies play a key role in the overall research strategy in the human sciences.

Levels of sophistication in single-case and small-n studies

Small-scale studies range in complexity from informal observation through formal case studies to formal observation of intervention effects. All these types of investigation have their place in the overall research effort, as we discuss in the next section, “Single-Case and Small-n Designs in the Research Process.” Our purpose here is to convince you that, if you are considering a small-scale study, it is worth the effort of going beyond careful observation to full experimental design and analysis.

As an example, consider a neuropsychologist in a rehabilitation ward, helping patients who have suffered a traumatic brain injury. These patients often have very individual problems, so it sometimes takes creative imagination as well as knowledge and experience to suggest a treatment that may help. The neuropsychologist makes a suggestion for a particularly difficult case. The patient is keen to try something new, the treatment goes ahead, and all members of the ward team watch hopefully for progress. The patient too is eagerly watching for improvements. At the next case conference, everyone agrees that the new treatment has made some positive difference.

Where can the team take this from here? The evidence of their own eyes seems to show they have made some progress on a difficult case, yet it remains possible that in their enthusiasm to make a difference, they have seen changes where none exist. Alternatively, they may have witnessed real change that is a part of the natural course of the patient’s condition and the treatment was irrelevant. It is possible to increase confidence in these results, but at some cost in terms of staff time, effort, and resources. The team may be tempted instead to spread the word about the apparent success so that others can research it further while they return to treating their patients.

But we want to convince clinicians and others that supporting their own observations with good experimental evidence is a realistic option for them. It is not true that the only way to provide convincing evidence of any effect is a large experiment with huge resources and a large staff followed by statistical analysis. It is also not true that statistical analysis always needs a large dataset. The key objective in designing an experiment is described next, and then we introduce the statistical tool that is the subject of this book.

Experimental design and internal validity

The key objective in any experimental design is the elimination of alternative explanations for any treatment effect observed. For instance, if cancer patients on the new drug live longer than those on the standard treatment, we want to know that the new drug is the explanation rather than some alternative such as the following: “The patients on the new drug were all treated by a specially dedicated team that gave emotional support as well, whereas the others attended the usual clinic where treatment was more impersonal.” Internal validity is the term used for this elimination of alternative explanations. A good experimental design removes all threats to internal validity. (An experiment has external validity if the results can be applied to a wider population than those participating, and we discuss this in Chapters 3 and 6.)

An important tool in achieving internal validity is randomization. Notice that this is not at all the same as random sampling. Random sampling means choosing a sample from a well-defined population by using random numbers. Parametric tests assume the sample used for the experiment is obtained in this way, though in practice this is hardly ever true, and we discuss this fully in Chapter 6. Randomization refers to the method of allocating different treatments, conditions, or interventions. In large-n designs, treatments or interventions are randomly assigned to participants. Of course this is impossible in single-case studies, but it may be possible to randomly assign treatments to observation occasions. If we make a series of observations on a single case, or on a few cases, we can think of each as an observation occasion. If we make just one observation on a participant, then that too is an observation occasion. Randomly assigning treatments to participants is just one example of randomly assigning treatments to observation occasions.

Randomization tests: the tool we need

There are two reasons why we may not be able to apply the familiar parametric tests to the data when we have single-case or small-n designs, even if we do randomly allocate treatments to observation occasions. We did mention that there are several assumptions that need to be at least approximately true for the tests to be valid, and the smaller the number of observations, the harder it is to tell whether the assumptions are reasonable. One solution to uncertainty about the assumptions is to use nonparametric tests such as Mann-Whitney or Wilcoxon. These and other common nonparametric tests use ranks rather than actual observations, so some information is lost and the tests may lack sensitivity when applied to small groups. The other problem, especially relevant to single-case designs, is that parametric tests assume that the observations are independent. If we take a series of measurements on a single case, it is quite likely that if one measurement is high for some reason unrelated to the treatment, then the next measurement may also be a bit high. In other words, neighboring observations may be correlated and the observations are not independent. We discuss this more fully in Chapters 3 and 7.

The problems outlined in the previous paragraph do not apply to randomization tests. To use randomization tests, we do not need to make the parametric assumptions. Also for single-case designs, we need not assume the observations are independent, though as we shall see, large correlations among neighboring observations will reduce our chance of detecting a treatment effect. Randomization tests also do not rely on the unrealistic assumption of random sampling from a population. Thus, randomization tests provide us with the tools we need to analyze single-case and small-n studies, but we need to plan for them at the design stage, so designs and tests should be studied together, as in this book. Randomization tests can also be used for large-n studies, but well-known techniques already in use are fairly satisfactory and there is always resistance to change. However, we discuss their possible future role in Chapter 6.

Before proceeding to the study of randomization tests, how they work, and how to plan for them with suitable designs, we briefly review the important but often unconsidered part that small-scale studies play in the wider research process. This is the subject of the next section. We follow this with a short explanation of why, if randomization tests are so useful, they are so little known. Finally, we end this chapter with some notes on how you might use the book.

Single-case and small-n designs in the research process

The scientific method of gaining new knowledge and understanding imposes similar structures and constraints on researchers in most areas of enquiry. We use the example of clinical research to show how single-case and small-n studies fit into the wider picture of the research effort.

A model of the clinical research process

In clinical research, as in other areas, the randomized controlled trial (RCT) is seen as the gold standard for research. This reputation is deserved and is an important achievement of 20th-century science, but it does not provide what is needed for every study or every stage in the research process. In particular, the early stages of developing a new idea do not lend themselves to large experiments. Also, in clinical work, although a well-run RCT can do a great job of answering questions about the average improvement due to an intervention, a clinician usually wants to know whether it works for a particular patient.

Robey (2004) proposed a hierarchical model (Table 1.1) for the whole clinical research process, in which single-case and small-n designs appear in the first two phases and again in the fourth phase. As can be seen, in this model there is a place for RCTs, single-case and small-n designs, and qualitative, meta-analytic, and cost-effectiveness studies. A similar structure, beginning with identification of a promising idea, through small and then large studies to replication and finally cost-effectiveness studies, could be identified in other research areas. To get the best results, we need to use all these methods in their proper place.

Table 1.1 Robey’s Model of the Clinical Research Process

Phase	Purpose	Methods
I	Identifying a therapeutic effect and estimating its magnitude	Exploratory qualitative, single-case, and small-group designs
II	Exploring the dimensions of the therapeutic effect and preparing for a clinical trial	Experimental single-case and small-group studies, and reliability and validity studies
III	Clinical trial to test efficacy	RCT, independent replication
IV	Field research to test effectiveness in target population, subpopulations, variations in service delivery, and variants of protocol	Meta-analyses, and experimental single-case and small-group studies
V	Who benefits, and at what cost?	Cost-effectiveness investigations

As can be seen in the above model, there are times when singlecase and small-n studies are appropriate and times when they are not. Exploratory studies and some individual clinical studies fall into Phase I and II of Robey’s model. Studying effects under different conditions would be part of Phase IV. However, the use of RCTs in Phase III of Robey’s model is wholly appropriate and small-n designs are unlikely to add much at this stage. There is now a huge body of knowledge about the best ways to design, manage, and analyze large RCTs and other experiments with large samples. In contrast, and despite the apparent usefulness outlined in Robey’s model, single-case and small-n designs have received far less attention. This book is a contribution to extending the knowledge of how best to use experimental opportunities with much smaller groups.

Consider the place of single-case and small-n research in each of the areas identified by Robey: exploratory studies, individual clinical studies, and studying effects under different conditions.

Exploratory studies

At some stage in the research process we must find an idea worth testing, and such an idea may come about by keeping an eye open for surprise developments and possible causes. When an observant scientist with a sense of curiosity notices an anomaly in research data or clinical experience and chooses not to ignore it, there is the possibility that a new and good idea will be born. Such new ideas may just as easily come from an experienced practitioner or from a young newcomer who brings a fresh perspective and is too inexperienced to have preconceived ideas. All practicing scientists need to use their senses and listen to their intuition, but it takes some confidence and judgment to know when to pursue an unusual observation.

As a first step, investigating a single case or a very small group may be possible with quite modest resources, and this may clarify the idea and suggest what form a larger study should take. Alternatively several smallgroup studies may be used to lay out the directions for future work or to prepare for a large experiment such as a major clinical trial.

Individual clinical studies

To find out about the effectiveness of an intervention for an individual patient, we have to collect data from that individual. We need a singlecase design. At this level of individual studies, it is clinicians or ...

Table of contents

• Cover
• Halftitle
• Title
• Copyright
• Dedication
• Contents
• Preface
• Chapter 1 Single-case and small-n designs in context
• Chapter 2 Understanding randomization tests
• Chapter 3 Obtaining the data: Choosing the design
• Chapter 4 Obtaining the data: Implementing the design
• Chapter 5 Analyzing the data: Using the macros
• Chapter 6 Analyzing the data: Wider considerations
• Chapter 7 Size and power
• Chapter 8 Going further
• Solutions to even numbered exercises
• Appendix 1
• Appendix 2
• Appendix 3
• Glossary
• References
• Author Index
• Subject Index