Research Methodologies of School Psychology

Critical Skills

Ryan J. Kettler

• Publisher: Routledge
• Year: 2019
• ISBN:
  9781317529859
• Edition: 1
• Topic: Psychology
• Subtopic: Research & Methodology in Psychology
• Index: Psychology

Research Methodologies of School Psychology is a comprehensive, actionable resource that offers graduate students and school psychologists the knowledge and skills to apply key scientific techniques in practice.

A volume in the Foundations of School Psychology Research and Practice Series, this book directly addresses the need for definitive resources on mastering research methodologies in the field. Covering topics such as development and evaluation of measures, application of various designs, and drawing inferences from data, Ryan J. Kettler provides rigorous yet accessible methodological guidance. Each chapter includes illustrative examples, summaries of essential learnings, and reflective concluding questions.

Using these engaging and invaluable strategies, graduate students and school psychologists will be effectively prepared to apply the scientific method in their own professional contexts.

Section 3

Research Designs in School Psychology

Chapter 5

Randomized Control Trials and Quasi-Experimental Designs in School Psychology

Experimental designs are school psychologists’ best methods for ­detecting and measuring any causal relationships between independent variables (IVs) and dependent variables (DVs). To verify the emphasis on experimental design within education in the United States, one need look no further than the Every Student Succeeds Act of 2015, which repeatedly and interchangeably refers to high-quality research, scientifically based research, and evidence-based research. The law delineates the various levels of evidence-based research on educational activities, strategies, or interventions as follows:

1. (i) demonstrates a statistically significant effect on improving student outcomes or other relevant outcomes based on –
2. (I) strong evidence from at least one well-designed and well-implemented experimental study:
3. (II) moderate evidence from at least one well-designed and well-implemented quasi-experimental study; or
4. (III) promising evidence from at least one well-designed and well-implemented correlational study with statistical controls for selection bias; or
5. (ii) (I) demonstrates a rationale based on high-quality research findings or positive evaluation that such activity, strategy, or intervention is likely to improve student outcomes or other relevant outcomes; and
6. (II) includes ongoing efforts to examine the effects of such activity, strategy, or intervention.
7. (II) includes ongoing efforts to examine the effects of such activity, strategy, or intervention.

(Section 8002, paragraph 17, Every Student Succeeds Act, 2015)

In doing so, the US Department of Education designated the two classes of group experimental design in this chapter, randomized control trials (RCTs) and quasi-experimental designs, as providing the strongest and next strongest (i.e., moderate) evidence for improving outcomes. Correlational studies are on a third level (promising), and a separate clause is included for strategies based on strong rationale that include ongoing evaluation (see Chapter 6 for more information about correlational studies). By making this distinction, the US Department of Education prioritizes internal validity ahead of external validity in research design, a choice that is consistent with the natural sciences (e.g., biology, chemistry, physics).

RCTs are held in such high regard because they provide the strongest evidence regarding a causal link between IVs and DVs. RCTs involve random assignment to groups, followed by manipulation of the IV, and ultimately followed by measurement of the DV. The groups are defined by levels of the IV and are sometimes called treatment groups because each group receives a different treatment (or no treatment at all for the control group). By incorporating random assignment to groups, RCTs ensure the units within a group do not systematically differ from the units in any other group, particularly on variables that might be related to the DV. The units within a group in school psychology research are often the participants, although sometimes the units are classrooms, schools, or districts. Shadish, Campbell, and Cook (2002) identified the features of an experimental design map nicely onto the three elements of a causal relationship shared in Chapter 1. The potential cause (i.e., the IV) is manipulated, the variation of the effect (i.e., the DV) along with variation in the cause is observable, and random assignment reduces the possibility of other explanations. Random assignment reduces this possibility by distributing other characteristics of participants among the groups in an unbiased manner. Quasi-experimental designs do not feature random assignment; they are otherwise analogous to RCTs and include features such as matching of the treatment groups on salient variables to reduce the chance variation in a variable other than the IV could cause variation in the DV.

Due to the aforementioned characteristics, RCTs are preferred for limiting threats to internal validity, which you will recall from Chapter 1 is the degree to which the shared variation between the IV and the DV in a study is attributable to the causal relationship being tested. In theory, internal validity may be viewed as a prerequisite to external validity, which is the generalizability of findings beyond the study conditions to another set of conditions (e.g., those within a school setting). In practice, it is challenging to design studies prioritizing both internal validity and external validity, and choices prioritizing one type of validity may be made at the expense of the other.

For example, a researcher may want to study the difference in effectiveness between reading instruction in phonics and reading instruction in whole language. She may design an RCT that involves randomly assigning students within participating schools to one of the two treatment groups, training the classroom teachers in both methods, and ultimately measuring growth in reading between the beginning of treatment and the end of treatment. The researcher need not be concerned with internal validity threats, such as whether there was something fundamentally different about the students who received phonics instruction versus those who received whole language instruction, because assignment was random. She may be concerned with external validity threats; for example, it is possible findings from the study do not generalize to typical classrooms because it is atypical for someone other than the teacher to assign groups.

Another researcher may design a quasi-experimental study allowing the teachers of each student to choose which method of instruction will be used and to assign the students to phonics versus whole word groups accordingly. The researcher using the quasi-experimental design need not worry about this external validity threat; he may worry about the internal validity threat that teachers could systematically assign students with different types of characteristics to phonics instruction versus whole language instruction.

Random Assignment

Shadish et al. (2002) indicated random assignment “is achieved by any procedure that assigns units to conditions based only on chance, in which each unit has a nonzero probability of being assigned to a condition” (p. 248). The key component that distinguishes random assignment from other methods of assigning units (often the participant of a study) to various conditions is each unit has an equal and independent chance of being assigned to a condition. Examples of methods used for random assignment include flipping a coin, rolling a die, or referencing a random number table or generator to determine the condition to which each unit will be assigned. Random assignment is the simplest way to ensure the results of a study will not be biased toward any one of a number of outcomes. The simplest studies employing random assignment do so to divide units between two different groups: a treatment group receiving the treatment being studied and a control group receiving no treatment or “business as usual,” which refers to the services that typically would be administered if the participants were not part of the study sample. Random assignment does not guarantee the results of the study will accurately reflect the relationship between the IV and the DV; it only guarantees, if done properly, the assignment process does not systematically favor any outcomes for the units in the treatment group versus units in the control group.

Random assignment can be difficult to accomplish in school-based research, although Shadish et al. (2002) identified several situations conducive to randomized experiments, including those in which demand outstrips supply, units are isolated, or change is mandated and solutions are acknowledged to be unknown. Whenever participants in a sample understand there is a limited amount of treatment available, they are more likely to accept random assignment as a fair method of distribution as compared to a situation in which treatment is plentiful. Also, whenever participants are isolated either in terms of time or location, they are less likely to know or be concerned about treatments persons in other groups are receiving. Whenever change is mandated and solutions are acknowledged to be unknown, participants may be more accepting of the research because it meets the need for change, and may be less concerned about group assignment because it is unknown which treatment (or business as usual) leads to the best outcomes. Any of these conditions may facilitate conducting an RCT in an educational setting.

Exemplary Randomized Control Trial

DuBois, Volpe, and Hemphill (2014) designed a study to determine the effectiveness of a two-week long computer tutoring model among students experiencing difficulties in early literacy. The treatment being studied was the Tutoring Buddy program, a one-on-one and computer-delivered intervention designed based on incremental rehearsal (IR; Tucker, 1989), a procedure that involves interspersing unknown facts among known facts to provide students many opportunities for success during learning. Core elements of the Tutoring Buddy Program include that (a) a student’s letter-sound expression is assessed, (b) a setup screen is used to select four known letters and two unknown letters, and (c) IR is followed to teach the students the unknown letters. The units in the study were 30 students in kindergarten or first grade. Participants were assigned to treatment and control groups using a random number generator. The researchers provided a table of demographic data depicting that the treatment and the control group were comparable with regard to age, grade, race, and language status. While the treatment group was about two-thirds male, the control group was about two-thirds female. (This gender distribution illustrates that using random assignment does not guarantee groups will be similar in all demographic variables of interest. Thus, even in an RCT study, it is important to report pertinent demographic information by condition.) The control group did not receive the treatment until the end of the study. They received business as usual instruction throughout the study. DuBois et al. (2014) measured student progress at four different points in time: (a) before treatment, (b) following one week of intervention, (c) following the full two weeks of intervention, and (d) one week after intervention concluded. Participants’ letter-sound expression (LSE), letter-sound fluency (LSF), and nonsense-word fluency (NWF) were measured at each of the four time points. LSE was measured using Tutoring Buddy, and LSF and NWF were measured by school psychology graduate students using the Dynamic Indicators of Basic Early Literacy Skills (DIBELS), Sixth Edition (Good & Kaminski, 2002). Across the three measures, rates of growth for the treatment group were greater than for the control group, and these differences were maintained for at least one week, leading the researchers to conclude the findings were supportive of Tutoring Buddy.

Symbols of RCTs and Quasi-Experimental Designs

RCTs and quasi-experimental research designs are often depicted using a common set of symbols to indicate assignment to groups, time points for measurement, and types of treatment. Within each depiction, each row represents a specific group. Whenever assignment to a group is randomized, an R is the first letter on the row pertaining to the group. Whenever assignment is not randomized, NR is used instead. A second way a lack of randomization is notated is by drawing a dotted line between the two rows, indicating the groups may not be entirely comparable in the way two randomly assigned groups are thought to be comparable. Whenever random assignment is used, there is no line between rows, indicating the groups are thought to be comparable. Following R or NR, the symbols are organized from left to right chronologically, with an O indicating each time point at which measurement occurs and an X indicating each period during which treatment occurs. A blank space on a row above or beneath rows including an O for measurement or an X for treatment indicates the group did not receive measurement or treatment, respectively, during that peri...

Table of contents

• Cover
• Half-Title
• Series
• Title
• Copyright
• Dedication
• Contents
• About the Author
• Acknowledgements
• SECTION I Research, Evaluation, and Our Field
• SECTION II Science Begins with Measurement
• SECTION III Research Designs in School Psychology
• SECTION IV Drawing Inferences from Data
• References
• Index