Methodological Advances in Educational Effectiveness Research

Bert Creemers, Leonidas Kyriakides, Pam Sammons

• Publisher: Routledge
• Year: 2010
• ISBN:
  9781136966316
• Edition: 1
• Topic: Education
• Subtopic: Education General

Methodological Advances in Educational Effectiveness Research is an important new work by some of the leading researchers in the field of Educational Effectiveness Research (EER). The book provides a state of the art snapshot of the methodology of EER now and clearly demonstrates the way it is applied in both research and evaluation. It shows how developments in the research methodology area such as the use of multilevel modelling approaches to analyse nested data have promoted the knowledge-base of educational effectiveness. But at the same time, as the authors show, the knowledge-base of educational effectiveness and the attempt to establish theoretical models do paradoxically challenge the development of methodologically appropriate studies including ways of analysing data.

Guiding readers though the effective and appropriate use in educational effectiveness of:

• 
  
  

• Longitudinal Studies

• 
  
  

• Experimental Studies

• 
  
  

• Mixed Research Methods

• 
  
  

• Meta-analyses of effectiveness studies

• 
  
  

• Using IRT to measure outcomes and factors

• 
  
  

• Using Generalisability Theory to test the quality of data

• 
  
  

• Multilevel modelling, and

• 
  
  

• Structural Equation Modelling Techniques

The authors draw in the expertise of scholars from around the world to show the mathematical background of each technique, the current and future applications, and Specific examples of applying this orientation to help the readers design their own effectiveness studies using specific methodological tools.

Part B
The contribution of different methodological orientations

The development of educational effectiveness research

Chapter 5
Longitudinal designs

Jan-Eric Gustafsson
University of Gothenburg, Sweden

Introduction

As has been made clear in the previous chapters of this book, there are many good reasons to use longitudinal designs in EER, and these have been used much more frequently during the last two decades. However, there are many different types of longitudinal designs and these are not always easily implemented. It is therefore necessary to have a firm grasp of the advantages and disadvantages of the different types of design. The main aim of the current chapter is to provide a basis for understanding the benefits and disadvantages of longitudinal studies. However, the field is enormously large and of great complexity so no claims of completeness are made. Rather, this chapter is a conceptually oriented introduction to the use of longitudinal designs in EER, with references for further reading.

What is special about longitudinal designs?

One fundamental characteristic of a longitudinal design is that it involves observations of the same units (for example, students) at more than one point in time. It is thus different from the cross-sectional (or survey) design, in which the units are only observed once. Another fundamental characteristic of longitudinal design is that it is observational; it does not involve manipulations, such as assigning different groups of subjects to different treatments. In this way it is different from the experimental design, which does exactly this.

However, the characterization of longitudinal design as involving repeated observations of the same units is much too simplified to carry any important information, and further complexities are needed to fill out the picture. However, it must also be stressed that it is possible to combine experimental and longitudinal designs in a large variety of ways. For example, one can create longitudinal intervention designs (Antoniou 2009; Demetriou 2009).

One may also ask what it means to observe units at more than one point in time. Suppose that we conduct a survey at the end of grade nine, in which we include questions about which school marks the students obtained in mathematics in grades six and nine. Given that we have information about the students’ level of school performance at two time points, it could be claimed that this is a longitudinal study. Such a design is commonly referred to as a retrospective longitudinal design, but it is considered to generate data of lower quality than a study with a prospective longitudinal design, which in our example would involve starting in grade six and following the students through to grade nine. There are two reasons why the prospective design is superior. One is that the retrospective design relies on memory, and it is a well known fact that human memory is unreliable and often systematically biased. The other is that the grade nine sample is likely to be a biased non-representative sample because of processes of attrition due, for example, to movement, non-promotion and grade-skipping. For these reasons, retrospective longitudinal designs should be avoided. However, it must be realized that prospective longitudinal designs often also include certain elements of retrospection in questionnaire items as well.

Even if it is agreed that there should be repeated observations of the same units at more than one point in time, one may still ask if there should be a certain minimum number of time points for a design to qualify as longitudinal. It has been observed (Rogosa et al. 1982) that when there are only two time points, the amount of information available for studying individual change and development is very limited, and that it is therefore usually desirable to have observations from more time points. There is, however, general agreement among methodologists that a design with observations at two points still qualifies as longitudinal.

Another part of the definition of a longitudinal design that lacks clarity is the meaning of the term ‘unit’. One of the characteristics of EER is that it operates with several different ‘units’, such as students, teachers, classroom, schools, school-districts and school-systems, and these are typically hierarchically nested. One obvious research design is to apply repeated measurements to the students along with observations of characteristics of their teachers, classrooms and schools. Such designs, in which the micro-level units are followed over time, correspond to our intuitive notions of the meaning of longitudinal research, and these are referred to with different labels. Keeves (1988) refers to such designs as ‘time series designs’, while others call them ‘panel designs’ or just longitudinal designs.

It is also possible to conduct longitudinal research in which there are new micro-level units at each wave of measurement. Consider, for example, research in which we are interested in the stability and consistency over time of achievement differences between schools. This research issue has the school as a unit, and to investigate it empirically would require that we conduct repeated measurements of a set of schools by observing the level of achievement of, for example, their grade nine students at each measurement. However, for each of these measurements, there will be a new set of grade nine students in each school, so the design is longitudinal with respect to the schools but not with respect to the students.

Another example of a design that is longitudinal at a macro-level but not a micro-level is the design adopted by many of the international investigations of educational achievement (for example, PIRLS, PISA and TIMSS) in which studies are repeated every third, fourth or fifth year. The repetition is done in such a way that samples are drawn from the same population and the achievement tests are linked so that results are expressed on the same scales. This provides a basis for investigating trends in the levels of achievement for those countries that participate repeatedly. Thus, this design is longitudinal at the level of school systems, but not at the student level. Keeves (1988) refers to this kind of longitudinal research as a ‘trend design’, and while this kind of design has not been common in previous educational research, it has been used more frequently within other disciplines (such as political science) under labels such as ‘time-series cross-sectional designs’.

From the discussion above, it is clear that even though the basic idea of the longitudinal design is simple, there are many different versions and possibilities. This is one of the reasons why this is such a useful tool in EER. The benefits of such longitudinal research in EER will now be discussed.

Benefits of the longitudinal approach

The longitudinal approach provides several benefits, but discussing these more explicitly will provide a clearer understanding of when and why the extra cost and trouble involved in adopting such a design is worthwhile.

The first and most obvious benefit of the longitudinal approach is that it allows investigation of issues that have to do with stability and change over time. There are many important questions within EER along these lines, such as the development of effects of schools and teachers over time, the constancy and consistency of effects over time and long-term versus short-term effects. As was pointed out in Chapter 1, much of the recent EER has directed attention to such issues, and this has necessitated adoption of longitudinal research approaches because alternati...

Table of contents

• Quantitative Methodology series
• Contents
• Contributors
• Figures
• Tables
• Preface
• Part A The state of the art of Educational Effectiveness Research
• Part B The contribution of different methodological orientations
• Part C Methodological advances and EER
• Appendices to Chapter 10
• Appendices to Chapter 12
• Index