Assessing Teacher Effectiveness

Different models

Jim Campbell, Leonidas Kyriakides, Daniel Muijs, Wendy Robinson

• Publisher: Routledge
• Year: 2012
• ISBN:
  9781134407224
• Edition: 1
• Topic: Education
• Subtopic: Education General

How can we really evaluate teacher effectiveness?

Systems of teacher appraisal and evaluation are being created across the world in order to monitor and assess teacher performance. But do the models used really give a fair evaluation?

Based on international research, the authors argue that teacher effectiveness is too narrowly conceptualised and methods of measuring it are not attuned to the real contexts in which teachers work. They propose a model of differential teacher effectiveness which takes into account that:

* teachers may be more effective with some categories of students than with others
* teachers may be more effective with some teaching contexts than others
* teachers may be more effective with some subjects or components than with others.

Building on and developing previous research on models of teacher effectiveness and current theories, the authors open up possible new debates which will be of interest to academics and researchers working in this area throughout the world.

Part 1

The background to teacher effectiveness research

1 Differentiated teacher effectiveness: framing the concept

The purposes of this chapter are to illustrate the influential ideas in teacher effectiveness, to argue the case for a differentiated model of teacher effectiveness, and to justify the structure of the book. We draw the ideas from key research studies, but there is no attempt to provide a comprehensive review of the literature, because the field has been systematically reviewed by others (see for example Creemers 1994, 1996, Mortimore 1998, Reynolds et al. 1994, Scheerens 2000, Scheerens and Bosker 1997, Teddlie and Reynolds 2000). The research studies we particularly draw upon are: Coleman et al. 1966, Jencks et al. 1972, Bernstein 1971, Brookover et al. 1979, Brophy and Good 1986, Rutter et al. 1979, Mortimore et al. 1988, Wittrock 1986, Creemers 1994, Mortimore 1998, Teddlie and Stringfield 1993, Reynolds et al. 1994, Teddlie and Reynolds 2000, Scheerens 2000, Wright et al. 1997.

The three terms, school effectiveness, teacher effectiveness and educational effectiveness, are used inconsistently in the literature, but are interrelated. We are taking school effectiveness to mean the impact that school-wide factors, such as leadership, school climate, and school policies, have on students’ cognitive and affective performance. Teacher effectiveness is the impact that classroom factors, such as teaching methods, teacher expectations, classroom organisation, and use of classroom resources, have on students’ performance. Educational effectiveness can refer to either of the above, but we are using it to mean the interactions between the school, classroom and individual student levels and their contributions to students’ performance. (Educational effectiveness can refer also to the functioning of the system as a whole, but we are not treating this aspect substantively.)

There is an issue of definition. The Shorter Oxford English Dictionary (SOED) defines effectiveness as ‘the quality of being effective’ which is not terribly helpful, especially as effective is defined as ‘concerned with, or having the function of, effecting’. The use of the term as an idea in research is much closer to the SOED's definition of efficacy, defined as ‘the capacity to produce effects; power to effect the object intended’. Thus we propose a definition of teacher effectiveness as follows: the power to realise socially valued objectives agreed for teachers’ work, especially, but not exclusively, the work concerned with enabling students to learn.

Four matters flow from this definition. The contexts and conditions in which students are enabled to learn can differ; students differ; the extent to which objectives for learning are achieved can differ; and the values underlying learning and effectiveness can differ. For these reasons we are working towards a concept of effectiveness which is differentiated.

By this we mean a concept of teacher effectiveness that moves beyond the generic to incorporate the idea that teachers can be effective with some students more than others, with some subjects more than others, in some contexts more than others, with some aspects of their professional work more than others. The task of research is to build concepts and methodologies that recognise this differentiation. We argue that a next stage for research development is a model of differentiated teacher effectiveness. We use ‘differentiated’ rather than ‘differential’, since the latter is used in the literature to mean different extent of achievement of effectiveness on the same dimension, whereas we mean effectiveness across different dimensions of teaching.

The background to educational effectiveness research

The conventional starting points for research into school effectiveness are the studies in the United States of America by Coleman et al. (1966), and Jencks et al. (1972), whose general findings were echoed in the early work of the English sociologist, Bernstein (1971). These studies were essentially sociological in perspective, attempting to unravel the uncertain relationships between society (defined in concepts of the economy, linguistic and cultural capital, the moral order, or social class relations) and the performance of its schools. One fundamental question they addressed was about the extent to which schools might ‘compensate for society’, to use Bernstein's phrase.

The general position portrayed in these studies was that, compared to social influences such as the family, media, poverty, or peer group culture, schools were relatively ineffective. Analysing the UK evidence, Musgrove (1971) considered that schools were ‘impotent’ to achieve the range of goals set for them. In the USA, Coleman and his colleagues (1966: 21) concluded, ‘when socio-economic factors are statistically controlled, it appears that differences between schools account for only a small fraction of the differences in pupil achievement’. According to Katz (1975: 142), ‘it is clear that the powers of schooling have been vastly overrated. Despite substantial financing and a captive audience, the schools have not been able to attain the goals set for them, with remarkably little change, for the last century and a quarter.’

In the UK, the focus was on social class. For example, with a national sample studied longitudinally, Douglas (1964) had shown that class-related differences in attainment at age eight had increased by age eleven. Moreover, at age eleven there were significant differences in attainment between manual and non-manual workers’ children whose attainment had been identical at age eight. In the United States, socio-economic status (SES) was increasingly, though problematically, conflated with race. The harshest judgement came from Katz, who argued that for poor, especially black or brown, children, education continued to control rather than to educate. Educational improvement was an illusion.

An influential meta-analysis using economic modelling (Carnoy 1976) covered studies of Puerto Ricans and Mexican/Americans as well as Afro-Americans. Carnoy (1976: 215) showed that it was possible to alter the school input allocation, including quality of teaching, but that equalising the input would not equalise the academic performance, as judged by exam scores, of ‘ethnic and racial minorities’. This judgement anticipated later work in the UK in which it was shown that improving the effectiveness of schools generally would not lead to reductions in social class differences in attainment and might even increase group differences (see Mortimore 1998).

Despite the significance attached to these studies, they embodied five problematic assumptions, the investigation of which helped to set the school effectiveness research agenda.

First, the fact that there was relatively little contribution at the level of the school to variance in pupil attainment was not the same as saying that there was none, so that the school-level contribution needed to be investigated rather than written off.

Second, the use of measures of student cognitive gain as a proxy for effectiveness ignored other possible measures, for example the extent to which schools were effective in transmitting social, cultural and moral values. As Jencks (1972:185) put it, ‘the best index [of a school's effectiveness] may not be reading scores but the number of rocks thrown through its windows in an average month’.

Third, the conclusion that schooling in general did not appear to reduce social class differences in achievement was interpreted as meaning that the school had no effect. However, it could be argued that without the intervention of schooling the differences in attainment between groups would have been even greater.

The fourth point is, in retrospect, obvious. The unit of analysis was the school, rather than the classroom: effectiveness would be demonstrated in a whole-school mode. Later work (Creemers 1996, Scheerens and Bosker 1997) showed that the greatest contribution to attainment from in-school factors was the classroom.

Fifth, the studies generated an attitude of resigned scepticism (a ‘climate of despair’ as Brookover et al. (1979) put it) about the potential for educational reform based on the school; only wholesale social and economic reform would improve the educational life chances of the disadvantaged. According to Jencks (1972: 185), changing the performance of schools would be at best a marginal activity.

Although revisionism through school effectiveness research has altered the social determinism in these studies, their strength was to remind us that there were limits to improved effectiveness, a point sometimes ignored in the contemporary zeal for increasing effectiveness.

The search for evidence of educational effectiveness

Not all educational researchers shared the pessimism, or social determinism, of the social theorists. Significance was attached by some to curriculum and teaching. In the UK, Bernstein's work included a language intervention programme designed to improve working-class children's access to the extended codes considered prerequisite for educational success (see Gahagan and Gahagan 1970). In the USA, Katz (1975) advocated that schools should offer a renewed emphasis on basic literacy skills and abandon goals associated with moral socialisation, in order to empower poor children.

Moreover, there was conceptual confusion in the American studies between education reform and educational effectiveness; the argument was that educational reform was difficult to achieve. But effectiveness might be demonstrated outside reform programmes. Secondly, the focus on overcoming social disadvantage diverted attention from an important but neglected question about the extent to which all schools, including those catering for socially advantaged intakes, were effective.

In the USA, Brookover and his colleagues (1979) investigated the impact of three school-based variables: input (student composition and teacher characteristics), school structure (parental involvement, differentiation in student programme, classroom organisation, time allocation, staff satisfaction) and school climate (norms, expectations and beliefs within the school) on the achievement, academic self-concept, and self-reliance of the students. They reported that, in combination, the three school variables explained most of the variance between schools in measures of achievement, self-concept and self-reliance. They found the data on the association between school climate and student achievement particularly impressive in three subsamples of school (high SES white, low SES white, and majority black schools (Brookover et al. 1979: 141–142).

Most importantly for the direction of later research and policy, they identified schools with similar student social or racial composition, but with very different climate variables, associated with different ‘sense of futility’ among students. The relevant variables reflected value-orientations: teachers’ belief in their students’ ability to learn, high expectations, and discriminating reinforcement of learning behaviour.

There are five methodological and conceptual issues in the Brookover et al. study. The test result data were pre-existing, state-wide, data collected for other reasons, and did not necessarily reflect curriculum coverage. Secondly, the data on climate and structure were, with the exception of four case studies, perceptions of climate and structure held by students, teachers and principals, which raises issues of validity. Third, the significant association between climate and attainment may be in part explained by the inclusion of teacher expectations in the climate variables. Fourth, the test data were attainment data, not progress data. Fifth, the school effect looked strong when the three school variables were used in combination, but treated separately the largest contribution came from the input variable in the sample as a whole, though climate made the largest contribution in the majority black schools.

In the UK, the seminal studies were by Rutter et al. (1979) and Mortimore et al. (1988), examining the school effect in secondary and junior schools respectively. In the Rutter et al. study, twelve secondary schools were studied, using four clusters of measures: intake, process, outcomes and ecology. The outcome measures were attendance rates, behaviour in school, examination success, employment, and delinquency. As with Brookover, Rutter found the combined effect of school process measures to be greater than any one measure and proposed that a distinctive school ‘ethos’ (again a value-orientation) might be necessary to explain the combination. The Mortimore et al. study involved 50 junior (i.e. 7–11 year olds) schools in London. Although the main concentration was upon school-wide factors, such as leadership, participation, and consistency, influencing pupil attainment, the study reported findings referring to teaching effectiveness. These included the unexceptional idea that lessons should be structured and intellectually challenging, work-centred, have high levels of communication, and a limited subject focus.

Important methodological implications for later studies were the fact that data were collected longitudinally on the same students as they progressed through school, so that causal explanations for progress could be essayed, and that classroom observation was involved so that direct evidence about teaching methods was available. Moreover, data on outcomes was not restricted to test scores, but included practical tasks, speaking and writing activities, self-concept measures, and measures of attendance and behaviour.

Models of teacher effectiveness

In America, a different perspective was being developed, which had a much lower dependence on school context for explaining teacher effectiveness. A series of key studies, culminating in the publication of the Handbook of Research in Teaching for the American Educational Research Association (Wittrock 1986), attempted to identify the characteristics of effective teacher behaviour in the classroom. The underlying model, referred to as the process-product model, led to a range of other models of teacher effectiveness (see Chapter 4).

The findings of this set of studies were influential for two reasons. They were able to identify the classroom behaviour of teachers most likely to lead to good student attainment, and thus to propose a set of context-free principles of teacher effectiveness. Also, some researchers had created intervention programmes based on the research. They thus linked analysis of classroom practice with its improvement.

Important though these studies were, there remained three principal problems: the model of teaching effectiveness was generic, lacking any sense that teachers could be effective with some students, in some contexts, in some subjects, but not in others; there was little explicit evocation of values underlying effectiveness; and the studies neglected the influence of school contexts on teachers’ classroom performance.

The American studies may have been re-inventing wheels, since they reflected a much earlier tradition. Robinson (2004) shows that similar models of effective teaching were being developed in England in the early decades of the last century, as educationists attempted to move teaching on to a more scientific basis, in tune with the emergent scientific rationalism of the time. These are discussed fully in Chapter 3, but a distinctive characteristic was ‘power to teach’, that is, the ability of the teacher to adjust general pedagogical principles in the light of her judgement about the needs of individuals or of particular contexts.

Methodologies for identifying differentiated effectiveness

From 1990 onwards the field has been dominated by technical developments in measuring effectiveness. These have been brought about by the invention of two techniques, namely Multi-level Modelling (MLM) (see Goldstein 1995) and Structural Equation Modelling (SEM) (see Raykov and Marcoulides 2000). Multi-level modelling has enabled researchers to tease out the interacting variables at the social, school, classroom and individual student levels. This has been particularly important for identifying the contribution to variance made by the classroom level, including the effect of teaching. Important developments occurred in the Netherlands (Creemers 1994, 1996) with the generation of a comprehensive model of educational effectiveness. This was able to predict that classroom variables would have the greatest effect of any school-based factors. It is early to evaluate these developments (but see Thrupp 1999, 2001a, 2001b, Slee et al. 1998, Lauder et al. 1998 for some critiques) because the modelling has tended to outstrip empirical testing. However, when Creemers’ model was tested empirically (e.g. by Kyriakides et al. 2000), the prediction that the classroom effect was the largest turned out to be supported.

Partly as a result of these technical developments, researchers in the UK and the USA (e.g. Borich 1996, Harris 2001, Watkins and Mortimore 1999, Hopkins and Reynolds 2001, Muijs and Reynolds 2000, Teddlie and Stringfield 1993) have been able to identify the issue of differentiated effectiveness. This has included different subjects (Askew et al. 1997, Medwell et al. 1999), different organisational structures (Harris 2001, Sammons et al. 1997) and different socioeconomic contexts (Teddlie and Stringfield 1993, Borich 1996). Educational effectiveness research was beginning to incorporate what Hopkins and Reynolds (2001) call ‘context specificity’.

Towards a model of differentiated teacher effectiveness: five possible dimensions of difference

It follows from the foregoing outline that there may be five potential dimensions of differentiation in teacher effectiveness.

The range of role activities expected of teachers in modernising education systems is extremely broad. These include social, pastoral, welfare dimensions and leadership of other adults, and other work outside classrooms, in addition to the formal instructional dimension.

Second, there is the issue of differentiated effectiveness across different subjects in the curriculum, or across different components (algebra as against number in mathematics).

Third, teachers may be differentially effective in promoting the cognitive progress of different groups of students according to background variables. The principal ones are ability, age (or developmental stage), sex, socio-economic status and ethnicity.

Fourth, teachers may be differentially effective in promoting the learning of students according to the students’ personal characteristics, such as their personality, cognitive learning style, and extent of motivation and self-esteem.

Fifth, teachers may be differentially effective in response to the different cultural and organisational contexts in which they work, such as the school culture, department structure and school size.

The nature and structure of the book

The book is organised in four parts.

Part I presents an overview of theory and research, examining the nature of teaching as work, historical antecedents and empirical research findings. After this introductory chapter, Chapter 2 examines the different roles that teachers occ...

Table of contents

• Cover Page
• Contents
• Preface
• Acknowledgements
• The background to teacher effectiveness research
• Differentiated teacher effectiveness: framing the concept
• Methodologies for Identifying Differentiated Effectiveness
• Historical models of teacher effectiveness
• Review of current research in teacher effectiveness
• Towards a differentiated model
• A critique of teacher effectiveness research
• Developing a model of differentiated teacher effectiveness
• Evidence in support of differentiated teacher effectiveness
• Building theory and methodology
• Values and policy implications
• Effective teaching and values
• Differentiated teacher effectiveness and teacher appraisal
• Educational policy implications
• Differentiated teacher effectiveness research: the model in practice
• A study of aspects of differentiated effectiveness
• Bibliography
• Index