Education Research and Evaluation: For Policy and Practice?

Robert G. Burgess

• Publisher: Routledge
• Year: 2014
• ISBN:
  9781135395858
• Edition: 1
• Topic: Education
• Subtopic: Education General

Much has been written on the styles, strategies and tactics associated with educational research and evaluation, but relatively little on the social processes associated with the methodology. Few books consider the relationship of research and evaluation to policy and practice and this book opens up key debates in that field. It identifies, through contributions from the USA and Britain, some of the major processes involved, examines the problems of conducting research and evaluation and the ways in which they can be overcome, and details case studies in which problems and processes are encountered.; Probably of worldwide interest to students, researchers, academics, policy makers and practitioners, the authors present an examination of a range of different dimensions associated with educational research and evaluation conducted for policy and practice.

1
Biting the Hand that Feeds You? Educational Research for Policy and Practice

Robert G. Burgess

The late 1980s witnessed changes to the styles of social and educational research. In recent years we have seen a shift away from the kind of curriculum research advocated by Michael F.D. Young in the early 1970s when he argued that there was no alternative to researchers devising their own problems on the school curriculum rather than taking questions devised by those actively engaged in policy formulation and the practice of education (Young, 1971). Indeed, in the world of research bids, evaluation studies, contracts and tenders, it is common to find the research problem and the research approach being specified by the sponsor. What, we might ask, are the implications of this move for research? Does it mean that research will no longer or can no longer be independent and critical? Does it mean that researchers are little more than contract labourers whose work is reduced to little more than the application of mere techniques? Does it mean that research in education must become policy-relevant? Does it mean that research must come closer to the requirements of practitioners? In this respect, it is the purpose of this chapter (and of the others in this volume) to examine some of the critical issues associated with research and evaluation in education by focusing on different aspects of the research process. In part, this chapter will be autobiographical as it will draw on my experience as a research director who in the course of negotiating contracts and designing research projects must consider not only how to get funding but also the implications this may hold for the collection, analysis and dissemination of data. In these terms, an analysis of research sponsorship becomes critical, as discussed in the following section.

Research Sponsorship

Given that research funding is critical to the research process, we might expect that those who have written on research methodology (myself included) might have had something to say about issues raised by research sponsorship. Yet a brief glance at the numerous texts and readers available quickly shows a glaring omission.

My own thoughts on this matter were stimulated by a discussion with a potential sponsor some years ago. The potential sponsor was the education officer of a large private company who had funded educational research and development work in the UK and who expressed interest in some of the work in which I was involved and in funding a conference. On hearing that the conference was concerned with curriculum and assessment policy he had a question to ask: would any of the people giving papers be making political statements or criticizing government policy? My response was to argue that political statements were not part of the business of social and educational research but a careful and critical analysis of policy and practice was part of the research agenda. At this point, the potential sponsor indicated that the directors of his company would not be happy to find that money had been used to voice criticism of government policy. In short, I would need to think if any of my speakers would be criticizing government policy if I was to bid for money from his company. Needless to say that company did not sponsor the conference, nor were they asked. Yet this was not a purely negative experience for it set me thinking about research sponsorship, styles of sponsorship, the involvement of sponsors with researchers and the researched and with activities concerned with development and dissemination.

We should begin by asking: who are the sponsors of educational research and educational evaluation? Here, there are the obvious organizations such as government departments (the Department for Education) and local authorities as well as a range of other government departments: the DSS, the DoH, the DTI and so on, and a number of public and private companies as well as local and national trusts. We might ask, however, what are the terms in which these organizations request research and evaluation bids, issue tender documents and draw up research and evaluation contracts? What are the implications for research, researchers and those who are researched? It is, therefore, appropriate to address these issues through an analysis of the research process.

What Kind of Research is Commissioned?

Anyone who is interested in gaining research funding needs to take account of the specific projects that will be funded by organizations, trusts and government departments. For example, the Department of Education and Science (now the Department for Education) specify that they are concerned to commission or fund projects that are ‘policy related’ if they:

1. will help to guide policy decisions that need to be taken;
2. will help to improve the quality of the educational process in areas of policy concern;
3. will facilitate the implementation of policy decisions; or
4. will evaluate the effects of the implementation of policy decisions (DES, 1982).

Such a range of projects suggests research, research and development, and evaluation activities. Furthermore, many projects originate within the Department so that they reflect current interests. For example, the Department regularly requests proposals for research-based evaluation of education support grants and, more recently, on curriculum and assessment given the direction of national policy. Similar trends can also be seen in requests to tender for research from such diverse organizations as the former Training Agency and the Economic and Social Research Council (ESRC) where initiatives such as Information Technology and Education result in a careful specification of project areas and project evaluation that is supported by research council money.

At this stage, researchers need to consider areas in which they have substantive and methodological competence to get involved. However, it does not end there. In some cases, budgets are small, time-scales short and a field force is assumed to be on stand-by to conduct fieldwork should the research bid be successful. Here consideration needs to be given to a number of issues by the researcher and by the research sponsor:

1. Are the funds that are available sufficient to hire adequately trained staff with the appropriate training, research skills and substantive experience?
2. Are the time-scales appropriate? Often tender documents specify a time budget (which is important if the project staff are to meet deadlines). However, it is important to consider whether sufficient time has been given to data collection and analysis, especially where case-study methods are specified. For example, some organizations consider that case-studies can be conducted in five days. In these circumstances, the research community needs to discuss carefully with the sponsor the expectations in terms of methodology.
3. Can appropriate staff be recruited in time for the project? Unless core funds are provided for a research centre or a department, it is virtually impossible to have a field force on stand-by in the way that market research companies have interviewers available. Furthermore, once we move beyond the ‘standard interview’ to engage in observational work and the analysis of documents, a broader range of skills are required.

More fundamental than these questions, however, is the topic to be researched.

What Is to Be Researched?

We have already acknowledged that there has been a change from problems specified by the researcher to a situation where research issues are specified by the sponsor. Here, we should recall the advice given by Lawrence Stenhouse (1984) on relations between proposal writing and sponsorship:

When it comes to research aims, I think I have always worked with double sets of aims — one for the sponsor and one for myself! The proposal sets out the job I hope to do for the sponsor.… The whole art of funded research is to find scope for your own aims within and alongside the sponsor’s aims — and without costing the sponsor anything (p. 213).

How might this duality work in practice? Here, I turn to a request I received from the Assessment of Performance Unit to tender for a four-year programme of work that involved a survey of 7-year-old pupils. The tender briefing documents stated that proposals will:

1. develop a variety of test instruments within and across the core curriculum of English language, mathematics, science and technology (as in the proposed National Curriculum) for 7-year-old pupils;
2. conduct a national survey in 1991 of the performance in the core curriculum areas of top infant pupils of all abilities, drawn from maintained and independent schools in England, Wales and Northern Ireland;
3. analyze the resulting data and produce a report thereon;
4. prepare appropriate materials for disseminating the findings from the survey so as to assist teachers in assessing children of this age group as well as enabling them to appreciate the relevance of the findings to their teaching practices (APU, 1988).

Such a programme of work raised several issues that had to be addressed prior to writing the proposal. First, is this the kind of work in which one wished to be involved? Second, did our team have the necessary expertise for such a project? Finally, could we manage such a project and keep to deadlines? In answer to these questions, first, the project aims signalled that research and development activities were to be involved in close liaison with schools, pupils and teachers. In these circumstances, it appeared that we would be able to engage in research-based development activity in the field of curriculum and assessment. Second, we did have expertise in research on curriculum and assessment, and in the methodology that was required. Furthermore, we did have links with a variety of local authorities, schools and teachers throughout the country which would facilitate such work. Finally, our experience involved working to deadlines — an essential requirement in this project as anyone who took on the project and failed to meet the deadline of having all test instruments prepared for the 1991 survey would be in considerable difficulty. On this basis, it seemed we were an appropriate organization to mount a bid given the degree of close correspondence between our aims and those of the potential sponsor.

Yet in any proposal it becomes important to work within and go beyond the aims and objectives that have been carefully specified by the sponsor. Furthermore, it is essential to demonstrate the key elements of the project that would be examined. In our proposal I specified six issues that emerged from the tender document and which would link with our interests and expertise. They were:

1. the importance of focusing on the curriculum in English language, mathematics, science and technology together with the implications for assessment and testing;
2. the importance of cross-curricular and subject-integrated approaches which are consistent with the attainment targets drawn up by the Mathematics and Science Working Groups;
3. the relationship of continuous assessment and standardized assessment tasks to the programmes of study drawn up by the Working Groups;
4. the importance of development and trialling of test items;
5. the moderation of tests by teachers;
6. systematic training in assessment procedures for teachers.

Here, I saw the importance attached not only in the briefing document but also in the TGAT report (DES, 1987a) to links made between the curriculum, assessment and testing so much so that detailed first-hand observations would need to be made of current curriculum content and current procedures for assessment and testing in a range of infant schools. In short, it was an opportunity within the project aims to document variations in the pattern of ‘delivery’ of the National Curriculum as we had yet to discover how infant school teachers in state and independent schools in different areas of England and Wales would come to terms with teaching a range of subjects to infant pupils. Second, it provided an opportunity to make comparisons across the state and independent sectors not only of curriculum delivery but also of performance and the way in which testing and assessment took place in different social settings. Third, it allowed case-studies and multi-site case-studies (thirty-two in my proposal) to be used for pilot work and data collection as a basis for the development and trialling of test items prior to the National Survey in 1991. The conceptual framework of the project therefore involved the construction of tests from aspects of practice which were compatible with the National Curriculum but were not directly specified in it; we had to keep in mind a principle that Stenhouse identified in his classic text An Introduction to Curriculum Research and Development where he stated:

For a curriculum one does not look at a book but at the school. If curriculum is defined in this way, then the study of curriculum can be reduced to the empirical study of schools. The curriculum is not the intention or prescription but what happens in real situations. It is not the aspiration, but the achievement. The problem of specifying the curriculum is one of perceiving, understanding and describing what is actually going on in school and classroom (1975, p. 2).

On this basis, he argued that we need to examine the school as an agency of teaching and learning, based on the interpretation of careful observation.

Certainly, it was this approach that appeared in our proposal where we developed the case-study approach to a multi-site case-study which facilitated making comparisons on curriculum and assessment but where, as an additional spin-off, further work could be done on the use of the case-study and multi-site case-study in research and development activities on curriculum and assessment as thirty-two schools would be used for the following purposes:

1. To conduct observational studies in schools and classrooms so as to identify some of the key features of English, mathematics, science and technology and strategies for testing in these areas. Here, the researchers would need to work alongside advisers, teachers and pupils. These observations would be complemented by in-depth interviews with advisers and teachers about:
   1. curriculum content,
   2. skills training, and
   3. testing procedures.
   In addition, the observational and interview material would need to be complemented with documentary evidence obtained from the schools in the form of syllabuses, schemes of work, tests and assessment schemes. This work would be conducted simultaneously with test development. It would therefore allow the project team to see how infant teachers were simultaneously handling the ...

Table of contents

• Cover
• Half Title
• Title
• Copyright
• Contents
• List of Abbreviations
• Preface
• 1 Biting the Hand that Feeds You? Educational Research for Policy and Practice
• Part One: Issues in Policy Focused Research and Evaluation
• Part Two: Educational Research and Evaluation for Policy and Practice: Some Empirical Studies
• Notes on Contributors
• Name Index
• Subject Index