Functions, Methods and Concepts in Evaluation Research

R. Stockmann, W. Meyer

• Publisher: Palgrave Macmillan
• Year: 2013
• ISBN:
  9781137012470
• Topic: Política y relaciones internacionales
• Subtopic: Política pública

By explaining the role of evaluation in modern societies and its historical development in the USA and Europe this book highlights the scientific roots of Evaluation and offers an overview of its fundamental theories and concepts. The process of collecting, analyzing and interpreting is explored giving an insight into the course of an evaluation.

1

The Role of Evaluation in Society

Reinhard Stockmann

1.1 Evaluation – an invention of modernity

Modern societies are characterized by the replacement of traditionally and religiously determined beliefs regarding the world order by faith in rationality and progress. As early as the end of the 1960s, Daniel Lerner (1968: 387) produced an incisive summary of the features of modernity: they include a growing, self-sufficient economy; democratic participation in the political sector; a culture which orients itself toward secular, rational norms; a society that is performance-oriented, equipped with personal freedom, and is spatially, socially and intellectually mobile. Connected with this construct is the idea that modernity is a universal phenomenon, and that it can be shaped and governed (cf. for example Degele & Dries 2005; Hill 2001 for introductions). But that is not all: in view of the social, economic and ecological problems which accompany modernity and have noticeably come to a head in recent years, the idea has meanwhile established itself that social development is by no means a thing that will simply look after itself, but urgently needs to be steered resolutely toward ‘sustainable development’ (cf. Federal Government 2001).

However, for it to be possible to have an effect on social processes of change, their determining factors and impact relationships must be known. It is a matter of recognizing which impacts can be brought about by purposive interventions under given framework conditions, so that effective measures, programmes, strategies and policies can be designed and implemented. As modern societies are characterized by a plurality of different lifestyles and opinions and equality of rights with regard to the say citizens have in social development, these objectives compete with one another. The effectiveness of interventions and their contribution to the public good (i.e. the good of all) thus become crucial decision-making criteria. Having said that, it is not only a matter of learning from experience – in other words understanding correlations in retrospect and seeing the rationale in them – and measuring ex-post changes and investigating whether or not they are actually consequences of the strategies deployed to bring about change. In view of the challenge of governing complex and extremely heterogeneous social systems and the risks associated with misgovernment, the task of shaping the main influencing factors of social processes as early as possible, on the basis of rational insights, becomes increasingly important. For this, too, sound and relevant data are required.

With the above, some of the main tasks of evaluation have already been outlined: evaluation is the instrument with which summatively observed social changes can be measured, analysed and assessed, and through which formative data for the rational control of processes generated. Thus the insights from social science research into social relationships and the investigative methods developed in order to gain those insights are utilized for the political practice of actively shaping social processes. A culture of common learning about government impacts is thus striven toward (cf. Chelimsky 2006). By making available up-to-date scientific information through public discussion, evaluation also contributes to the rationalization of political debates about social objectives and thus shows what is feasible for all those involved in a clearly recognizable way.

Evaluation is an invention of modernity (for the relationship between evaluation and modernity see especially Dahler-Larsen 2011). It is on the one hand linked to the vision of economic and social progress, the pursuit of growth and continuous improvement, and on the other hand to faith in the feasibility and controllability of social development. Evaluation offers itself both as an instrument of enlightenment which sheds light on development processes, and as an instrument of control which aims to influence those processes purposively. Above and beyond that, evaluation is also suitable for reflexive use, as an instrument for the criticism of modernity itself. Because it can be used to record not only the intended impacts of interventions but also their unintended ones, evaluation provides the empirical basis for social self-reflection.

It has meanwhile become clear that in modern societies, which are becoming more and more complex, development strategies and policies must be questioned more radically than before, because of side effects which are undesired and in some cases decidedly harmful. This means that problems that up to now have only been treated as external (e.g. the environment), unintended consequences of purposively rational acts and the ability of those consequences to endure in the future (sustainability) must be integrated in the assessment to a greater extent. By these means, social action can be placed on a more rational basis and the public capacity for management increased.

From these considerations, the conclusion may be drawn that evaluation has never been as necessary as it is today. Evaluation does not merely support faith in progress by simply comparing the targets and achievements of the desired objectives with actual statuses. By also focusing especially on side effects and unintended impacts in its analyses, it detaches itself from a purely technocratic view of things, questioning progress itself. Only with a holistic perspective and a comprehensive impact approach can it pay heed to the sustainability of the solutions implemented.

1.1.1 Purposes

From the previous remarks, it becomes clear that evaluation can be carried on (1) in the service of social enlightenment. In this case it is primarily a question of assessing political strategies, programmes and measures with the instrument of evaluation to see whether or not they make a contribution to solving social problems. Creating transparency as regards the objectives and impacts of such strategies and measures enables assessments to be made on a rational basis. For example, by disclosing which political objectives are being achieved and which neglected, who benefits from such measures and who does not, which problems are solved and what risks are associated with solving them, evaluation can trigger public discussion. By doing so, it opens up the possibility ‘[of helping] society shape its own future in a qualified way through systematic, data-based feed-back. A society which seeks evaluation is one which prefers rational thought and critical inquiry to tradition, ideology, and prejudice’ (Dahler-Larsen 2006: 143).

Be that as it may, evaluation must render its assessment criteria transparent in order not to be exposed to accusations along the lines of only having adopted the perspective of the political elites and decision-makers. Evaluation findings should be discussed in the public sphere, i.e. the central institution in which modern societies guarantee the exchange between the state and its citizens. Making evaluation findings accessible to the general public stimulates the debate about social problems and the political solutions proposed for them. Only if the assessment criteria are identified can evaluation promote an objective discourse, defuse ideologically motivated conflicts and contribute by means of solution-oriented recommendations to a consensus-oriented conclusion (cf. also Section 6.2).

Evaluation findings are always assessive judgements. It is not until the criteria applied have been disclosed that the rationale can be seen in the judgements made in an evaluation; only then does the possibility manifest itself of arriving at other assessments by applying other criteria. It is not the findings of an evaluation, based on systematically gathered data on specified aspects, that represent a subjective value judgement, but rather the assessment criteria stipulated in advance, and at the end of the day that judgement cannot be objectified. As long as the assessment criteria on which findings are based are made transparent in the relevant public discourse, evaluation contributes to separating these interest-guided value judgements from the objective realm of facts, thus making them accessible to social discussion.

By observing and assessing public action and rendering it transparent with the aid of its concepts and procedures, evaluation assumes a social enlightenment function which is similar to that of journalism. Eleanor Chelimsky (2006: 33), for many years director of the Institute for Program Evaluation at the U.S. General Accounting Office (now known as the Government Accountability Office) and thus familiar with the system of politics and evaluation from the inside, characterizes the special merit of evaluation thus: ‘its spirit of scepticism and willingness to embrace dissent help keep the government honest’. By disseminating evaluation findings, it enhances the degree to which the public is kept informed about government action, but also about the activities of civil society with its many different non-government organizations (NGOs). It is only through the independent examination of the effectiveness and problem-solving competence of government programmes and measures that civil society is empowered to express competent criticism and to elaborate alternative proposals for solutions (cf. also Section 6.2).

Evaluation is not only part of society’s control of the state, but also (2) an essential element of democratic governance. Evaluation is used on the one hand by legislatures, having been made compulsory in laws and ordinances for certain purposes and accordingly having to be implemented by the executive agencies. In other words the legislators use evaluation as a means of keeping an eye on the impacts of executive measures and thus of enabling themselves to make objective judgements in further developing legal framework conditions in parliaments and their subordinate (e.g. specialist) committees. Both the juridically fixed framework conditions, i.e. the extent of the obligation to evaluate, and the scope and type of the prescribed evaluations vary from country to country and are subject to change over time. This will be looked at more closely in the section that follows about the historical development of evaluation research. In general it can be said that in the last twenty years in particular there has, in all modern societies, been a clear increase in the number of public evaluation assignments and the degree to which they are binding.

Not only the legislatures however, but also the executives, in other words governments and their ministries and public administrations, are using evaluation more now. If these public institutions use evaluation to prove that they are achieving their set objectives (effectiveness), what impacts (including the unintended ones) have been triggered, what the ratio of cost to benefit is (efficiency) etc., the credibility and legitimacy of policy can be improved. If clear and logical reasons can be provided as to why certain programmes are being discontinued, cut or expanded, the acceptance of decisions, or at least people’s understanding of them, increases. At the same time, the disclosure of the difficulties associated with political measures and a knowledge of correlations and the impacts caused by political strategies also promote the readiness of civil society to take part actively in solving these problems and support the government with contributions of their own for the good of all.

However, a prerequisite for this is that evaluation findings be used as a rational basis for political decisions. Donald Campbell (1969) picked up on this idea in his concept of the ‘experimenting society’, in which a kind of ‘work sharing’ between evaluation and political decision-making is propagated. According to that concept, the rational knowledge gained in evaluations should be translated directly and quasi-automatically into political decisions. This form of link between evaluation and politics has been harshly criticized as a reduction of political issues to technical ones and referred to as ‘social engineering’. Not only that, but studies showed early on ‘that the official political machinery did not actually behave according to the assumed rationalistic model’ (Dahler-Larsen 2006: 143). The use of evaluation findings is a complex social and political process, which should be further rationalized in organizations for example by the introduction of knowledge management systems (cf. for example Becerra-Fernandez & Leidner 2008; Haun 2005; Amelingmeyer 2004; Götz & Schmid 2004a, b; Winkler 2004; Willke 2004; Ipe 2003; Alvesson & Karreman 2001 for an overview). Conducting evaluations is without doubt not adequate sufficient condition for rational politics, but it is at least a necessary one: unless the results achieved by governmental and administrative action are disclosed, it is very difficult indeed to form a democratic opinion on the basis of assessments which are really rational.

Evaluation can make a contribution not only to social enlightenment and to s...

Table of contents

• Cover
• Title
• Introduction
• 1 The Role of Evaluation in Society
• 2 Science-Based Evaluation
• 3 Evaluation Approaches and Their Fundamental Theoretical Principles
• 4 The Evaluation Process
• 5 Gathering and Assessing Information
• 6 The Evaluation Environment
• Bibliography
• Index