The Handbook of Environmental Policy Evaluation

Ann Crabb, Pieter Leroy

• Publisher: Routledge
• Year: 2012
• ISBN:
  9781136553561
• Edition: 1
• Topic: Economics
• Subtopic: Sustainable Development
• Index: Economics

Policy evaluation is an important and well-established part of the policy process, facilitating and feeding back to promote the ongoing effectiveness of policies that have been implemented or anticipating policies in the making. Environmental policy is a special case, presenting new complexities uncommon to other areas, which standard evaluation tools are ill-equipped to grapple with. It is also an area that is experiencing rapid growth throughout the world and knowledge is now needed at all levels of government and in NGOs, businesses and other organizations, all of whom are required to assess the effectiveness of their policies.

This handbook is the first guide to environmental policy evaluation in practice. Beginning with an introduction to the general principles of evaluation, it explains the particular complexities native to the environmental sphere and provides a comprehensive toolkit of evaluation methods and techniques which the practitioner can employ and refer to again and again. The authors also consider design issues which may face the policy evaluator, including involvement of stakeholders, the sensitivities between them, the a priori assessment of the evaluability of a field, the maximization of the utilization of the evaluations outcomes, and much more. Throughout, the theory is illustrated with practical examples from around the world, making this the essential companion guide for anyone tasked with ensuring that environmental policy fulfils its aims and achieves its potential.

1

Policy Evaluation and Environmental Policy

__________________________

The phenomenon of policy evaluation raises various questions with environmental policy-makers and with those civil servants and consultants who advise them: What is policy evaluation? How can it help (me)? How should policy evaluation be conducted? Which methods are available? The aim of this handbook is to help answer such questions in a practical way.

Anticipating the systematic presentation of a series of methods for (environmental) policy evaluation in Chapters 3 and 4, we first need to consider and to elucidate some of the basic concepts involved in policy, in policy evaluation and in the particularities of environmental policy evaluation. This chapter introduces some basic and yet fundamental concepts of policy evaluation, whereas Chapter 2 deals with the particular features of environmental policies and the specific issues that may arise from them.

The concepts and the elucidations dealt with hereafter are directly related to the actual definition of policy evaluation that is applied in this handbook: policy evaluation is a scientific analysis of a certain policy area, the policies of which are assessed for certain criteria, and on the basis of which recommendations are formulated.

This definition contains certain elements that are crucial to the reader’s understanding of this and subsequent chapters: policy, analysis, evaluation, criteria and recommendations:

• Policy: while everyone has some concept of what policy is, it is worthwhile considering some frequently applied definitions, which we will do in this chapter. Depending upon how ‘policy’ is viewed, a different perspective emerges on the notion of policy evaluation (see Section 1.1 ‘What is policy?’).

• Analysis and evaluation: in order to evaluate policy, one must first analyse it. This entails research into the what, how and why of a specific policy context. Such analytical questions always precede the actual evaluation: how effective, how fair, how enforceable and so on, is a given policy?

• Criteria: the essential difference between policy analysis and policy evaluation is that the latter is based explicitly on a set of specific assessment criteria. The precise nature of these criteria needs to be determined anew for each individual setting. The evolution of policy evaluation reflects a gradual change in the type of criteria applied (see Section 1.2 ‘Criteria for policy evaluation’).

• Recommendations: policy evaluation is generally not an end in itself. Its purpose is, rather, to improve policy in one way or another, even when the impact of evaluation studies on actual policies is an issue in itself, which will be briefly touched upon.

Finally, we need to point out that our definition of policy evaluation implies that three types of knowledge come into play when evaluating policy: analytical knowledge, which describes and explains (what is …?; how does … work?; why is …?); evaluative knowledge, which assesses (how good is …?; how suitable is …?); and prescriptive knowledge, which recommends alternatives (how might … be approached differently?). This handbook focuses deliberately on methods designed to generate evaluative knowledge about policy processes, policy products and policy fields. It devotes no or very little attention to the technicalities of methods. In other words, it leaves aside questions such as how do I conduct interviews? How do I reconstruct a policy process? Nor does it touch upon methods of policy prescription, with questions such as how should I formulate recommendations? How should I test the feasibility of these recommendations? What the reader will find, though, is a broad outline of various policy evaluation methods in Chapters 3 and 4.

This chapter consists of two sections. First, light is shed on some of the key concepts: ‘policy’ and the different views on policy are the focus of Section 1.1; ‘policy evaluation’ and the various criteria involved are discussed in Section 1.2. In order to prevent the chapter from being a purely theoretical survey, we provide boxes with various well-known and less well-known definitions; give concrete examples; and point out practical problems. For the same reason, we use very few references in the chapter itself, but we provide an overview of some of the publications we have used as our main sources of inspiration at the end of the chapter. More detailed references to specific scientific literature are provided in Chapters 3 and 4.

1.1 WHAT IS POLICY?

1.1.1 The policy cycle

From the huge number of books in policy studies and policy analysis, one can draw an even greater number of definitions and concepts of policy. All of them attempt to grasp the very heart of the complexity that ‘policy’ represents, and to reduce its complexity by using relatively accessible and understandable models. One of the most popular concepts in policy studies is to simplify the policy-making processes into a series of stages. This conception originates from the early days of policy analysis, and has been altered and sophisticated since, but essentially draws on a parallel with production processes. It conceives policy-making as an ongoing iteration and reiteration of a policy cycle (see Figure 1.1), in which more or less consecutive stages and the according policy processes can (analytically) be distinguished.

Figure 1.1 The policy cycle

To further elaborate upon the policy cycle idea, we briefly discuss some of its steps:

• A number of problems or societal focal points, experienced to different degrees by people or groups, are highlighted socially and politically. In a number of cases, this may lead to the problem being placed on the political agenda. This is the agenda-setting phase. As the political agenda constantly tends to be overloaded, the existing problems are selected and prioritized. This implies that certain problems may be removed from the agenda, while others may be reformulated. These processes of problem definition and selection are determined by (environmental) incidents, by deliberate agenda-setting and problem definition strategies, and by the structural characteristics of the political system, which make it more sensitive to some problems than to others.

• In the policy-making process, solution options are developed. These solution options are presented, selected and compared in the media by interest groups, citizens and civil servants, as well as within political circles. Sometimes, scientific considerations or methods, such as cost-benefit analysis, also come into play. Opinions start to form, advocates and opponents begin to group together, and political pressure in favour of divergent solutions gradually builds up. In short, various policy options are generated and selected. ‘Good’ problem analysis and ‘good’ strategy formulation must, after all, promote a successful approach and its implementation. As a matter of fact, problem analysis and strategy formulation run largely parallel to one another.

• The forming of opinions on the remaining options takes on a more decided character. The main social and political forces adopt a position on the various solutions. This is not a ‘neutral’ choice, but an entirely political one: although goal-oriented, it is partly informed by political and societal viewpoints (on the role of government, the responsibilities of the business community, etc.), and, of course, by the balance of power between various social and political forces in society. This forming of opinions leads to political decisions, in which the basic ideas of the policy, its aims and its strategies are set out.

• The next phase is that of policy implementation. Basic political decisions must first be ‘concretized’ into specific measures: mobilizing and setting aside the required resources (manpower and budgets), specifying procedures and rules, and determining the division of tasks and the coordination between the various organizations entrusted with implementation. In sum, it involves a mobilization of instruments and organizational planning, the actual functioning of which is also referred to as ‘policy output’. The question remains, though, whether all the resources, knowledge, power, legitimacy, etc. that are necessary for implementing the policy are, indeed, available. Furthermore, policy implementation is partly down to actors (government organizations, companies and others) who are not necessarily in favour of policy action, which entails a risk that implementation will be hindered or distorted. Policy-making is a multi-actor and a multilevel phenomenon, which partly explains the sometimes substantial differences between policy on paper and policy in practice. While the latter may be referred to as ‘policy performance’, the differences between ambitions and actual performance are known as the implementation deficit. This may, but need not, have a detrimental effect on the success of a policy.

• In the fifth stage, the policy effects manifest themselves. First and foremost, effects concern the intended changes in the behaviour of the target group (such as companies, households and motorists). This is referred to as the ‘policy outcome’. Second, the term effects may refer to the intended environmental impacts (i.e. to an improvement in environmental quality). For that matter, the relationship between the two is not easy to identify. After all, a number of the outcome effects are only partly realized or not realized at all. And the end effects, the effects on the environment, are often realized only in the long run. Chapter 2 will go into these difficulties of evaluation that are partly specific to the environmental domain. In addition to these intended effects, unintended side effects may be generated that can potentially neutralize the intended effects. The controversial role of biofuels in recent debates and policies on climate change, including their unintended effects on agricultural policies, and therefore on global poverty and development, provides an example.

• In addition, while the policy process goes on, simultaneous and rather autonomous societal developments are occurring as well: while the environmental policy is being implemented, economic growth, individualization, globalization and other societal developments continue to unfold. Simultaneously decisions are taken in other policy fields, such as spatial planning, traffic and others, that will have an impact upon either the environmental quality or upon the target groups’ behaviour. These developments, together with the outcomes of the environmental policy process itself, will lead to newly emerging social problems and therefore to renewed problem formulation and agenda-setting. In short, the policy cycle will be reiterated.

BOX 1.1 TYPES OF POLICY EFFECTS

The question of whether (environmental) policy is having an impact – many more aspects of which will be discussed below – inevitably gives rise to a counter-question: what kind of impact?

Traditionally, a distinction is made in the literature between:

• Policy output or policy performance: this entails the quantity and quality of the products and services delivered by policy-makers – for example, the number of permits granted, the revenue generated through environmental taxation, actual expenditures on environmental subsidies, the amount of information provided, etc.

• Policy outcome or social change (also referred to as behavioural effect): this comprises policy impact in terms of behavioural change among citizens, companies, farmers, consumers and other groups in society – for example, modified driving behaviour, private investment in water treatment infrastructure, increased purchases of biological products, etc.

• Environmental impact or environmental change (also referred to as ecological impact): examples of this include a reduction in nitrogen oxide (NO_x) or sulphur dioxide (SO₂) emissions, a decrease in water pollution or more land for organic farming.

The policy cycle is a useful concept and metaphor. After all, it reveals the different ideal-typical stages in the policy process and, in so doing, the policy-making activities that theoretically take place in each of those stages. The policy process is dissected into a kind of production process involving various sub-activities, each of which requires its own raw materials and treatment methods, as well as its own semi-finished products. This means that the activities in each of these policy stages can be methodically analysed and substantiated. What we are dealing with, then, are methods for policy development and for policy analysis. Many books are available in which, either for the entire policy cycle or for parts of it, various methods are described that may be used to support policy development. At the end of this chapter, a list is provided of some overview publications, while Chapters 3 and 4 refer to more specialized methodological literature.

BOX 1.2 NEITHER EX ANTE, NOR EX POST, BUT RATHER EX NUNC

A traditional distinction in policy evaluation is that between ex ante and ex post evaluation. The concepts are self-explanatory: ex post evaluation refers to the evaluation of policy after it has been developed and implemented, while ex ante evaluation means that policy is evaluated prior to implementation (i.e. while it is still very much on the drawing board).

Three comments are in place with regard to this distinction:

1 Methods for ex ante evaluation are, in fact, at once methods for policy design and policy development. Many handbooks deal with them under a single heading.

2 It is clear that ex post policy evaluation leads to new insights for policy development and ex ante policy evaluation. This learning effect, however, requires a minimum amount of ex post policy evaluation research that should, furthermore, provide sufficient generic (and not simply case-specific) insights. In Chapters 3 and 4 we also discuss approaches to policy evaluation in which specific attention is paid to this learning effect.

3 Most policy evaluation research is neither ex ante, nor ex post, but rather, ex nunc or ex in between, if you will: it deals with the evaluation of current policy, the interim evaluation of which may lead to modifications. Since policy is rarely completed, let alone terminated, there is very little ex post policy evaluation in the very strict sense of the word. Still, we will persist with the distinction between ex post and ex ante.

In the context of this book, a further aspect of the policy cycle and the position and role of evaluation is important. Many textbooks look at ‘evaluation’ as the penultimate stage of the policy cycle, with its final outcomes and the impact of simultaneous societal developments conceived to be the last stage. Other textbooks and our Figure 1.1, however, place ‘evaluation’ at the centre of the cycle, suggesting that each phase of the policy cycle may be evaluated. The methods we list in Chapters 3 and 4 differ in many respects, including that some of them are appropriate to one or, in particular, several of the policy cycles’ evaluation. It might be clear, indeed, that policy evaluation questions may relate to each or to several of these phases. By way of illustration, we consider some environmental policy problems with examples of evaluative questions about successive stages of the policy cycle:

• Problem formulation: livestock manure. From the late 1970s until today, different European countries have faced environmental problems related to intensive livestock farming (i.e. problems of increasing volumes of manure). From international comparative research on this period, it is clear that these problems have been defined successively as a non-issue, a regional surplus, a processing problem and a matter of volume control. For domestic reasons, varying from a country’s physical geography to the power relations between its main stakeholders, the problem has further been defined in terms of water pollution, acidification, an odour problem, a technical problem within the agricultural sector, etc.

From a policy evaluation point of view,...

Table of contents

• Cover
• Halftitle
• Title
• Copyright
• Contents
• List of Figures, Tables and Boxes
• About the Authors
• Introduction
• Acknowledgements
• List of Acronyms and Abbreviations
• 1. Policy Evaluation and Environmental Policy
• 2. Problems and Limitations of Environmental Policy Evaluation
• 3. Approaches to (Environmental) Policy Evaluation
• 4. Approaches for Designing Evaluation Research
• Index