Evaluation for Participation and Sustainability in Planning

Angela Hull, E.R. Alexander, Abdul Khakee, Johan Woltjer

• Publisher: Routledge
• Year: 2012
• ISBN:
  9781136700422
• Edition: 1
• Topic: Architecture
• Subtopic: Urban Planning & Landscaping

Planning evaluation is required to establish the success of planning interventions – both of physical developments and new approaches. Yet this should not be a task undertaken purely by professionals without participation by those affected by the process and outcomes of the projects. This book provides case studies and advice on how to balance conservation with economic growth, the cost effectiveness of plans alongside the effects upon the community and the importance of engaging with all stakeholders involved in a project.

Practical aspects of the evaluation process covered include:

• 
  
  

• how evaluation is used in planning
• introducing new kinds of information or criteria
• alternative ways of collecting/presenting information
• how strategic planning objectives are implemented in local practice.

International contributors provide empirical studies and cases of application which are of practical value to those involved in the evaluation of planning. The book concludes by offering a new paradigm – a locally oriented, context-specific, participatory and multi-disciplinary approach to planning evaluation.

Part 1

From Appraisal to Reflection: Advances to Existing Methods

Introduction

Abdul Khakee

This section contains four chapters and all of them seek either to develop a holistic and comprehensive framework or model of evaluation or to address the perceived gaps in existing models. Specifically, they reflect on how existing planning evaluation methods can address the issues of complexity born of competing forces attempting to determine the objectives of the planning system; vagueness in long-term unifying concepts such as “sustainable development”; interdependence between spatial planning and other public policies; “wickedness” of planning problems and the bounded capacity of human rationality. All are based on normative conceptions of the purpose and value of evaluation and engage with the conceptual and methodological issues relevant to the retrospective analysis of planning process and planning results. These issues include linearity of planning process, “grounded theory”, intended versus unintended outcomes, policy analysis and evaluation, causality and values in planning.

Each of the chapters in this section makes a useful contribution to the critique of existing methods for evaluation. The chapters by Cecilia Wong and E. R. Alexander have a broad framework. Whereas Wong’s aim is to develop robust and reliable indicators to evaluate the performance of spatial planning, Alexander’s aim is to develop a holistic and comprehensive approach to evaluate planning. The other two chapters present more specific methods; Morten Edvardsen introduces “backward mapping” in order to carry out ex post evaluation of planning efforts and David Prosperi and Julia Lourenco put forward quadralogue to evaluate multiple place qualities and changing spatial identities.

Cecilia Wong, Decision-Making and Problem-Solving: Turning Indicators into a Double-Loop Evaluation Framework

There are several reasons for recent years’ tremendous interest in and use of indicators to evaluate policies and plans. Indicators provide necessary information in order “to produce high quality decisions”. They are “quick and cost-effective” and are “readily understood and easily comprehended”. Sustainable development, improvement in the delivery of public services and selection of best practice cases among public agencies are three policy issues where indicators have had ample use. However there are some major problems associated with the use of indicators. Indicators can be “mechanistic” and associated with “hierarchical top-down” management. They often make use of “inconsistent” and “out-of-date” data. In the absence of clear policy objectives, indicators “misrepresent priorities” and they may also oversimplify by ignoring “side-effects” and downgrading “qualitative measures” (Pearce, 2005: 131–7; see also Smith, 1995).

Cecilia Wong’s chapter addresses several of the shortcomings that prevail in the use of indicators. Her aim is to develop robust and reliable output indicators in order to evaluate the performance of spatial planning in England. The emphasis in Wong’s chapter is on the need to develop a “double-loop” evaluation framework that allows “key stakeholders to express their vision in the policy formulation process as well as providing a feedback loop to frame policy problems”. Wong calls for a more structured approach to evaluation from developing shared commitment through problem-structuring and flexible problem-solving, and to provide for continual feedback and double-loop learning. This approach is particularly useful because spatial planning problems are complex. The complexity depends on competing forces attempting to determine the objectives of the planning system, vagueness in long-term unifying concepts such as “sustainable development”, interdependence between spatial planning and other public policies, wickedness of planning problems and bounded capacity of human rationality. Wong’s double-loop model takes care of these issues in an innovative manner.

Wong’s chapter is divided into two sections: the first section briefly presents a conceptual analysis of different types of indicators and the second puts forward a set of spatial output indicators. The novelty in Wong’s chapter is the integrative, double-loop learning evaluative framework that promotes shared commitment as well as providing a comprehensive set of indicators for sustainable development.

E. R. Alexander, Evaluating Planning: What is Successful Planning and (How) Can we Measure it?

Alexander’s aim is to evaluate planning itself–as distinct from planning products and their outcomes, which are the subject of conventional project and programme evaluation; this means evaluating planning agencies and institutions, their activities and impacts. The practical need for such evaluation arises from enhanced concern for government accountability and the spread of public sector performance assessment programmes (Carmona and Sieh, 2004).

Here Alexander combines two separate aspects–each of which addresses a different question–to form the evaluation framework. The first question (Who wants to use this evaluation, for what purposes?) raises the issue of institutional design (Alexander, 2005a) of evaluation (Alexander, 2005b). The second (What is this planning effort for?) introduces the teleological aspect of evaluation (Alexander, 2009).

This framework outlines procedures for evaluating public planning agencies and their activities through a four-step model. The first defines the evaluation itself in its institutional setting as a problem of institutional design, where solutions depend on the sponsors’ identities and intentions. The second step is dialogical, to answer the teleological question and adopt an appropriate evaluation approach. The third step entails the identification of the subject of evaluation (entire planning systems or their constituents) and expected outputs (services, plans, policies and decisions). The last step includes the choice and application of evaluation approaches for the expected planning outputs.

The four-step model provides holistic and comprehensive assessment of a planning agency and its activities. Its innovative aspect is to ensure integrative and exhaustive knowledge for a complex set of practices and outcomes that are part and parcel of public planning.

Morten Edvardsen, Evaluations of Local Planning Efforts: A Simple Test of Policy Implementation and Corresponding Results?

Nathaniel Lichfield has defined evaluation as “the systematic procedure of observing, analysing and formulating a judgement on the operation and the results, both foreseen or effected, of a plan/programme/policy process”. For Lichfield, evaluation should include ex ante as well as ex post evaluation. Despite this, he notes that “plan review is very poorly linked to monitoring and ex post evaluation” because the latter is “almost non-existent”. He finds this surprising especially because “ex post evaluation is much more developed in the close field of urban programmes”. One of the major reasons for this situation is that methods for ex post evaluation are weakly developed when compared to ex ante evaluation and often consist of a “patchwork of methods and approaches” (Lichfield and Prat, 1998: 287–92).

Morten Edvardsen’s contribution addresses this imbalance and proposes a new method for assessing results of the planning process. He introduces “backward mapping” in order to carry out ex post evaluation of planning efforts. Since ex post evaluation is not well covered in planning evaluation literature, Edvardsen chooses a research design that is made up of case studies based on qualitative analysis as well as the inductive and exploratory approach. His proposal to use backward mapping in ex post evaluation is innovative because it enables the evaluation process to indicate the causal relations between planning outcomes with implementation efforts and the plan design. Backward mapping has been used successfully in policy analysis (Elmore, 1986). Edvardsen’s study is a promising effort for improving ex post planning evaluation.

Edvardsen’s chapter starts with a discussion of research design using a series of case studies based on qualitative analysis. This is followed by a brief discussion of several conceptual and methodological issues, which the author contends have relevance for the retrospective analysis of planning process and planning results. These issues include linearity of planning process, “grounded theory”, intended versus unintended outcomes, policy analysis and evaluation, causality and values in planning. The application of backward mapping to evaluate planning results in two Norwegian municipalities illustrates the causal relationship between planning efforts and the end results.

David C. Prosperi and Julia M. Lourenco, The Quadralogue Concept for Strategic Project Assessment

The use of discourse has become a key aspect in evaluation with the advent of the post-positivist challenge of interactive participatory evaluation (Guba and Lincoln, 1989). For example, communicative planning emphasises both interaction and iteration, which take place in an extensive institutional context, and where the aim is to obtain commitment and consensus among all the stakeholders. The central aspects of evaluation centre on how best to organise an inclusionary discourse, to promote a learning process that is emancipatory and expedites progress, and to emulate political, social and intellectual capital. A central aspect of evaluation is to focus on the quality of both the planning process and the programme of actions. Evaluation itself becomes a form of interactive discourse where all those involved can explain their values, problems and concerns (Healey, 1997; Khakee, 2003).

Interactive discourse can be carried out by means of dialogues, trialogues and, as David Prosperi and Julia Lourenco propose, “quadralogues”. The latter approach is especially useful in order to overcome intractable problems in a common discourse and to evaluate multiple place qualities and changing spatial identities. According to Prosperi and Lourenco urban social dialogue differs from regular forms of social dialogues in various ways because of scalar institutional dynamics and multiple realities. Previously researchers in the Flemish part of Belgium, the International Society of City and Regional Planners and the European Union have made use of trialogue in the planning efforts.

Quadralogue represents an improvement on the fast-track process of the trialogues by developing a concise mission (as in business strategic planning) and avoiding relapsing into singular identities and discrete relational webs. In order to do that it is necessary to include such concepts as scalar dynamics, unseating the normative hegemony in planning discourse and multiplicative versus additive realities.

Prosperi and Laurenco go on to operationalise the quadralogue concept in the context of six development projects–three in South Florida and three in the Algarve. Besides presenting the context of the projects and the results of the operationalisation of the quadralogue approach, they describe the urban social dialogue that really took place in each of the case studies. Their comparative study provides useful insights with regards to applying quadralogue in different social, economic and urban contexts.

References

Alexander, E. R. (2005a) Institutional transformation and planning: From institutionalization theory to institutional design, Planning Theory 4 (3): 209–223.

Alexander, E. R. (2005b) Implementing norms in practice: The institutional design of evaluation, in D. Miller and D. Patassini (eds) Beyond Benefit Cost Analysis: Accounting for Non-Market Values in Planning Evaluation, Aldershot: Ashgate, pp. 295–310.

Alexander, E. R. (2009) Dilemmas in evaluating planning, or back to basics: What is planning for? Planning Theory & Practice 10 (2): 233–244.

Carmona, M. and Sieh, L. (2004) Measuring Quality in Planning: Managing the Performance Process, Abingdon: Taylor & Francis.

Elmore, R. F. (1986) Backward mapping: Implementation research and policy decisions, Policy Science Quarterly, 94 (4): 601–616.

Guba, E. G. and Lincoln, Y. S. (1989) Fourth Generation Evaluation, Newbury Park, CA: Sage.

Healey, P. (1997) Collaborative Planning, London: Macmillan.

Khakee, A. (2003) The emerging gap between evaluation research and practice, Evaluation 9 (3): 340–352.

Lichfield, N. and Prat, A. (1998) Linking ex ante and ex post evaluation in British town planning, in Lichfield et al. Evaluation in Planning. Facing the Challenge of Complexity, Dordrecht: Kluwer Academic Publishers, pp. 283–298.

Lichfield, N., Barbanente, A., Borri, D., Khakee, A., and Prat, A. (eds) (1998) Evaluation in Planning: Facing the Challenge of Complexity, Dordrecht: Kluwer Academic Publishers.

Pearce, B. (2005) The use and abuse of indicators for evaluating land use and environmental planning: Experience from the UK, in Miller, D., and Patassini, D. (2005) Beyond Benefit Cost Analysis: Acounting for Non-Market Values in Planning Evaluation, Aldershot: Ashgate.

Smith, P. (1995) Performance indicators and control in the public sector, in Berry, A. J., Broadbent, J., and Otley, D. (eds) Management Control: Theories, Issues and Practices, London: Macmillan.

1

Decision-Making and Problem-Solving

Turning Indicators into a Double-Loop Evaluation Framework

Cecilia Wong¹

Introduction

Planning, as coordinator, integrator and mediator of space, has continued to seek a knowledge-driven and ev...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright
• Contents
• List of illustrations
• Contributors
• Introduction: new challenges in planning evaluation
• Part 1: From appraisal to reflection: advances to existing methods
• Part 2: Evaluating the impacts of policy-programme-project alternatives
• Part 3: Integration of ecological aspects into planning evaluation
• Part 4: Evaluation methods for structuring participatory knowledge
• Glossary
• Index