Handbook on Impact Evaluation

Quantitative Methods and Practices

Shahidur R. Khandker, Gayatri B. Koolwal, Hussain A. Samad

• Publisher: World Bank
• Year: 2010
• ISBN:
  9780821380284
• Topic: Business
• Subtopic: Ricerca e sviluppo
• Index: Business

This book reviews quantitative methods and models of impact evaluation. The formal literature on impact evaluation methods and practices is large, with a few useful overviews. Yet there is a need to put the theory into practice in a hands-on fashion for practitioners. This book also details challenges and goals in other realms of evaluation, including monitoring and evaluation (M&E), operational evaluation, and mixed-methods approaches combining quantitative and qualitative analyses. This book is organized as follows. Chapter two reviews the basic issues pertaining to an evaluation of an intervention to reach certain targets and goals. It distinguishes impact evaluation from related concepts such as M&E, operational evaluation, qualitative versus quantitative evaluation, and ex-ante versus ex post impact evaluation. Chapter three focuses on the experimental design of an impact evaluation, discussing its strengths and shortcomings. Various non-experimental methods exist as well, each of which are discussed in turn through chapters four to seven. Chapter four examines matching methods, including the propensity score matching technique. Chapter five deal with double-difference methods in the context of panel data, which relax some of the assumptions on the potential sources of selection bias. Chapter six reviews the instrumental variable method, which further relaxes assumptions on self-selection. Chapter seven examines regression discontinuity and pipeline methods, which exploit the design of the program itself as potential sources of identification of program impacts. Specifically, chapter eight presents a discussion of how distributional impacts of programs can be measured, including new techniques related to quantile regression. Chapter nine discusses structural approaches to program evaluation, including economic models that can lay the groundwork for estimating direct and indirect effects of a program. Finally, chapter ten discusses the strengths and weaknesses of experimental and non-experimental methods and also highlights the usefulness of impact evaluation tools in policy making.

Table of contents

• Contents
• Foreword
• Preface
• About the Authors
• Abbreviations
• Part 1 Methods and Practices
• Part 2 Stata Exercises
• Answers to Chapter Questions
• Appendix: Programs and .do Files for Chapter 12–16 Exercises
• Index
• Box 2.1 Case Study: PROGRESA (Oportunidades) in Mexico
• Box 2.2 Case Study: Assessing the Social Impact of Rural Energy Services in Nepal
• Box 2.3 Case Study: The Indonesian Kecamatan Development Project
• Box 2.4 Case Study: Monitoring the Nutritional Objectives of the FONCODES Project in Peru
• Box 2.5 Case Study: Mixed Methods in Quantitative and Qualitative Approaches
• Box 2.6 Case Study: An Example of an Ex Ante Evaluation
• Box 3.1 Case Study: PROGRESA (Oportunidades)
• Box 3.2 Case Study: Using Lotteries to Measure Intent-to-Treat Impact
• Box 3.3 Case Study: Instrumenting in the Case of Partial Compliance
• Box 3.4 Case Study: Minimizing Statistical Bias Resulting from Selective Attrition
• Box 3.5 Case Study: Selecting the Level of Randomization to Account for Spillovers
• Box 3.6 Case Study: Measuring Impact Heterogeneity from a Randomized Program
• Box 3.7 Case Study: Effects of Conducting a Baseline
• Box 3.8 Case Study: Persistence of Unobserved Heterogeneity in a Randomized Program
• Box 4.1 Case Study: Steps in Creating a Matched Sample of Nonparticipants to Evaluate a Farmer-Field-School Program
• Box 4.2 Case Study: Use of PSM and Testing for Selection Bias
• Box 4.3 Case Study: Using Weighted Least Squares Regression in a Study of the Southwest China Poverty Reduction Project
• Box 5.1 Case Study: DD with Panel Data and Repeated Cross-Sections
• Box 5.2 Case Study: Accounting for Initial Conditions with a DD Estimator—Applications for Survey Data of Varying Lengths
• Box 5.3 Case Study: PSM with DD
• Box 5.4 Case Study: Triple-Difference Method—Trabajar Program in Argentina
• Box 6.1 Case Study: Using Geography of Program Placement as an Instrument in Bangladesh
• Box 6.2 Case Study: Different Approaches and IVs in Examining the Effects of Child Health on Schooling in Ghana
• Box 6.3 Case Study: A Cross-Section and Panel Data Analysis Using Eligibility Rules for Microfinance Participation in Bangladesh
• Box 6.4 Case Study: Using Policy Design as Instruments to Study Private Schooling in Pakistan
• Box 7.1 Case Study: Exploiting Eligibility Rules in Discontinuity Design in South Africa
• Box 7.2 Case Study: Returning to PROGRESA (Oportunidades)
• Box 7.3 Case Study: Nonexperimental Pipeline Evaluation in Argentina
• Box 8.1 Case Study: Average and Distributional Impacts of the SEECALINE Program in Madagascar
• Box 8.2 Case Study: The Canadian Self-Sufficiency Project
• Box 8.3 Case Study: Targeting the Ultra-Poor Program in Bangladesh
• Box 9.1 Case Study: Poverty Impacts of Trade Reform in China
• Box 9.2 Case Study: Effects of School Subsidies on Children’s Attendance under PROGRESA (Oportunidades) in Mexico: Comparing Ex Ante Predictions and Ex Post Estimates—Part 1
• Box 9.3 Case Study: Effects of School Subsidies on Children’s Attendance under PROGRESA (Oportunidades) in Mexico: Comparing Ex Ante Predictions and Ex Post Estimates—Part 2
• Box 9.4 Case Study: Effects of School Subsidies on Children’s Attendance under Bolsa Escola in Brazil
• Figure 2.1 Monitoring and Evaluation Framework
• Figure 2.A Levels of Information Collection and Aggregation
• Figure 2.B Building up of Key Performance Indicators: Project Stage Details
• Figure 2.2 Evaluation Using a With-and-Without Comparison
• Figure 2.3 Evaluation Using a Before-and-After Comparison
• Figure 3.1 The Ideal Experiment with an Equivalent Control Group
• Figure 4.1 Example of Common Support
• Figure 4.2 Example of Poor Balancing and Weak Common Support
• Figure 5.1 An Example of DD
• Figure 5.2 Time-Varying Unobserved Heterogeneity
• Figure 7.1 Outcomes before Program Intervention
• Figure 7.2 Outcomes after Program Intervention
• Figure 7.3 Using a Tie-Breaking Experiment
• Figure 7.4 Multiple Cutoff Points
• Figure 8.1 Locally Weighted Regressions, Rural Development Program Road Project, Bangladesh
• Figure 11.1 Variables in the 1998/99 Data Set
• Figure 11.2 The Stata Computing Environment
• Table 11.1 Relational and Logical Operators Used in Stata