Program Evaluation

Forms and approaches

John M Owen

• Publisher: Routledge
• Year: 2020
• ISBN:
  9781000246933
• Edition: 3
• Topic: Medicine
• Subtopic: Health Care Delivery

Program Evaluation 3rd edition has the hallmarks of thoroughness, insight and fluency of other editions but brings us up to date with a map of the evaluation territory new travelers will find invaluable.' Professor Murray Saunders, UK and European Evaluation Societies John Owen's Program Evaluation helps practitioners clarify distinctions between what we evaluate, the various motivations for conducting evaluation, and the approaches attendant to each. Timely evaluation examples liberally sprinkled throughout this book make it exceptionally useful and helpful to practitioners. I highly recommend Program Evaluation.' Professor Marvin Alkin, UCLA Graduate School of Education + Information Studies Everyone involved with policy and program development and delivery is being asked to plan more carefully, reflect more critically and justify their decisions. The key to this is evaluation. Program Evaluation offers a conceptual yet straightforward and practical overview of the evaluation process for both beginners and experienced practitioners. It shows evaluators how to identify appropriate forms, approaches and methods, using an original framework. John Owen examines the contributions of evaluation to program provision, and offers proven techniques for involving stakeholders in the planning process and for disseminating the evaluation findings.This third edition has been thoroughly revised to incorporate recent research on evaluation and new examples of good practice. It includes sections of management of evaluation, negotiating evaluation plans, program logic and ex ante evaluation, evidence based practice, performance management and accountability.With international examples from a range of health, education, welfare, community and other settings, Program Evaluation is an essential reference for anyone involved in evaluation in both the public and private sectors.

1
Evaluation Fundamentals

This book is based on the premise that evaluation, as defined in this and following chapters, can and should enhance the quality of interventions (policies and programs) designed to solve or ameliorate problems in social and corporate settings. Evaluation should be seen as a process of knowledge production, which rests on the use of rigorous empirical enquiry. Evaluation will be worth the investment of time and money if the knowledge produced is reliable, responsive to the needs of policy and program stakeholders, and can be applied by these stakeholders.

Evaluation is not an alien activity for most of us. We engage informally in evaluative activity in our everyday lives. What clothes to wear on a given day, whether the plans for home renovations are satisfactory, how the new worker in the office is coping, whether our sporting team played well, are all examples of evaluation in our daily lives. In most of these examples, the evaluation is performed informally, for private purposes. That is, we often assemble information ‘in our heads’ based on a variety of sensory inputs, such as observation, and our existing knowledge, to make judgments about the issue under consideration. For example, an evaluation of how the new worker in the office is coping may rely on informal observations of performance and the opinions of others, including the worker herself.

This book should assist you to extrapolate your everyday understandings to more formal evaluation frameworks and procedures than those just mentioned. A comprehensive conceptual framework for anchoring your existing evaluation knowledge and practice is put forward. We have adopted an eclectic view of the field in order that you:

• may also see new directions for evaluation work; and can
• identify aspects of evaluation theory and practice that you wish to explore in more detail. This can be achieved by following up the relevant concepts and ideas through the references provided at the end of each chapter.

There is no doubt that evaluation practice has expanded and become more complex during the period of more than fifty years since the first formal evaluations were undertaken. Our intention is to encapsulate this complexity into a format that is both understandable and workable for you.

THE LOGIC OF EVALUATION

At the outset, we introduce you to what has been termed the ‘logic of evaluation’. Anyone interested in undertaking or commissioning evaluative work must be familiar with this logic. We can anchor the logic of evaluation in the everyday scheme of things by asking you to read the following extract from a ‘test report’ on breakfast cereals, typical of those found in Consumer Reports in the United States of America, or Choice magazine in Australia. The intervention or ‘object’ being evaluated here is a product, something that we buy to use or consume. Specifically, the extract describes the evaluation of a range of breakfast cereals.

When reading this extract, think about the following issues:

• What is the underlying basis for selecting criteria to judge the worth of each breakfast cereal?
• What evidence was used and on what standards was the judgment of worth made?
• How were the conclusions made and presented?
• Decision-making: you have been asked to recommend one brand of cereal to members of your household or to a friend. Which one will you choose?

Review

Let’s go back to our questions, and your answers, with a view to ‘pulling apart’ the evaluation exercise (Example 1.1). Later we will review this exercise using some concepts developed by evaluation theorists.

Example 1.1 Test report: Evaluating breakfast cereals

Most of us eat them, but just how healthy are they? We’ve assessed more than 80 breakfast cereals to find out which are the most nutritious, and compared them to other breakfast options.

Breakfast is probably the most important meal of the day, but it’s generally the most neglected one. By morning, around ten hours have usually passed since you last ate, so your body is running low on fuel. You need to replenish your stores, or your performance will suffer.

Studies have shown that by lunchtime, people who eat breakfast are functioning better than those who don’t. Adults who haven’t eaten breakfast are more likely to be involved in industrial accidents, and children who miss this meal suffer significant drops in concentration levels in the late morning.

Breakfast-eaters are also more likely to eat well throughout the rest of the day than those who give it a miss. If you meet your nutritional needs at the start of the day, you’re less likely to binge on sweet or fatty snacks for morning tea. If you’re trying to lose weight, there’s another reason you shouldn’t miss breakfast: studies have shown when you do, the body’s metabolic rate remains lower for the rest of the day—and with a low metabolic rate you burn fuel more slowly so you’re not shedding those kilos.

The ideal breakfast

It’s also important to feed your body the right kind of fuel in the morning. The best breakfast is one that is high in complex carbohydrate. Once digested, carbohydrate is stored in the muscles and liver as glycogen, a convenient storage form of glucose, which your body can then draw on throughout the day to fuel mental and physical activity.

Unprocessed cereals and grains—the starting point of breakfast cereals—fit this bill perfectly. They are high in complex carbohydrate, as well as being a good source of protein, fibre, vitamins and minerals. They also fit in with the Dietary Guidelines for Australians, issued by the National Health and Medical Research Council, in that they contain very little fat, sugar or salt.

But how nutritious are they after they’ve been processed, when things are often added to them or taken away? We looked at more than 80 breakfast cereals, assessing their nutritional profiles to find which are the most and the least nutritious.

There are over 300 different breakfast cereal products on the Australian market, which is valued at more than $600 million a year, so we weren’t able to look at them all. Our selection includes those widely available in supermarkets and all the top sellers in the ready-to-eat segment.

What makes a good breakfast cereal?

A nutritious breakfast cereal should be low in fat, sugar and salt as well as high in complex carbohydrate and dietary fibre. Some of the processed cereals on the market have retained many of the nutritional virtues of the original whole grain. But many more have had fat, sugar and salt added to them. Some have also had other nutrients added to them, often ones which were naturally in the original grain but were lost during processing, like fibre and thiamine (vitamin B1).

All up, we found 30 cereals that are a good choice to eat for breakfast; the data on these cereals are included in Table 1.1.

Table 1.1 Characteristics of breakfast cereals

Brand/type (alphabetically within groups)	Category	Fibre (g/) all = per 100g	Fat (g/)	Sugars (g/)	Sodium (mg/)	Carbo (g/)	Energy (kj/cal/)
Highly recommended (4% or less fat, 5% or less sugar, 7% or more fibre, 600 mg/100 g or less sodium)
HOME BRAND Wheat Biscuits	Biscuit	12.2	2.7	2.3	270.0	64.5	1380/330
KELLOGG’S Mini-Wheats Whole Wheat	Shredded wheat	9.3	2.6	0.9	3.0	77.1	1523/363
KELLOGG’S Wholegrain Wheat Flakes	Wheat-based	11.0	1.1	2.0	468.0	81.3	1437/382
SANITARIUM Lite-bix	‘Light’	12.0	2.7	1.2	20.0	62.0	1340/320
SANITARIUM Puffed Wheat	Wheat-based	7.5	2.6	1.0	17.0	71.0	1440/344
SANITARIUM Weet-Bix	Biscuit	12.2	2.7	2.3	270.0	64.5	1380/330
UNCLE TOBYS Organic Vita-Brits	Biscuit	12.4	1.4	1.8	400.0	65.6	1320/315
UNCLE TOBYS Shredded Wheat	Shredded wheat	13.2	1.2	2.0	8.0	82.0	1330/318
UNCLE TOBYS Wheeties	Wheat-based	10.1	1.4	2.5	340.0	69.8	1390/332
Recommended (9% or less fat, 19% or less sugar, 7% or more fibre, 600 mg/100 g or less sodium)
GOODNESS Tropical Toasted Muesli	Toasted muesli	7.4	8.7	15.4	9.0	51.5	1569/375
KELLOGG’S Just Right	Combination	9.3	1.4	18.2	295.0	65.0	1534/381
KELLOGG’S Mini-Wheats Apricot	Shredded wheat	11.9	1.1	16.4	26.0	66.5	1503/358
KELLOGG’S Sustain	Sports	7.5	2.9	15.0	112.0	67.8	1607/399
LOWAN Australian Rolled Oats (A)	Rolled oats	11.5	8.7	1.2	6.3	65.3	1624/388
MORNING SUN Natural Apricot and Almond Muesli	Natural muesli	17.6	8.3	17.8	32.0	51.4	1420/339
THE OLD GRAIN MILL Australian Gold Classic Muesli	Natural muesli	12.2	8.7	19.3	46.0	46.7	1610/385
THE OLD GRAIN MILL Australian Gold Natural Muesli	Natural muesli	11.9	6.3	17.5	66.0	56.5	1548/370
SANITARIUM Bran Bix	Biscuit	22.0	4.4	5.7	410.0	45.0	1180/280
SANITARIUM Crunchy Bix	Biscuit	7.7	5.5	13.0	320.0	58.7	16/380
SANITARIUM Natural Muesli	Natural muesli	7.1	5.1	18.6	124.0	44.0	14...

Table of contents

• Cover
• Title Page
• Copyright Page
• Contents
• Figures and tables
• Acknowledgments
• 1 Evaluation fundamentals
• 2 The nature of interventions: What we evaluate
• 3 Focusing evaluative enquiry: Evaluation forms and approaches
• 4 Negotiation and evaluation planning
• 5 From evaluation questions to evaluation findings
• 6 From evaluation findings to utilisation
• 7 Managing evaluation
• 8 Codes of behaviour for evaluators
• 9 Proactive evaluation
• 10 Clarificative evaluation
• 11 Interactive evaluation
• 12 Monitoring evaluation
• 13 Impact evaluation
• Index