A Guide to Useful Evaluation of Language Programs

John McE. Davis, Todd H. McKay, John McE. Davis, Todd H. McKay

• Publisher: Georgetown University Press
• Year: 2018
• ISBN:
  9781626165786
• Topic: Languages & Linguistics
• Subtopic: Teaching Language Arts
• Index: Languages & Linguistics

Departments and language programs often are asked to evaluate the efficacy of their own programs and make curricular decisions on the basis of evidence. This guide, designed to help language educators meet the needs of program evaluation and assessment often requested by their institutions, provides step-by-step advice to help language educators conduct evaluation and assessment and to show how it can lead to meaningful programmatic decisions and change. With discussions about evaluation planning, advice for selecting data-collection tools, explanations for data analysis, examples based on actual evaluations, and more, this book provides everything you need to complete a successful language program evaluation that will give educators useful data on which to base curricular decisions. This short book is practical and timely and will find an audience in instructors of all languages and all levels.

1

An Introduction to Useful and Used Evaluation in Language Education

JOHN McE. DAVIS

THE GOAL OF THIS book is to help language educators conduct useful evaluation in language education programs. It comes at a time in language education when instructors are increasingly called upon to conduct evaluation themselves in their classrooms and institutions. Growing desires for educational accountability and quality assurance have led to increased evaluation and assessment requirements imposed by accreditors, school districts, and federal and state governments (to name a few). While contemporary evaluation in language education is still often conducted by peer reviewers and evaluation professionals (as has been the practice for many years), more and more commonly the responsibility for doing evaluation now falls to the language instructor, lead teacher, section chair, or curriculum committee. The primary aim of this book, then, is to help language educators meet the various contemporary demands for evaluation and assessment now commonplace in US language education, and to conduct evaluation in ways that lead to meaningful programmatic decisions and change.

In addition, this guide advocates for a particular view of evaluation as an inherently worthwhile mode of educational inquiry, rather than merely a process of judging educational quality. We further suggest that evaluation is most effective when its impetus is less a reaction to outside requirements and more something that emanates naturally and organically from within a program where instructors, administrators, and leadership take ownership of evaluation and engage in consistent, evidence-based decision-making. In the same way that time, effort, and resources are routinely dedicated to teaching, curriculum review, assessment, and professional development, so too should evaluation be an integral part of the daily business of teaching languages. To these ends, our aim is to impart foundational evaluation knowledge and skills that help create a “culture of evaluation” and enable stakeholders to better understand and improve language teaching and learning.

Evaluation Use and Usefulness

Conducting useful evaluation is a deceptively straightforward aim. Doing evaluation that results in concrete action, decision-making, and meaningful change is, in fact, a tremendous challenge. An important goal for this book, then, is to help readers plan and implement evaluation in a way that enhances the ultimate usefulness and use of evaluation in language programs.

This may seem like a banal and obvious point. Of course evaluation should be useful. Why make this distinction? Naturally, educators and other stakeholders need information to better understand and improve their programs, and evaluation is a process that helps them do this. By its very nature, evaluation is a useful activity. Why state the obvious?

In fact, research has shown, to the contrary, that evaluation is vulnerable to non-usefulness and lacking use of evaluation findings. Despite a considerable potential for shedding light on issues and problems in educational programs, evaluation does not always straightforwardly lead to this end. Indeed, readers may have experienced instances where evaluation activities failed to have intended or useful impacts. Perhaps recommendations from a program review failed to be implemented. Perhaps student and teacher evaluations are collected consistently but without any demonstrable use toward improvement or change. Perhaps requirements from the university or college administration to engage in assessment of student outcomes has created additional work and taken up valuable time but without any tangible benefits to student learning or teaching.

If these scenarios sound familiar, they are examples of a lesson long learned in evaluation research and practice—the efficacy and usefulness of an evaluation is not a given. Rather, actual use of evaluation findings is something evaluators must carefully plan for and nurture throughout an evaluation project (Norris 2016). This being the case, we advocate in this book for an evaluation approach that calls for an intentional and systematic focus on evaluation use and usefulness throughout evaluation planning and implementation. It is an approach strongly influenced by the ideas of Michael Quinn Patton and his “utilization-focused” evaluation model, and also elaborated by the main proponent of a use-focused evaluation approach in language education, John Norris (2006, 2008, 2016).

Patton (2008, 37) defines utilization-focused evaluation as follows:

Utilization-focused evaluation is evaluation done for and with specific intended primary users for specific, intended uses. Utilization-focused evaluation begins with the premise that evaluations should be judged by their utility and actual use; therefore, evaluators should facilitate the evaluation process and design any evaluation with careful consideration for how everything that is done, from beginning to end, will affect use. Use concerns how real people in the real world apply evaluation findings and experience the evaluation process. Therefore, the focus in utilization-focused evaluation is on intended use by intended users.

“Careful consideration for how everything . . . will affect use” calls for specific evaluation planning and implementation techniques. Adding to the traditionally technical aspects of evaluation such as data collection, analysis, or reporting, a use-focused evaluation approach integrates specific strategies into the evaluation process that help increase the likelihood that evaluation findings actually get used. To give a brief example of what this entails, in a use-focused evaluation approach, the evaluator does not conduct an evaluation assuming that program leadership will use the evaluation results and implement the evaluator’s findings. Instead, the evaluator identifies and engages specific users of the evaluation during evaluation planning and designs the evaluation carefully to meet those individuals’ needs. In addition, the evaluator does not assume or guess what decision-makers will do with the evaluation findings once the evaluation report has been submitted. Rather, a use-focused approach calls for the evaluator to sit down with decision-makers and list out in explicit terms specifically what they want to do with evaluation results. The evaluator then organizes the subsequent evaluation steps and phases in specific ways so that the project supplies decision-makers with the findings they need in order to do what they need to do in their programs.

These and other use-focused strategies are key elements of a use-focused evaluation approach. They are intended to help ensure that when the evaluation results are presented in the final report or presentation, specific intended users will actually use the information from the evaluation for specific intended uses. We propose that language education would profit from integrating an explicitly use-focused evaluation approach into current evaluation activities in language programs (along with Norris 2006, 2008, 2016). A methodological focus on usefulness is now an established part of evaluation outside of language education and an essential aspect of professional evaluation practice, as formalized in the JCSEE¹ Program Evaluation Standards (Yarbrough et al. 2011). The JCSEE evaluation standards lay out expectations of high-quality evaluation for professional evaluators, one of which emphasizes “utility” and ensuring that the evaluation is useful for—and used by—program stakeholders. Furthermore, the factors that lead to evaluation usefulness have been a focal concern of program evaluation research, a key finding of which is that intentionally planning for evaluation usefulness leads to evaluation use. Lastly, a number of recent evaluation case studies in language programs have showcased use-focused techniques and productively useful program development outcomes as a result of adopting this approach (see Norris and Davis 2015; Norris et al. 2009). A use-focused approach to evaluation, then, has much to recommend and offers considerable potential for language educators to better understand and improve language instruction.

Evaluation

We define evaluation in this book using a definition from Norris (2006, 579), which highlights key aspects of the evaluation process we aim to emphasize:

Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value . . .

Following Norris, evaluation in our view is more than judging program value or effectiveness, though this is certainly an important evaluation purpose, as Norris has captured here. Rather, we envision evaluation more generally as a mode of inquiry in which stakeholders within language programs collect, learn from, and use evidence toward programmatic actions and decisions. In addition, we echo Norris’ (2006) observation that the link between gathering information and using that information is a tenuous one. It is precisely this link in the sequence of events of an evaluation project that this guide attempts to strengthen, and we offer the use-focused evaluation approach as a strategy to do so.

Intended Audience

This volume is intended to be a practical manual that language educators and administrators can use to implement evaluation in ways that enhance methodological quality and lead to useful actions and decisions. It is not aimed at professional evaluators, academics, or evaluation consultants. Rather, the intended audience is readers with little or no evaluation experience who need straightforward, practical guidance on how to plan an evaluation, collect useful information on a programmatic issue of interest, and enable stakeholders to make programmatic decisions on the basis of evidence. To that end, this book is meant to be a step-by-step guide for evaluation novices within language educational institutions who are responsible for evaluation activities. Again, such responsibilities now commonly and increasingly issue from external mandates and create a burden of labor for language professionals who typically lack the resources and expertise to meet those demands successfully.

Organization of the Volume

The book is organized according to the chronological steps for planning an evaluation project, laying out the key considerations and strategies needed to plan and conduct evaluation in a language education program.

Chapter 2 (Best Practices for Evaluation Success) explains how useful evaluation relies heavily on specific program conditions, infrastructures, and personnel dimensions, each of which should (optimally) be present in programs before evaluation begins. The chapter emphasizes that evaluation can succeed or fail depending on the conduciveness of the programmatic context to high-quality evaluation activities.

Chapter 3 (Planning for Useful Evaluation: Users, Uses, Questions) describes the essential aspects of use-focused evaluation planning. Specifically, the chapter emphasizes two cornerstone planning components of Patton’s framework: the identification of (1) evaluation users and (2) evaluation uses. In addition, the chapter describes how to develop evaluation questions so that an evaluation project proceeds systematically with clear aims.

Chapter 4 (Identifying Indicators) provides advice on identifying and developing “indicators” of language program functionality—that is, the educational processes or phenomena within the program that will be captured or measured during the evaluation project. Chapter 4 stresses the importance of identifying program indicators before selecting data-collection methods and discusses the threats to evaluation usefulness when failing to think first about the types of information needed to answer evaluation questions.

Chapter 5 (Selecting Methods, Collecting Data) introduces the range of evaluation data-collection methods available to language educators. Chapter 5 also discusses issues that impact the usefulness of evaluation information and proposes strategies for choosing methods that help ensure the information will be what users and stakeholders want and need.

Chapters 6, 7, and 8 provide more detailed methodological advice on the three most common tools or strategies for collecting evaluation information: focus groups, interviews, and questionnaires (with the exception of language assessment). Each chapter discusses (1) the evaluation-related purpose of the method; (2) the unique advantages (and pitfalls) of a particular method; (3) the main steps in planning, designing, and administering the data-collection method; and (4) best practices for implementing each technique.

Chapter 9 (Analyzing Evaluation Data) provides introductory advice on analyzing the likely data coming from questionnaires and focus groups. In addition, key considerations of data trustworthiness are discussed as they pertain to evaluation projects.

Chapter 10 (Key Points to Remember for Useful Evaluation) summarizes the most important points from the guide, providing a list of priority evaluation best practices.

Chapter 11 (Example Evaluation Plan) provides an example evaluation plan so that readers can understand what an evaluation plan looks like and how the various use-focused evaluation components come together, including a description of the program background, stakeholder engagement plan, project users, uses, evaluation questions, indicators, data-collection tools, and project timeline.

Evaluation Scenarios

To help illustrate the concepts presented in this guide, three fictional evaluation scenarios are provided and elaborated in subsequent chapters as example cases of how to plan and implement use-focused evaluation techniques. The scenarios are developed throughout the book to help readers better understand evaluation concepts from a given chapter. For example, at the end of chapter 3, fictional evaluation users, uses, and questions are provided for each scenario so that readers can see what these specific elements of evaluation planning might look like. The three scenarios are (1) a community college language lab, (2) a language program needs analysis for adult learners, and (3) a high school Chinese telecollaboration program. Each scenario is described in detail below.

Scenario 1: Community College Language Lab

A group of language instructors at a c...

Table of contents

• Cover
• Half title
• Title
• Copyright
• Contents
• List of Illustrations
• Preface
• Chapter 1. An Introduction to Useful and Used Evaluation in Language Education
• Chapter 2. Best Practices for Language Program Evaluation Success
• Chapter 3. Planning for Useful Evaluation: Users, Uses, Questions
• Chapter 4. Identifying Indicators for Evaluation Data Collection
• Chapter 5. Selecting Methods and Collecting Data for Evaluation
• Chapter 6. Conducting Focus Groups for Evaluation
• Chapter 7. Conducting Evaluation Interviews
• Chapter 8. Questionnaires for Evaluation
• Chapter 9. Analyzing Evaluation Data
• Chapter 10. Key Points to Remember for Useful Evaluation
• Chapter 11. Example Evaluation Plan
• References
• List of Contributors