Educational Technology Program and Project Evaluation

J. Michael Spector, Allan H.K. Yuen

• Verlag: Routledge
• Jahr: 2016
• ISBN:
  9781317530060
• Auflage: 1
• Thema: Didattica
• Thema: Didattica generale

Educational Technology Program and Project Evaluation is a unique, comprehensive guide to the formative and summative evaluation of programs, projects, products, practices and policies involving educational technology. Written for both beginning and experienced evaluators, the book utilizes an integrative, systems-based approach; its practical emphasis on logic models and theories of change will help readers navigate their own evaluation processes to improve interventions and conduct meaningful educational research. Key features include:

• 
  
  

• evidence-based guidelines for constructing and conducting evaluations

• 
  
  

• practical exercises to support the development of knowledge, skills, and program evaluation portfolios

• 
  
  

• a variety of interdisciplinary case studies

• 
  
  

• references and links to pertinent research and resources

Using the TELL, ASK, SHOW, DO model first introduced in this series, Educational Technology Program and Project Evaluation provides comprehensive coverage of the concepts, goals, design, implementation, and critical questions imperative to successful technology-enhanced evaluation.

part one
INTRODUCTION AND OVERVIEW

one

The Nature of Learning, Performance and Instruction

Hence the central problem of an education based upon experience is to select the kind of present experiences that live fruitfully and creatively in subsequent experiences.

(from John Dewey’s Experience and Education)

As reported elsewhere (Spector, 2012, 2016), M. David Merrill was fond of saying that people learn what they do. There is a natural link between doing and learning, as implied by the opening quotation by Dewey (1938). Many instructional design practitioners and researchers realize that natural connection and have argued for instructional approaches that integrate human performance on realistic tasks. These approaches include anchored instruction (CTGV, 1990), authentic learning (Blumenfeld et al., 1991), cognitive apprenticeship (Collins, Brown, & Newman, 1987), and situated learning (Lave & Wenger, 1990), among others.

Instructional designers and educational researchers generally accept the premise that people learn what they do. This premise resonates well with popular beliefs, such as practice makes perfect, as well as with neural science and how repetition can result in neural reinforcement and explain how some procedural activities become automatic after many repetitions by a person. Accepting that premise has strong implications for both design and evaluation. For design, it means that it is important to include in an instructional sequence the actions and activities that are pertinent to the targeted competencies and skills (e.g., authentic tasks). For evaluation, the implications are that these two things are important: (a) the degree to which the selected actions and activities are appropriately linked to the targeted competencies and skills, and then (b) the degree to which learners are able to develop those competencies and skills.

As a consequence, designers and evaluators have a strong interest in needs assessment, problem definition and requirements analysis. Both designers and evaluators generally start with questions that address what people now know and are able to do, and then what knowledge, skills and understanding they should develop with appropriate education, training and experience. As typically happens, technology is involved in the education, training and experience that is intended to take an individual or group of individuals from one state of understanding or level of performance to a higher state or level.

Design and development play a critical role in the process of promoting knowledge and understanding, developing skills and improving performance. Instructors, tutors and coaches also play critical and well-acknowledged roles in that process, as do technical specialists and other support personnel. Less well acknowledged in many cases, is the vital role played by evaluators in the processes associated with planning and implementing solutions aimed at improving knowledge, understanding and performance. This book is about the vital roles that evaluators play and the value they add to progress in using technology effectively and efficiently in support of learning, instruction and performance.

We begin with definitions and reminders presented in the Foundations volume in this series. We then put the key concepts of learning, performance and instruction together in a context involving educational technology and conclude with a representative challenge.

Learning

Defining learning

The classical view of a definition involves the essence of the thing being defined—that which makes it what it is and not something else. One might be tempted to ask about the essence of learning. What is it that counts as evidence of learning as opposed to something else, such as evidence of enthusiasm or evidence of short-term recall or evidence of appreciation for a teacher?

The definition of learning introduced in the Foundations volume (Spector, 2012, 2016) addresses a process view of learning, as opposed to an event or product view—namely, learning is a process that is aimed at stable and persistent changes in what an individual or a group of individuals know and can do. Indicators of success include relevant (in terms of contributing to intended outcomes) changes in abilities, attitudes, beliefs, knowledge, mental models, motivation, skills, and so on.

There are several critical aspects of this definition that have implications for evaluation. The first is the fundamental notion of change. Without reliable indicators of relevant change, there is no evidence of success. An early task for an evaluator (or evaluation team) is to identify the targeted changes and help those involved develop credible ways to detect, document and report changes. Another early task of an evaluator is to be sure that the indicated changes to be monitored and reported are clearly linked to the intended outcomes of the effort. These early tasks are part of an evaluator’s responsibility to help an effort achieve intended outcomes. In too many cases, an evaluator is not involved until after the indicators have been determined. What can happen in such cases is that the design and development team might focus on the technical or academic aspects of the effort and fail to link what is being done to the goals and objectives identified by those sponsoring the effort.

As mentioned, the concepts of change and relevant indicators are an essential part of a process view of learning. We recognize that some prefer an events perspective that might be associated with moments of discovery reported by individuals. The problem with an events view, however, is that it overlooks precursor activities that lead up to a discovery moment. In Gagné’s (1985) classic work entitled The Conditions of Learning and Theory of Instruction, he discusses five types of things to be learned (verbal information, motor skills, attitudes, intellectual skills and cognitive strategies). Gagné also recognized and emphasized the processes involved in what he called association learning, and these were later expanded to include specific cognitive processes occurring in a learner engaged in an instructional sequence. In addition, Gagné identified what he called nine events of instruction (gaining attention, identifying the learning objective, stimulating recall of prior learning, presenting what is to be learned, providing learning guidance, eliciting performance, providing feedback, assessing performance, and enhancing retention and transfer). Gagné argued that these instructional events typically occur in successful learning activities. He did not refer to those instructional events as learning events, nor did he say that they were discrete. Gagné believed that those instructional events often occurred in combinations and multiple times within an instructional sequence (see Spector, Polson & Muraida, 1993). The nine events of instruction were never intended to be a step-by-step recipe for the design of learning, as too many have misunderstood.

A second problem with an events view of learning is that it overlooks the fact that not all those involved will experience such discovery moments during instruction. Examining the entire instructional sequence and the progress of individuals over time and through instruction and experience is relevant to the intended outcomes of educational technology efforts aimed at improving learning.

A products view of an educational technology learning effort also suffers limitations. Learning as an educational product objectifies an effort in terms of things such as certificates, degrees, grades, graduation rates, job placements, and so on. The problem inherent in such a view is that it rarely links an instructional intervention or learning approach directly to the product. Unless there has been a controlled experimental study, it is quite difficult to link the product outcome to what preceded that outcome, and such studies are rare and difficult to conduct in actual educational settings. Moreover, the notion of change is often overlooked; when included in a product account of learning, the change typically involves a comparison with a baseline study of a group prior to the instructional intervention or new learning approach. Again, this approach can only indicate a correlation and often fails to link specific learning and instructional activities to the outcomes reported. However, efforts to design, develop and deploy educational technology products (e.g., learning applications, devices and management systems) are among the things that can and should be evaluated.

For these reasons, we proceed with a process view of learning, as such a view allows for research and evaluation, and it is especially well suited for assessing individual progress of learning over time, which is an important component of many educational technology evaluations.

Types of learning

An important distinction is that between intentional and non-intentional learning. The kind of learning typically involved in educational technology efforts is intentional, in that the learning is goal-directed, planned and purposeful. This is essential for evaluation, since a critical part of an evaluation effort of an educational technology effort aimed at improving learning will be to determine and report the extent to which those goals were met for a group of individuals. Keep in mind that in this textbook series the word ‘assessment’ is used to refer to individuals or groups (i.e., as assessing human knowledge and performance) whereas the word ‘evaluation’ is used to refer to projects, programs, products, practices and policies. Of course what is evaluated often involves human learning, so assessments will become part of an evaluation, but the focus of the evaluation is on the larger context as opposed to being focused just on individuals.

Much learning is of course non-intentional in the sense that it is not planned or purposeful and occurs incidentally or accidentally in the course of other activities. Some educators who focus on the significance of discovery moments also focus on the significance of non-intentional learning in the course of an individual’s life. An example of such learning is discussed in the Foundations volume (Spector, 2012, 2016) involving Leo Tolstoy on a visit to Paris in the middle of the nineteenth century. Tolstoy happens upon a public execution by accident, and witnesses what he considered a barbaric and brutal execution. Tolstoy did not plan to witness the execution, nor did he intend to learn anything about French civilization. However, he records in his journal (published after his death as Confessions) that the event changed his life. He gave away his fortune; he quit writing novels aimed at fame and fortune; he took up teaching in a rural school in Russia. Those changes were seen by others and they persisted long after his witnessing the execution. In short, Tolstoy learned something that day that persisted years afterwards from that unplanned event.

However, to imagine that one can plan such life-changing events as part ...

Inhaltsverzeichnis

• Cover Page
• Half Title page
• Series
• Title Page
• Copyright Page
• Dedication
• Contents
• Preface
• Acknowledgments
• Part I Introduction and Overview
• 1 The Nature of Learning, Performance and Instruction
• 2 Projects, Programs, Products, Practice and Policy
• 3 Evaluation Goals and Scope
• 4 Types of Evaluations
• 5 Major Evaluation Tasks
• Part II An Evaluation Framework
• 6 Needs Assessment
• 7 Symptoms and Underlying Causes
• 8 A Theory of Change
• 9 Outcome Measures and Indicators
• 10 Mediators, Moderators and Contextual Factors
• Part III Logic Models and Implementations
• 11 Interventions and Implementation Plans
• 12 Developing a Logic Model
• 13 Identifying Indicators and Measurements
• 14 Measurement Instrumentation and Protocols
• 15 Fidelity of Implementation Plans
• Part IV Translating Plans into Actions
• 16 Milestones for Effective Technology Integration
• 17 Ongoing Formative Evaluations
• 18 Dynamic Improvements to Ensure Success
• 19 Collecting and Analyzing Data
• 20 Informing and Improving Policy Implementation
• 21 Documenting, Reporting and Recommending
• Part V Additional Resources
• Appendix A: A Sample Evaluation Plan
• Appendix B: Professional Evaluation Associations
• Appendix C: Peer-Reviewed Evaluation Journals
• Glossary of Terms
• Consolidated References
• Index