Complete Training Evaluation

The Comprehensive Guide to Measuring Return on Investment

Richard Griffin

• Publisher: Kogan Page
• Year: 2014
• Print ISBN:
  9780749471002
• Edition: 1
• eBook ISBN:
  9780749471019
• Topic: Business
• Subtopic: Human Resource Management
• Index: Business

While substantial advances have been made in the L&D profession over the last decade, evaluation remains by far the weakest part of the L&D cycle. Most organisations wish to evaluate the impact of their investment in training but few do it well, and the lack of effective methods is one of the key barriers.

Complete Training Evaluation addresses these issues by providing practitioner friendly but academically robust information and guidance on how to evaluate all forms of learning and development. It draws on the author's own multidisciplinary research along with his practical experience of working with private and public sector organisations carrying out evaluation. The book provides practitioners with accessible 'how-to' knowledge and tools to undertake evaluations of both formal and informal learning.

Full of case studies and practical examples of application of methods and insights, Complete Training Evaluation equips practitioners with a range of approaches that can be used depending on the training programme, capacity and capability. An online supporting resource includes a bank of evaluation questions grouped around particular aspects of training.

Part One

THE STATE OF TRAINING EVALUATION

01

Introduction

Let’s start with what’s the matter with training evaluation. The vast majority of training evaluation that takes place focuses on learning outcomes rather than performance ones. Few organizations evaluate beyond level 1 or 2 of Kirkpatrick’s (1959) model – measuring reactions to training and learning acquisition (Brown, 2005). It is assumed, incorrectly, that if you got a positive reaction to training, say via a satisfaction survey, or you can show, perhaps through a test, that people have learnt something, then that training was effective. Counter-intuitively the opposite might be true, for reasons discussed later in this book. Focusing on reactions is a very narrow view of training – a bit like judging a cake by the quality of the ingredients before they go in the bowl and oven. While the ingredients might tell us something about how good the cake could be, the proof will only come later.

REFLECTION 1.1: WHAT EVALUATION ACTIVITY DOES YOUR ORGANIZATION CARRY OUT?

Think about the resources invested in workforce education and training in your organization: money, materials and time. How much of this activity is evaluated and what is the nature of the evaluation carried out? If it were necessary, for example, to review the relative effectiveness of different programmes would the information be available to allow decision-making? If your organization’s senior management asked for evidence of the return on training would you be able to show them? Have you ever ended a programme because of the results of evaluation?

In response to global competition and technological change, the last three decades have seen substantial and enduring changes in the organization and nature of work, as we will discuss in Chapter 3. One consequence of this has been a focus on the importance of organizational resources and capabilities, including employee learning and development, as a means of securing competitive advantage. However, while the strategic importance of training is seen not just by companies but also by governments across the world as a key driver of competitiveness (and much more), few organizations actually evaluate the impact of their investment in formal workforce learning. Even fewer consider the effect of informal learning. In Chapter 16 we will Consider the effect that fostering a positive learning culture can have on performance. You might be surprised at quite how important it is.

It is not though just the way people work that has undergone a fundamental transformation in the last few decades; so has the way we learn at work. To take one example: around 40 per cent of learning is now delivered via virtual learning environments (Chapter 14 looks at evaluating e-learning).

A WORD ABOUT TERMS AND REFERENCES

Terms such as ‘education’, ‘training’, ‘workplace learning’ and ‘development’ have distinct, although sometimes contested, meanings in academic literature. There are also many ways that different types of workplace learning can be classified such as formal/informal, general/specific, deliberate/unintentional or expansive/restrictive (for some reason they always come in pairs). In this book terms such as ‘training’ and ‘workplace learning’ are used for any event, experience or activity associated with the workplace that seeks to achieve a change in an employee’s knowledge, technical or soft skills and/or attitudes regardless of duration, location, trigger or whether it results in a formal qualification or not. I frequently interchange ‘learning’, ‘education’ and ‘training’.

This is not an academic book. It is aimed firmly at practitioners. The ideas and insights in it are however based on research, as well as on my own experiences evaluating training. I have tried not to clutter the text with too many references but have included them when particularly important or controversial points are made or of course when I am summarizing the work of others. References and suggestions for further reading are included at the end of the book.

The core argument of this book is that current evaluation practices – probably the ones that you are currently using – are based on an outdated concept of workplace learning, one more suited to the era of scientific management than the reality of 21st-century work. As the nature of work has changed so has workplace learning. Training no longer means employees attending a standalone course in order to acquire learning to perform frequently standardized and physical tasks, often under supervision. Workplace learning now addresses notions such as ‘emotions’, ‘resilience’, ‘change’, ‘attitudes’, and ‘empathy’ through a wide range of learning approaches, from classroom teaching, to e-learning, the use of social media and the encouragement of knowledge exchange. The location where learning is possible has also changed. We can learn as much on the move as in a classroom. The focus is as much on the learning organization as organizational learning. The problem is that the approaches and practices used to evaluate the rapidly changing field of learning have not kept up with these changes.

What I want to argue is that we need ways of evaluating training that are ‘scientifically robust but practitioner friendly’. What does this mean? Well, ‘scientifically robust’ methods of evaluation I think:

• are based on research;

• allow outcomes that can be attributed to training;

• provide results that are reliable and consistent;

• produce results able to support decision-making;

• adopt a ‘whole systems’ approach;

• address all ethical issues.

While ‘practitioner-friendly’ approaches to evaluation are:

• operational within available financial and time resources;

• results-orientated and meet the needs of stakeholders;

• technically feasible;

• credible;

• simple to operate and understand;

• flexible.

Moving ahead: complete evaluation

Most organizations want to evaluate the impact of their investments in training but very few actually do – actually probably less than 5 per cent. There are a number of reasons for this, as we will see in Chapter 5; three of these I think are particularly important:

1 Training evaluation is believed to be technically challenging.

2 Training evaluation is seen to be resource-intensive.

3 Current methods appear not to work because they are based on an outdated conception of workplace learning.

Plenty of people have argued that training evaluation is just too hard. Berge (2008:393) has described it as ‘complex, difficult’, Gibb (2002:107) as ‘difficult and challenging’, Reid and Barrington (1999:342) as ‘one of the more difficult of training officers’ tasks’ and Sims (1993:591) as ‘difficult, tedious and time-consuming’. It is perhaps not surprising that no less an august body than the ASTD (American Society of Training and Development) has said that training professionals have a ‘love/hate’ relationship with evaluation: they know it’s necessary but they struggle to do it. In fact our growing understanding of workplace learning means that there has never been a better time to think about evaluation.

This is where this book comes in. It draws not only on the latest research but also on the author’s practical experience of carrying out evaluations, often with limited resources. It aims to provide practitioners with accessible ‘how-to’ knowledge and tools to undertake evaluations including impact evaluation. It challenges the idea that training evaluation is too complex and difficult. I do not think it is tedious either!

In the following pages I argue that complete training evaluation practice needs to embrace the complex reality of the workplace in order to move from its current restrictive approach – those largely based on assessing the acquisition of learning and narrow reactions to training – to expansive approaches that can address issues such as: what is the nature of the organization’s learning culture; are trainees motivated to learn; what barriers are there to the transfer of learning; what unintended consequences have arisen from learning; what impact do managers have on the effectiveness of training and what is the long-term effect of training? This is the scientifically robust bit.

This book sets out a range of approaches that can be adapted depending on the learning, the audience for the evaluation and capacity and capability. It aims to be comprehensive in setting out the methods that you can use to evaluate. It assumes no prior knowledge of evaluation. Fundamentally it aims to be practical. For brevity’s sake, I describe this approach as Complete Evaluation.

Summary of contents

This book is organized into four sections:

• Part one – The state of training evaluation is an overview of the evaluation and workplace learning, comprising a discussion of what evaluation is and what it can address (Chapter 2) and the changing nature of work and workplace learning and its implications for evaluation in the 21st century. The evidence of the positive effect training can have on performance is also discussed (Chapter 3). The fina...

Table of contents

• Complete Training Evaluation
• Complete Training Evaluation
• Contents
• List of Figures
• List of Tables
• Preface
• Part One THE STATE OF TRAINING EVALUATION
• 01 Introduction
• 02 The power of training evaluation
• 03 The workplace, learning and performance
• 04 Where are we now?
• Part Two GETTING READY
• 05 Barriers to evaluation and how to overcome them
• 06 Planning an evaluation
• 07 Surveys, peer review evaluation, observation and other ways to evaluate
• 08 Stakeholder evaluation
• Part Three EVALUATION
• 09 Meaningful reaction measures: Moving beyond happy sheets
• 10 Rating scales: Measuring the effectiveness of training
• 11 ‘Can do, will do’ – the importance of self-efficacy 1
• 12 Training transfer and retention: Making training stick
• 13 ‘What difference did the training make to you?’
• 14 Evaluating e-learning
• 15 Evaluating learning outcomes
• 16 Evaluating informal learning
• 17 Costs
• Part Four BRINGING IT TOGETHER
• 18 Presenting evaluation results
• 19 Complete evaluation checklist
• 20 Bringing it all together
• 21 Boosting performance through complete evaluation
• Annex 1
• References
• Index
• Copyright Page