Learning Analytics

Using Talent Data to Improve Business Outcomes

Cristina Hall, John R Mattox, Peggy Parskey

• Publisher: Kogan Page
• Year: 2020
• ISBN:
  9781789663013
• Edition: 2
• Topic: Business
• Subtopic: Human Resource Management
• Index: Business

Effective evaluation and measurement of learning and development initiatives is critical to maximise the impact of training, identify gaps for improvement and ensure that efforts are aligned to the business' needs. Learning Analytics outlines how analytical approaches can respond to these challenges, the types and benefits of technological solutions and how to ask the right questions of organizational data in order to build a learning organization that boosts performance and competitive advantage.
Drawing upon case studies from organizations who have applied such approaches such as The Gap, Hilton Worldwide University and Seagate Technology, Learning Analytics will enable those involved in learning and development to make the business case for their activities and deliver an evidence-based service to their organizations. Alongside updated chapters on learning technology tools and moving beyond learning analytics to talent management analytics, this second edition also features new content on measuring informal learning, increasing data literacy, and framing L&D's contributions through a portfolio evaluation approach.

Part One

The L&D value gap and how to close it

Technology, data and the internet are accomplishing truly amazing things. Software can analyse an X-ray and diagnose early-stage breast cancer better than oncologists and radiologists; self-driving cars will soon be a commercially viable product; retailers provide just-in-time electronic reminders on our smartphones to buy toilet paper and other necessary products. Well, the reminders may not be truly amazing, but they are certainly technology-enabled, useful for the shopper and profitable for the retailer – capabilities that provide value to consumers and providers.

Data is at the heart of these innovations. Cancer screening software is trained by scanning thousands of mammograms. Self-driving cars constantly scan the environment and process data about the road, control features, other cars, pedestrians and road hazards. I’m We’re not sure how the retailers know when we’re out of paper.

In the business world, Human Resources (HR) is not usually the first department to benefit from valuable technology. As part of HR, Learning and Development (L&D) is also late to the game. With a few exceptions, it is nearly impossible to push a button to answer the question ‘Does training really work?’, whereas Finance can answer whether the business is performing in line with forecasts and Operations can report whether products are being produced within tolerances (quality) and meeting target numbers (quantity).

Despite this challenge, L&D has been evolving, and the learning industry has benefited from some valuable technologies like learning management systems, e-learning platforms and video tools to support virtual instruction. Data is embedded within many of these systems. Learning management systems (even learning record stores) store courses and track which programmes have been completed by which learners. Activity information can be summarized to show learning hours per person and the amount of e-learning consumed. This is important information, but it is only one piece of a bigger data puzzle. Business leaders want L&D to provide leading indicators that show the effectiveness of training. This information typically comes from post-course evaluations (eg Level 1s). Additionally, business leaders want to know if training is influencing business measures (eg sales, revenue, customer satisfaction, quality, productivity, etc). Like learning activity, this information is stored in business systems.

Learning analytics is gaining traction as a viable solution in this data-rich, information-poor environment. Why? Because business leaders need to know if their investment in learning is working. They want to know if they should double-down or seek other interventions to meet business goals. Learning leaders also have information needs. In addition to understanding the effectiveness of training, they need to know which courses need improvement so they can allocate resources to revise and improve them. Minimally, they need effectiveness information (Level 1 – ROI) and that is covered in Part One of this book. Ideally, they will also capture efficiency, effectiveness and outcomes data to create a complete picture of the training’s impact (covered in Part Three).

Measurement is a journey with four specific stages: 1) know where you are (eg define your current state); 2) have a destination and timeframe in mind; 3) implement a plan to navigate from here to there; and 4) adjust the plan depending on the circumstances. The first Part of this book covers Stage 1, and it will help you figure out where your organization is on the map. It will help you answer the questions: Do you have the right leadership and governance in place? Are your tools and processes standardized? Are you leveraging technology to make the processes scalable?

How well (or poorly) you answer these questions defines how large the gap is between you and a successful measurement strategy.

Practically speaking, Chapter 1 focuses on defining the value gap and why we need learning analytics. The second stage of a journey is to have a destination in mind, and Chapters 2 and 3 will help create a vision for a possible destination for your organization. Chapter 2 defines learning analytics and how it can be used. Chapter 3 introduces the Talent Development Value Framework and how it provides a programmatic playbook for many of the challenges that L&D organizations face when implementing sustainable measurement practices.

By the end of Part One it should be clear that measurement is a journey, and there are models to help you map and navigate the landscape. Additionally, learning analytics, along with some sound business principles, is a useful guide for informing stakeholders, building trust and sure-footedly closing the information gap between L&D and the business.

01

The rise of learning analytics

In 2019 the world wide web celebrated its 30th birthday.1 For 30 years the web has connected the world, shared information and changed economies and cultures. Where were you in 1989? What were you doing?

The things you are doing today such as surfing the net on your phone, booking a hotel via Airbnb, or getting an Uber from the airport, are not the things you were doing then. Technology has changed our world. It continues to leap forward in ways that most of us cannot foresee. If the internet is the highway that connects computers around the world and the world wide web is our on-ramp, then we (our clicking actions) are the cars driving on it at the speed of information.

Welcome to the Digital Age, also known as the Data Age or the Information Age. Many innovations brought us to this world we know today like the vacuum tube, the transistor, the silicon wafer, programming languages, algorithms and plasma displays among many other things. This new age shifted away from the Industrial Age – a time last century when manufacturing drove the economy and culture. While manufacturing has declined, it has not disappeared. In fact, companies that manufacture hardware and software are thriving in the Information Age. The shift is certainly economic, but the average person does not feel it like an economist studying macro-trends. The shift is felt culturally in the way we work and socialize and recreate.

Michelle Evans provides a convincing argument that information is the new currency and that information is vast.2 Simple clicks on the internet describe what people are reading, watching, communicating, creating, posting, buying, selling and in general doing. Those actions have value to marketers who want access to a community of buyers. Those actions are valuable to a guru who wants to build a following, or a sales professional who wants to sell a new product to someone who bought a gateway product.

Knowledge itself has value, but it has its limits. Experts hoard knowledge and have mastered a domain of content and skills. In many ways, knowledge is less valuable now because of Google, which can provide facts on almost any topic. Skills, wisdom and insight on the other hand are still valuable. The value of knowledge is transient and may be replaced by someone or something that facilitates connections. Mavens are valuable because they have deep knowledge of a domain. Connectors are valuable, not because they know the domain, but because they know many people who know many domains.3 Research from Gartner has found that the most effective managers are not the ones who micro-manage; they are not mavens. They are the ones who connect their direct reports to others who can help them do their jobs.4

Amid the technological changes flooding the market, a watershed moment occurred when Harvard Business Review dedicated an entire edition to big data in October 2013.5 Through various articles, authors defined big data, shared how it was gathered and described how it would create competitive advantage for organizations. During the ensuing 18–24 months, technology vendors created new big data products and services to capture the market. Time eventually highlighted that very few companies actually played in the big data space. The real players included Walmart and its massive volumes of daily global in-store transactions; telecom companies that host calls and internet traffic; Amazon’s online transactions; Netflix, Hulu, Apple Music and other streaming services that transfer content across the internet; Google responding to millions of enquiries per second; Facebook and LinkedIn sharing information across personal and professional networks; Salesforce.com hosting a platform for tracking and reporting sales leads from tens of thousands of companies and untold users around the world. The big data focus brought attention to the need for analytics in general, which eventually rolled to HR analytics and learning analytics.

While many companies realized that they did not have truly big data, two things remained important – essential – for decision-making with data: security and quality. In the past few years, we have seen massive data breaches including Yahoo (3 billion people in 2013), Marriott/Starwood Hotels (500 million in 2018) and Equifax (146 million in 2017).6 When customer information is lost to breaches, trust is lost and business suffers as customers leave for competitors, or worse, file lawsuits.

Quality is essential for mining the data for insights. If the inputs are wrong, if information is incomplete or if the sources are spurious, the analysis conducted and the conclusions drawn could be wildly off-base or simply wrong. In such cases, big data is bad data and is no more valuable than fool’s gold.

Because business leaders understand that value is locked inside their data, they are pushing their business units to analyse what is available and provide insights. Marketing has changed substantially in the past decade. Marketers not so long ago would say tongue-in-cheek: ‘We know that 50 per cent of advertising works. We just don’t know which 50 per cent.’ Today, because of their ability to post content, track views, clicks, time on page, bounce rates and desired outcomes (eg online purchases), they are much better equipped to say: ‘We know which 5 per cent works, and we going to reinvest to drive more sales with our advertising dollars. Expect an uplift of 2–4 per cent this quarter.’

The term Internet of Things (IoT) is a l...