Learning Analytics Goes to School

A Collaborative Approach to Improving Education

Andrew Krumm, Barbara Means, Marie Bienkowski

• Publisher: Routledge
• Year: 2018
• ISBN:
  9781317307860
• Edition: 1
• Topic: Education
• Subtopic: Education General

Learning Analytics Goes to School presents a framework for engaging in education research and improving education practice through the use of newly available data sources and analytical approaches. The application of data-intensive research techniques to understanding and improving learning environments has been growing at a rapid pace. In this book, three leading researchers convey lessons from their own experiences—and the current state of the art in educational data mining and learning analytics more generally—by providing an explicit set of tools and processes for engaging in collaborative data-intensive improvement.

Chapter 1
Introduction

The daily activities of schools and universities—from taking attendance to assessing students—can leave a trail of data that, under the right conditions, can be used to explore teaching and learning like never before. Until recently, though, researchers had to choose between collecting rich data on a small number of individuals or amassing less rich data for larger numbers of individuals. And in both cases, collecting data on the same individuals over time required significant costs and complexities. For activities that take place in digital learning environments like games, learning management systems, and intelligent tutoring systems, surprisingly rich data can be collected on dizzyingly large numbers of learners over time. While opportunities to collect and analyze new forms of data increase every day, critical challenges need to be overcome in order to use these data to improve teaching and learning.

Along with new forms of data, such as system log data (i.e., records of users’ interactions with a digital learning environment), familiar forms like text, audio, and video are becoming increasingly open to in-depth analysis—at scale—through machine learning and artificial intelligence. These newly found and newly analyzable data are often described as “big data” both inside and outside of education. Over the past decade, analyzing educational big data has largely occurred in research labs at universities, technology companies, and non-profit research institutes, and this basic research, with few exceptions, has yet to diffuse widely or to fundamentally change teaching and learning (Baker, 2016; Martin & Sherin, 2013). Where there have been successes, such as with the ASSISTments platform (Roschelle, Feng, Murphy, & Mason, 2016) and in examples described later on in this chapter, new forms of data and new analytical techniques have been grounded in problems facing practitioners and used to develop and assess potential changes related to those problems.

As some have argued, improving teaching and learning at scale will require new ways of organizing the work of educational research (Bryk, Gomez, Grunow, & LeMahieu, 2015). Starting around the same time as educational data mining and learning analytics—some of the most recognizable fields in what may be termed data-intensive research—an approach to conducting educational research referred to as research-practice partnerships was taking shape (Coburn, Penuel, & Geil, 2013). While the idea of forming partnerships is not new, frustrations with the status quo, a critical mass of success stories, and new funding opportunities have coalesced into an overarching approach where researchers work on pressing problems of practice in an iterative and collaborative fashion with practitioners (Penuel & Gallagher, 2017). In many ways, researchers working under the banner of research-practice partnerships have found a way to directly impact teaching and learning—by working directly with teachers and learners. While a disarmingly simple idea, this approach has profound implications both for who participates in the work of improving learning environments and for how that work is carried out.

In this book, we describe multiple efforts to use data-intensive research methods to improve teaching and learning. In particular, we highlight the important role that partnerships between researchers and practitioners can play in activating educational big data as a resource for improvement. Through the lens of what we refer to as collaborative data-intensive improvement (CDI), we aim to make explicit the ways in which educational researchers can engage in longer-term partnerships with the goal of not just understanding learning but also of improving outcomes in real-world learning environments. Doing this well, we believe, will require a fundamental rethinking of how data are used for research and improving practice.

Data-Intensive Research in Education

This book offers an introduction to the developing fields of educational data mining and learning analytics by describing goals, methods, and examples. In outlining the past, present, and potential future for these fields, throughout this book, we focus our descriptions on using data and complex data analyses to improve learning experiences and educational outcomes. We illustrate this potential with firsthand examples that span multiple academic content areas, learning environments, and learner types. We provide examples of decision making at the classroom, school, and education system levels taken from schools, universities, and community colleges.

Along with examples from our own work, we will describe how other researchers have employed educational data mining and learning analytics to address problems that originate in one form or another from the front lines of teaching and learning. In describing multiple examples and analytical approaches, we will highlight potential benefits and costs associated with each. The reader should know, however, that we are not attempting to provide a balanced treatment of all approaches. Our emphasis will be on collaborative data-intensive research approaches that prioritize shaping practical improvements over advancing analytic methods. While we will not restrict our coverage solely to collaborative data analysis approaches, they will be our lens for choosing what to highlight in a rapidly changing landscape. We hope that both researchers and practitioners will find this lens useful in making sense of new sources of education data, new analytic techniques, and new opportunities to form partnerships.

The Challenge of Jargon

One challenge facing newcomers to the field of data-intensive research is the wave of jargon they are likely to encounter. Already, in the first few pages of this book, we have referred to educational data mining, learning analytics, system log data, and big data. In an attempt to keep jargon to a manageable level, we have made explicit choices about the terminology we use in this volume, recognizing that some key details, distinctions, and research histories will be lost in this translation.

Before progressing further, we would like to orient the reader to a few key terms: educational data mining, learning analytics, data-intensive research, and educational data scientist.

Educational data mining and learning analytics represent distinct fields that have a high degree of overlap (Siemens & Baker, 2012). For simplicity, and to contrast these fields with other research traditions, we will refer to both of them as examples of data-intensive research in education. The additional fields that we want to integrate into learning analytics and educational data mining include studies of data use in schools (e.g., data-driven decision making) and collaborative research approaches (e.g., design-based implementation research and improvement science). As we will describe in Chapter 5, these additional fields are important both to the past and to the present of data-intensive research in the same ways that learning analytics and educational data mining are.

Educational data mining, which predates the field of learning analytics, largely concentrates on using machine learning techniques to identify patterns within large educational datasets, often from specific digital learning environments like intelligent tutoring systems. Oftentimes, these same technologies are what deliver interventions aimed at improving learning. Learning analytics, on the other hand, tends to focus less on machine learning techniques and more on statistical and visualization approaches, whereby interventions aimed at improving learning are delivered as much by an individual as a technology. As Baker and Inventado (2014) point out, the differences between these two fields grew out of different interests and backgrounds of the researchers in the two areas, and do not reflect any fundamentally opposing beliefs about how people learn. They agree on the assumption that data collected at scale and analyzed with rigorous methods will help arbitrate between different theories and proposed practices (Bienkowski, Feng, & Means, 2012).

Data-intensive research “involves data resources that are beyond the storage requirements, computational intensiveness, or complexity that is currently typical of the research field” (Dede, 2015, p. 2). The field of education more generally is gradually expanding its data repertoire to include data from digital learning environments and from increasingly sophisticated administrative data systems. In addition, other familiar forms of data, such as video and audio files, can now be explored at scale with greater speed. Therefore, we use the term data-intensive research to integrate these developing examples as well as those stemming from educational data mining and learning analytics.

An educational data scientist is someone who practices data-intensive research in education. The term “data scientist” is expansive and touches on multiple knowledge, skills, and abilities (see O’Neil & Schutt, 2013). Anyone who uses data-intensive research methods is often referred to as a data scientist. And while data science has become a hot new career (Ferenstein, 2016), the knowledge, skills, and abilities needed to perform this role are often ill-defined, especially in education (Piety, Hickey, & Bishop, 2014). Generally speaking, a data scientist is an individual with some combination of computer science skills, a background in statistics and mathematics, and relevant domain expertise (O’Neil & Schutt, 2013). Agasisti and Bowers (in press) define an educational data scientist as an individual who has “the technical skills to collect, analyze, and use quantitative data, and at the same time the managerial and communication skills to interact with decision-makers and managers at the school level to individuate good ways of using information in the practical way of improving practices and initiatives” (p. 6). In the coming chapters, we elaborate on these descriptions and make the case that an educational data scientist is someone who clarifies how data-intensive research methods can be used to address questions of importance to educators, carry out the actual analyses, and help develop and refine ideas for improvement.

Focus of the Book

Given the continuing proliferation of data and the increased sophistication of data-intensive research techniques, now is a good time to take stock of data-intensive research in education, articulate fruitful directions for advancing the field, and provide an onramp for newcomers. In working to achieve this ambitious and multifaceted aim, it is important to clarify what this book will and will not deliver. First, this book is not a how-to guide on data-intensive research methods in education. The interested reader can explore a growing number of learning analytics focused Massive Online Open Courses (MOOCs) for this purpose, such as Ryan Baker’s Big Data and Education, Tim McKay’s Practical Learning Analytics, and the University of Texas at Arlington’s upcoming MicroMasters on Learning Analytics. In addition to educational applications of analytics, a researcher or data scientist, at some point, will need to group rows of data and apply a function, such as identifying the average amount of time a student spent in a digital learning environment across multiple sessions. Depending upon one’s chosen software package, without too much difficulty, one could use a search engine to identify a serviceable answer. Less searchable are strategies for identifying sources of data in the first place and knowing how to work with practitioners to apply the right analytical technique to the right data and how to structure a meeting where researchers and practitioners come together to interpret and draw implications from a data-intensive analysis. In many ways, that is what this book is about.

This book is also not a standard course in educational research design or a program in educational leadership, though it does include elements and insights from these fields. It presents some fundamental research and leadership concepts as they relate to each other and to the goal of using data-intensive research to improve education outcomes. We seek to equip readers with an understanding of methods to enable clearer thinking about how new sources of data and new analytical techniques could help them create more desirable outcomes for students.

Examples of Data-Intensive Improvement

When Romero and Ventura (2007) surveyed the data mining literature for education applications published between 1995 and 2005, they found only two articles published before 2000. In contrast, by 2016, a Google Scholar search returned over one million articles on this topic. And educational applications of data-intensive research have moved beyond scholarly publications to capture the public’s imagination through popular press coverage such as a recent New York Times article, “Will You Graduate? Ask Big Data” (Treaster, 2017). In the following sections, we describe thre...

Table of contents

• Cover
• Title
• Copyright
• Contents
• List of figures
• List of tables
• List of boxes
• Preface
• Acknowledgements
• 1 Introduction
• 2 Data Used in Educational Data-Intensive Research
• 3 Methods Used in Educational Data-Intensive Research
• 4 Legal and Ethical Issues in Using Educational Data
• 5 Foundations of Collaborative Applications of Educational Data Mining and Learning Analytics
• 6 Supporting Conditions for Collaborative Data-Intensive Improvement
• 7 Five Phases of Collaborative Data-Intensive Improvement
• 8 Lessons Learned and Prospects for the Future
• Glossary
• Index