Naturalistic Decision Making and Macrocognition

Laura Militello, Raanan Lipshitz, Jan Maarten Schraagen

• Publisher: CRC Press
• Year: 2017
• ISBN:
  9781317089582
• Edition: 1
• Topic: Technology & Engineering
• Subtopic: Industrial Health & Safety
• Index: Technology & Engineering

This book presents the latest work in the area of naturalistic decision making (NDM) and its extension into the area of macrocognition. It contains 18 chapters relating research centered on the study of expertise in naturalistic settings, written by international experts in NDM and cognitive systems engineering. The objective of the book is to present the reader with exciting new developments in this field of research, which is characterized by its application-oriented focus. The work addresses only real-world problems and issues. For instance, how do multi-national teams collaborate effectively? How can surgeons best be supported by technology? How do detectives make sense of complex criminal cases? In all instances the studies have been carried out on experts within their respective domains. The traditional field of NDM is extended in this work by focusing on macrocognitive functions other than decision making, namely sense-making, coordination and planning. This has broadened the scope of the field. The book also contains a theoretical discussion of the macro-micro distinction. Naturalistic Decision Making and Macrocognition will be relevant to graduate students, researchers and professionals (including professionals and researchers in business, industry and government) who are interested in decision making, expertise, training methods and system design. The material may be used in two ways: theoretically, to advance understanding of the field of naturalistic decision making; and practically, to gain insight into how experts in various domains solve particular problems, understand and deal with issues and collaborate with others.

PART I
Orientation

Chapter 1
The Macrocognition Framework of Naturalistic Decision Making

Jan Maarten Schraagen, Gary Klein, and Robert R. Hoffman

Introduction

Naturalistic Decision Making (NDM), as a community of practice, has the goal of understanding cognitive work, especially as it is performed in complex sociotechnical contexts. In recent years, the concept of “macrognition” has emerged as a new and potential umbrella term to designate the broader paradigm that underlies NDM. In addition, the notion of macrocognition presents challenges and opportunities for both theory and empirical methodology. The present volume is a contribution to this literature, the seventh volume in the NDM series.

In this chapter we accomplish a number of things. First, we chart the history of NDM as a community of practice and then describe its stance concerning cognitive work and research methodology. Next, we chart the history of the concept of macrocognition and then show how NDM converges with it philosophically. Finally, we use these contexts to overview the chapters in this volume.

Emergence of the NDM Community of Practice

NDM as a community of practice began with the first conference in 1989 in Dayton, Ohio. That first conference was kept small—only about 30 researchers were invited, based on their interests and activities. Many had been funded by the Basic Research Unit of the Army Research Institute. Judith Orasanu, who was then working in this unit, provided ARI funding to organize the 1989 meeting. The goal of the meeting was simply to assess whether these researchers did in fact have a common, and perhaps distinctive set of goals and methods, and whether those were in any way coherent—even though many of them were studying different domains for different reasons.

The 1989 meeting was intended as a workshop to allow sharing of recent results and interests, but it sparked demand for follow-on gatherings. The NDM community has met every two to three years since then, alternating between North American and European venues. Seven such meetings have been held to date. Thus far, each of the NDM meetings has generated a book describing the research and the ideas of the conference participants (Hoffman, 2007; Klein et al., 1993; Zsambok and Klein, 1997; Flin et al., 1997; Salas and Klein, 2001; Montgomery, Lipshitz and Brehmer, 2005).

How the NDM researchers have managed to maintain their community of practice is perhaps somewhat mysterious. There is no formal society, no officers, no dues, and no newsletters. At the end of each conference, all the attendees who are interested in helping with the next conference gather together to select a host and site. There are always several volunteers to organize the next conference. The community is sustained by common interests and by a desire to find out what the other researchers have been up to. There is of course a great deal of behind-the-scenes work focused on securing sponsorships that really make the meetings possible. Supporters have included the Office of Naval Research, the Army Research Institute, the Army Research Laboratory, the Human Effectiveness Directorate of the US Air Force, the National Aeronautics and Space Administration (NASA), the US Navy, and TNO Human Factors.

In addition to the NDM meetings, many NDM researchers gather every year as part of the Cognitive Ergonomics and Decision Making Technical Group within the Human Factors and Ergonomics Society, and at meetings on Situation Awareness.

The Paradigm of Naturalistic Decision Making

In the 1980s, a number of researchers adopted a concept of decision making that seemed quite different from the standard “option generation and comparison” framework. Lipshitz (1993) tabulated nine different models of decision making that had been proposed by this emerging community of researchers over that decade. Two of the most widely cited models were Rasmussen’s (1983, 1988) Skills/Rules/Knowledge account along with the “decision ladder,” and Klein’s (1989) Recognition-Primed Decision (RPD) model. The concept of decision making had often been defined in terms of a gamble: given two or more options, with certain information about the likelihood of each option to succeed, which is selected? However, the early NDM studies found that people (domain practitioners, consumers, managers, and so on) rarely made these kinds of decisions. Some have suggested the Klein, Calderwood, and Clinton-Cirocco (1986) study of firefighters marks the beginnings of NDM. Using a structured interview method, the researchers found that fire fighters do not evaluate options. They do not conduct anything like a “utility analysis” in which a list of options is generated, and each option is evaluated. More importantly, this is a domain in which decisions could not possibly be made using utility analysis. Thus, what purchase on reality was had by “normative” models that describe how rational decisions should be made? The house would burn down, or worse, people would die. In many domains, decision makers often have to cope with high-stakes decisions under time pressure where more than one plausible option does exist, but the decision makers use their experience to immediately identify the typical reaction. If they cannot see any negative consequence to adopting that action, they proceed with it, not bothering to generate additional options or to systematically compare alternatives. Thus, the metaphor of a decision as a gamble didn’t seem to apply very often. If the metaphor of the decision as gamble failed to describe what practitioners usually encounter and usually do, NDM would abandon the metaphor and follow the phenomena.

NDM wanted to explore how domain practitioners make decisions in the “real world,” under difficult conditions, in order to help them do a better job (Orasanu and Connolly, 1993). Such a goal statement should seem straightforward and yet it triggered a surprising amount of controversy. The lead article by Lipshitz et al. (2001) in a special issue on NDM in the Journal of Behavioral Decision Making was accompanied by skeptical commentaries from the Judgment and Decision Making community. Some questioned whether there was anything new about NDM that had not already been embraced by Behavioral Decision Making, and others doubted that NDM had much chance of succeeding. Those criticisms are orthogonal, of course, but were voiced with almost equal vigor and sometimes by the same people.

What is it that arouses such strong feelings, pro and con, for the NDM enterprise?

Points of Contention

We see three points of contention: approach to subject-matter experts, approach to improving decision making, and approach to research. First, NDM researchers do not see domain practitioners as infallible, but nevertheless respect their dedication, skills, and knowledge. And researchers deeply appreciate any “face time” with experts. NDM researchers want to document practitioner abilities in order to make sure that the subtle skills they have are recognized, understood, and supported in training programs and in decision support systems. NDM researchers seek to understand the true work (for example, information needs and decision requirements). This stance towards the participants in research puts NDM in conflict with some other decision researchers for a number of reasons. Some argue that experts are not special in any way, or that “expertise” is a biased, elitist notion. Some researchers take a fundamental stance: The belief that people tend to follow economic (or “rational”) models of costs and benefits when they make decisions. NDM also conflicts with the “heuristics and biases” approach to decision making–NDM sees the strengths in the heuristics, but it looks beyond superficial attributions of human limitations (Flach and Hoffman, 2003), and does not assume that experts are as prone to biases as the literature on heuristics and biases suggests, or even that bias is an inherent and inevitable feature of human decision making.

Second, the NDM stance on improving decision making is to help practitioners apply their expertise more effectively, and help non-experts achieve expertise faster. NDM researchers do not assume that the practitioner has to be force-fed a probability scaling task in order to avoid one or another of the dozens of biases that are believed to pervade human thought. This stance seems to conflict with the position of Behavioral Decision Making to formulate strategies and aids that can replace or fix unreliable human judgment, for example, by having them work through a Bayesian probability evaluation procedure.

Third, the NDM stance on studying decision making emphasizes cognitive field research and cognitive task analysis (Hoffman and Woods, 2000; Schraagen, Chipman, and Shalin, 2000). Today, we have a rather large palette of cognitive task analysis methods (Crandall, Klein, and Hoffman, 2006; Hoffman and Militello, 2008), including the Critical Decision Method, Concept Mapping, various forms of task and goal analysis, and various types of experimental methods such as the Macrocognitive Modeling Procedure (see Klein and Hoffman, this volume). NDM researchers sometimes use simulations, but these have to reflect key challenges of the tasks and engage practitioners in realistic dilemmas. One thing NDM research does not do is use artificial paradigms that can be run on college “subjects” in 50-minute sessions.

One thing NDM research usually does is rely on methods of structured interviewing and task retrospection. So the very nature of the investigations causes discomfort to some experimental psychologists. NDM deliberately looks for “messy” conditions such as ill-defined goals, high stakes, organizational constraints, time pressure. Such conditions are difficult to capture in the laboratory but certainly determine the types of decision strategies people use in the “real world.”

The mission of NDM—to understand how people make decisions under difficult conditions, and how to help them do a better job—meant that researchers could not confine themselves to particular tried-and-true paradigms or stovepiped “fundamental mental operations.” Instead, NDM researchers expanded their focus from decision making to cognitive functionalities such as sensemaking, planning, replanning, and related phenomena such as mental modeling and the formation, use, and repair of “common ground” by teams. For example, McLennan and Omodei (1996) have examined pre-decision processes that appear to be critical to success. Mica Endsley’s (1995a; Endsley and Garland, 2000) work on situation awareness is central to much of the NDM research. So is David Woods’ examination of resilience and disturbance management (Hollnagel and Woods, 1983, 2005) and Vicente’s (1999) description of Cognitive Work Analysis methodology. The very notion that decisions are things that are “made” came into question (Hoffman and Yates, 2005).

Since the original 1989 workshop, the NDM community has expanded its mission to “understanding how people handle difficult cognitive demands of their work, and trying to help them do a better job.” To have retained an exclusive focus on decision making would have lost sight of the phenomena that were being studied, and could have disenfranchised some NDM researchers, including the authors of this chapter. Thus, NDM does not seem to be just the study of decision making. Certainly no one ever saw benefit to actually limiting the scope of investigations to decision making. The focus of interest has been more directed by the real-world settings that NDM researchers explore and by the demands that these settings place on the people who are responsible for getting the job done, efficiently, effectively, and safely. As a result, some have wondered whether NDM should change its name.

As the NDM framework broadened, researchers came to realize that they were interested in the cognitive functions that were carried out in natural settings. “Naturalistic Decision Making” was evolving into “Naturalistic Cognition.” The same kind of mission still applied, and the same cognitive field research and cognitive task analysis methods still applied. But it was time to recognize that the interests of the NDM community had expanded. It came to be generally understood that the designation of NDM made sense primarily in historical context—as a reminder of the initial successes in discovering how decisions are made under time pressure and uncertainty and the importance of studying decision making in real-world contexts—but no longer captured the spirit and mission of the movement.

Origins of the Concept of Macrocognition

The line of discussion that led to the term “macrocognition” began in 1985 when at a NATO-sponsored conference on intelligent decision support systems for process control, Gunnar Johanssen distinguished micro-and macro-levels in an analysis of decision-making situations:

Decision making is required on all levels of social life and in all work situations…The macro-operational situations are characterized by the need for decision making during such tasks as goal-setting, fault management, and planning in systems operations or maintenance of man–machine systems…The micro-operations situations involve decision making as an ingredient of control processes, either manual or supervisory, in man–machine systems. [pp. 328–31]

Although this is not quite the sense of macro–micro we rely on today in NDM, it is clearly pointing in the direction of looking at the phenomenology of cognitive work (see Klein et al., 2003).

Not surprising, given that David Woods was a participant in the 1985 conference and co-editor of the resultant volume (Hollnagel, Mancini, and Woods, 1985), the notion of macrocognition, and the distinction with microcognition was manifest in Woods and Roth’s (1986) discussion of a hierarchy of decision-making situations—including organizational, macro-operational, and micro-operational levels, in reference to process control for nuclear power.

Ten years later, Pietro Cacciabue and Erik Hollnagel (1995) contrasted macrocognition with microcognition in order to present a view for human–machine systems design that would not take an information-processing approach. This alternative description is of cognitive functions that are performed in natural as opposed to laboratory settings:

Micro-cognition is here used as a way of referring to the detailed theoretical accounts of how cognition takes place in the human mind…the focus is on “mechanisms of intelligence” per se, rather than the way the human mind works. Micro-cognition is concerned with the building of theories for specific phenomena and with correlating the details of the theories with available empirical and experimental evidence. Typical examples of micro-cognition are studies of human memory, of problem solving in confined environments (for example, the Towers of Hanoi), of learning and forgetting in specific tasks, of language understanding, and so on. Many of the problems that are investigated are “real,” in the sense that they correspond to problems that one may find in real-life situations—at least by name. But when they are studied in terms of micro-cognition the emphasis is more on experimental control than on external validity…Macro-cognition refers to the study of the role of cognition in realistic tasks, that is in interacting with the environment. Macro-cognition only rarely looks at phenomena that take place exclusively within the human mind or without overt interaction. It is thus more concerned with human performance under actual working conditions than with controlled experiments. [pp. 57–8]

Cacciabue and Hollnagel argued that the forms taken by macrocognitive theories and microcognitive theories are different, with macrocognitive theories being unlike, for instance, information-processing flow diagrams or sets of procedural rules.

At this point in time the notion of macrocognition had two elements. One was what we might call the Johanssen-Woods assertion that cognitive work can only be understood through study at a number of levels or perspectives (see also Rasmussen, 1986). The other was the Cacciabue-Hollnagel assertion that the information-processing approach provides an incomplete and incorrect understanding of cognitive work.

In 2000, Klein et al. suggested the concept of macrocognition as an encompassing frame for studying the cognitive processes that emerged in complex settings. They attempted to encourage a dialog between laboratory and field researchers. Like Cacciabue and Hollnagel, Klein et al. defined macrocognition as the study of complex cognitive functions, including decision making, situation awareness, planning, problem detection, option generation, mental simulation, attention management, uncertainty management and expertise. In other words, it was dawning on people that macrocognition is what NDM is really about, after all.

Expansion of the Notion of Macrocognition

Klein et al. (2003) saw macrocognition as a broader framework for NDM, more than the Johanssen notion of levels or perspectives, and more than the mere expansion of NDM to cover phenomena other than decision making. There are explanatory models such as the Recognition-Primed Decision-making model (RPD) (Klein, 1998), decision pre-priming (McLennan and Omodei, 1996), and levels of situation awareness (Endsley, 1995a). There are emergent functional phenomena such as sensemaking (Klein, Moon, and Hoffman, 2006a, b), and problem detection (Klein et al., 2005). Macrocognition is seen as the study of cognitive phenomena found in natural settings, especially (but not limited to) cognitive work conducted in complex sociotechnical contexts. The concept of macrocognition retains the essence of NDM, but with a broader mandate. Figure 1.1 describes the key macrocognitive functions listed by Klein et al. (2003): decision making, sensemaking, planning, adaptation/ replanning, problem detection, and coordination. Some of these, such as problem detection, emerge in field settings, are rarely considered in controlled laboratory-based experiments, and would be unlikely to emerge in typical laboratory studies of cognition (for example, studies of how people solve pre-formulated puzzles would be unlikely to demonstrate the phenomenon of problem-finding)...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• Contents
• List of Figures
• List of Tables
• Notes on Contributors
• Foreword
• PART I ORIENTATION
• PART II THE HEARTLAND OF MACROCOGNITION
• PART III MICRO-MACRO RELATIONSHIPS
• PART IV ALTERNATIVE APPROACHES
• Index of Names
• Subject Index