Modeling Human Behavior With Integrated Cognitive Architectures

Comparison, Evaluation, and Validation

Kevin A. Gluck, Richard W. Pew

• Publisher: Psychology Press
• Year: 2006
• ISBN:
  9781135610487
• Edition: 1
• Topic: Computer Science
• Subtopic: Artificial Intelligence (AI) & Semantics
• Index: Computer Science

Resulting from the need for greater realism in models of human and organizational behavior in military simulations, there has been increased interest in research on integrative models of human performance, both within the cognitive science community generally, and within the defense and aerospace industries in particular. This book documents accomplishments and lessons learned in a multi-year project to examine the ability of a range of integrated cognitive modeling architectures to explain and predict human behavior in a common task environment that requires multi-tasking and concept learning.This unique project, called the Agent-Based Modeling and Behavior Representation (AMBR) Model Comparison, involved a series of human performance model evaluations in which the processes and performance levels of computational cognitive models were compared to each other and to human operators performing the identical tasks. In addition to quantitative data comparing the performance of the models and real human performance, the book also presents a qualitatively oriented discussion of the practical and scientific considerations that arise in the course of attempting this kind of model development and validation effort.
The primary audiences for this book are people in academia, industry, and the military who are interested in explaining and predicting complex human behavior using computational cognitive modeling approaches. The book should be of particular interest to individuals in any sector working in Psychology, Cognitive Science, Artificial Intelligence, Industrial Engineering, System Engineering, Human Factors, Ergonomics and Operations Research. Any technically or scientifically oriented professional or student should find the material fully accessible without extensive mathematical background.

Part I
OVERVIEW, EXPERIMENTS, AND SOFTWARE

Chapter 1

Background, Structure, and Preview of the Model Comparison

Kevin A. Gluck
Air Force Research Laboratory

Richard W. Pew
BBN Technologies

Michael J. Young
Air Force Research Laboratory

The U.S. military services have developed a variety of systems that allow for synthetic human behavior representation (HBR) in virtual and constructive simulation. Examples include the Army’s Modular Semi-Automated Forces (ModSAF), the Navy and Marine Corps’ Small Unit Tactical Trainer (SUTT), the Air Force’s Advanced Air-to-Air System Performance Evaluation Model (AASPEM), and the Joint Services’ Command Forces (CFOR) project. Pew and Mavor (1998) described these systems and others and then noted that, although it is possible to represent human behavior in these systems, the state of the human representation is almost always rudimentary. In the words of Pew and Mavor:

This lack of human performance representation in models becomes more significant as the size, scope, and duration of wargaming simulations continues to grow. In the future, these limitations will become more noticeable as greater reliance is placed on the outcomes of models/simulations to support training and unit readiness, assessments of system performance, and key development and acquisition decisions. (see page)

To begin addressing the problems associated with limited HBR capability, developers of these and future military modeling and simulation systems should begin to draw more from cognitive, social, and organizational theory. In particular, Pew and Mavor (1998) suggested that these modeling systems would benefit from a closer association with the developers of integrative HBR architectures.

In the psychology literature, the term architecture is often used instead of system (e.g., Anderson, 1983, 1993; Newell, 1990; Pylyshyn, 1991). A psychological architecture differs from other modeling and simulation systems in that it makes a priori assumptions that constrain the representations and processes available for use in a model on the basis of the theories underlying the architecture. By virtue of these constraints, architectures are a distinct subset of the total set of possible human representation systems. Chapter 3 of the Pew and Mavor (1998) text provides a description of the major characteristics of 11 integrative architectures. Ritter et al. (2003) described seven more. Morrison (2004) reviewed the characteristics of many of the same architectures and added still another six to the list. This totals at least 24 human representation architectures included in recent reviews.

We recently went through that list of two dozen architectures to confirm the availability of each as implemented in software that can be used to develop models and that perhaps also could be integrated into a larger simulation system. The subset of psychologically inspired human representation architectures that meet this availability criterion¹ is listed in Table 1.1. We do not review the characteristics of these architectures here because that would be redundant with the three recent reviews, but we encourage the interested reader to seek out these references or read about the architectures on their respective Web sites.

The existence of such an assortment of HBR architectures is an indication of the health and vitality of this research area. Yet there is considerable room for improvement. All of the architectures have shortcomings in their modeling capabilities, and none of them is as easy to use as we would like them to be. There is enormous interest in greater breadth, increased predictive accuracy, and improved usability in models of human performance and learning. These interests motivated the creation of a research project that would move the field in those directions.

THE AMBR MODEL COMPARISON

This unique project, called the Agent-Based Modeling and Behavior Representation (AMBR) model comparison, was sponsored primarily by the U.S. Air Force Research Laboratory (AFRL), with additional funding from the Defense Modeling and Simulation Office (DMSO) and the Office of Naval Research (ONR). The AMBR model comparison involved a series of human performance model evaluations in which the behaviors of computer models were compared to each other and to the behaviors of actual human operators performing the identical tasks.

TABLE 1.1
Human Behavior Representation Architectures Available for Use

The Approach

Considered in isolation, there is nothing unique about developing models and comparing them to human data. Cognitive science and other related disciplines are replete with such activities. The unique nature of the project is revealed only through consideration of the details of our approach and how it relates to similar efforts.

A previous research project with which the AMBR comparison shares a close affinity is the Hybrid Architectures for Learning Project sponsored by ONR in the mid- to late 1990s. Hybrid Architectures was committed to improving our understanding of human learning by funding the development of various cognitive architecture-based and machine learning-based models in three different learning contexts. The modeling goal was “... to run the basic hybrid model on a selected task to verify the model’s performance relative to the actual human data and to evolve the model, increasing the match between the learned performances, to obtain a better predictive/explanatory model of the human process” (Gigley & Chipman, 1999, p. 2). The emphases on (a) iterative improvements to computational models and model architectures, and (b) evaluating these improvements through comparison to human data both find parallel emphases in AMBR. There was even an intention in Hybrid Architectures to eventually conduct a thorough comparison of the models that had been developed for the various tasks, but unfortunately the funding for the project disappeared before a final comparison took place. The major methodological differences between the two projects are that (a) all of the AMBR modelers developed models of the same tasks to facilitate comparison, and (b) detailed comparison of the models was an integral part of AMBR and took place on a recurring basis.

Another effort that can help illuminate some of the distinctive characteristics of the AMBR model comparison is the comparison of models of working memory that took place in the late 1990s. The working memory model comparison initially took the form of a symposium and eventually evolved into a book on the topic (Miyake & Shah, 1999). Their goal was to compare and contrast existing models of working memory by having each modeler address the same set of theoretical questions about their respective model’s implementation. There are probably more differences than similarities between their effort and AMBR, although both approaches were effective in achieving their objectives. One distinction is that the AMBR models were all implemented in computational process models that can interact with simulated task environments, whereas the working memory models came from an assortment of modeling approaches, including verbal/conceptual theories. Another distinction is that the AMBR model comparison was partially motivated by an interest in encouraging computational modelers to improve the implementations and/or applications of their architectures by pushing on their limits in new ways, whereas the working memory model comparison did not fund the development of new models or architectural changes. A third distinction is that, as mentioned previously, all of the AMBR modelers were required to address the same task scenarios, whereas the working memory modelers each focused on a task of their own choosing. In chapter 12 of the Miyake and Shah (1999) book, Kintsch, Healy, Hegarty, Pennington, and Salthouse (1999) applaud the success of the editors’ “common questions” approach to comparing the models. It is noteworthy that they then go on to recommend the following for model comparisons:

... we would like to emphasize that, to the extent that direct experimental face-offs among models are possible, they should certainly be encouraged. Obviously, such comparisons would be very informative, and much more could be and should be done in this respect than has heretofore been attempted. (p. 436)

Although not originally inspired by this quote, the strategy adopted in AMBR of having each model address the same experiment scenarios is consistent with the Kintsch et al. recommendation. It is also consistent with the proposal a decade earlier by Young, Barnard, Simon, and Whittington (1989) that HCI researchers adopt the use of scenarios as a methodological route to models of broader scope.

Hopefully the previous paragraphs gave the reader an appreciation for the general research approach selected for the AMBR model comparison, but this tells us little of the precise process that was followed. There were two experiments in the AMBR model comparison pursued sequentially. The first focused on multitasking, and the second focused on category learning. Each of the two experiments involved the following steps:

1. Identify the modeling goals—what cognitive/behavioral capabilities should be stressed?
2. Select a task domain that requires the capabilities identified in (1) and that is of relevance to AF modeling and simulation needs.
3. Borrow/modify/create a simulation of the task domain that either a human-in-the-loop or a human performance model can operate.
4. Hold a workshop at which the model developers learn about the task and modeling environment and exchange ideas with the moderator concerning potential parameters that can be measured and constraints of the individual models that need to be accommodated.
5. Moderator team collects and disseminates human performance data.
6. Modeling teams develop models that attempt to replicate human performance when performing the task.
7. Expert panel convenes with the entire team to compare and contrast the models that were developed and the underlying architectures that support them.
8. Share the results and lessons learned with the scientific community to include making available the simulation of the task domain and the human performance data.

We should note that some of the data were withheld from the modelers in the second comparison, which focused on category learning. We say more about that later.

Manager, Moderator, and Modelers

The project involved people from a variety of organizations representing government, industry, and academia. The Air Force Research Laboratory’s Warfighter Training Research Division managed the effort. BBN Technologies served in the role of model comparison moderator. They designed the experiments, provided the simplified air traffic control (ATC) simulation environment implemented in D-OMAR (Deutsch & Benyo, 2001; Deutsch, MacMillan, & Cramer, 1993), and collected data on human operators performing the task. Additional data for the second comparison (category learning) were collected at the University of Central Florida, with supervision from colleagues at NAVAIR Orlando (Gwen Campbell and Amy Bolton). There were four modeling teams. Two of the teams (CHI Systems and a team from George Mason University and Soar Technology) were selected as part of the competitive bidding process at the beginning of the first comparison. A team from Carnegie Mellon University joined the first comparison in mid-course, with funding from the Office of Naval Research. Finally, a fourth modeling team, this one from the Air Force Research Laboratory’s Logistics and Sustainment Division, participated on their own internal funding.

Goals

There were three goals motivating the AMBR model comparison, all of which bear a striking resemblance to the recommendations made by the National Research Council (NRC) Panel on Modeling Human Behavior and Command Decision Making (Pew & Mavor, 1998).

Goal 1: Advance the State of the Art. The first goal was to advance the state of the art in cognitive modeling. This goal is consistent with the spirit of the entire set of recommendations from the NRC panel because their recommendations were explicitly intended as a roadmap for improving human and organizational behavior modeling. The model comparison process devised for this project provides a motivation and opportunity for human modelers to extend and test their architectures in new ways. As should be apparent in the subsequent chapters in this book, there is ample evidence that these modeling architectures were challenged and improved as a direct result of their participation in this project.

Goal 2: Develop Mission-Relevant HBR Models. The second goal was to develop HBR models that are relevant to the Department of Defense (DoD) mission, and therefore provide possible transition opportunities. This is consistent with the NRC panel recommendation to support model development in focused areas of interest to the DoD. The two modeling focus areas selected for AMBR were multitasking and category learning. We say more about each of those areas shortly.

Goal 3: Make Tasks, Models, and Data Available. The third goal was to make all of the research tasks, human behavior models, and human process and outcome data available to the public. This is consistent with the NRC panel recommendation for increased collection and dissemination of human performance data. We have described various subsets of the results from the AMBR model comparison at several different conferences over the last 3 years, resulting in almost three dozen conference papers and technical reports. This book, however, is the most comprehensive source of information regarding the scientific output of the AMBR model comparison.

Experiment 1: Multitasking

The AMBR model comparison was divided into two experiments, with a different modeling focus in each. The modeling focus for Experiment 1 was multiple task management because this area represents a capability that is not widely available in existing models or modeling architectures, and because more knowledge regarding how to represent this capability provides an opportunity to improve the fidelity of future computer-generated forces (CGFs). It was the responsibility of the moderator (BBN) to select a task for simulation that emphasized multiple-task management.

Two approaches, representing ends of a continuum of intermediate possibilities, were considered....

Table of contents

• Cover
• Haft title
• Title Page
• Copyright Page
• Dedication
• Table of Contents
• Contributors
• Preface
• Acknowledgments
• Part I Overview, Experiments, and Software
• Part II Models of Multitasking and Category Learning
• Part III Conclusions, Lessons Learned, and Implications
• Author Index
• Subject Index