Invariant Measurement with Raters and Rating Scales

Rasch Models for Rater-Mediated Assessments

George Engelhard Jr., Stefanie Wind

• Publisher: Routledge
• Year: 2017
• ISBN:
  9781317661597
• Edition: 1
• Topic: Business
• Subtopic: Human Resource Management
• Index: Business

The purpose of this book is to present methods for developing, evaluating and maintaining rater-mediated assessment systems. Rater-mediated assessments involve ratings that are assigned by raters to persons responding to constructed-response items (e.g., written essays and teacher portfolios) and other types of performance assessments.

This book addresses the following topics: (1) introduction to the principles of invariant measurement, (2) application of the principles of invariant measurement to rater-mediated assessments, (3) description of the lens model for rater judgments, (4) integration of principles of invariant measurement with the lens model of cognitive processes of raters, (5) illustration of substantive and psychometric issues related to rater-mediated assessments in terms of validity, reliability, and fairness, and (6) discussion of theoretical and practical issues related to rater-mediated assessment systems.

Invariant measurement is fast becoming the dominant paradigm for assessment systems around the world, and this book provides an invaluable resource for graduate students, measurement practitioners, substantive theorists in the human sciences, and other individuals interested in invariant measurement when judgments are obtained with rating scales.

Part I
Introduction

1
Introduction and Overview

The essence of all forms of rating devices is the same. Some particular psychological continuum is defined … landmarks or guideposts [cues] are supplied along this continuum to aid a judge in the evaluations of samples to be placed on that continuum … a straight horizontal line is drawn to represent the continuum … the judge merely checks … that point on the continuum where he believes the sample should fall.

(Guilford, 1936, p. 263)

This chapter provides a general introduction to the principles of invariant measurement within the context of rating scales and rater judgments. As noted by Guilford (1936), there is a very simple underlying structure for all rating scales; however, the creation of a continuum to represent meaningful and useful constructs in the social, behavioral, and health sciences is far from simple. This book describes some of the key components to consider in the development of invariant measures obtained from raters within a variety of ecological contexts. Guilford (1936) highlighted several essential steps in the development of rating scales:

• Identify a continuum,
• Select cues (landmarks or guideposts) to define the continuum,
• Create a structure for the rating scale,
• Ask raters to evaluate the location of persons on the continuum by comparison with the cues on the continuum using a rating scale, and
• Assign ratings to persons based on their judged locations on the continuum.

These essential aspects of rating-scale methods must be addressed whenever rating scales are used to collect judgments in a variety of settings. The idea of a continuum is a very powerful way to represent constructs in the social sciences that are difficult to measure and to conceptualize. Figure 1.1 gives a visual representation of rating scales embodied in the opening quote. In this figure, the sample consists of three persons (A, B, and C) with true locations on the continuum (θ_s). The continuum ranges from low/easy to high/hard going from left to right on the line. The rater’s judgmental task is to locate the three persons on the continuum based on three cues (shown as short vertical lines on the continuum) that represent “cues,” such as performance-level descriptions. A rater judges that Person A is above the first cue and below the other two cues, and the rater assigns a rating pattern of [1 0 0] to Person A. The observed ratings are summarized in a similar fashion for Persons B and C in the table. The next step is to connect the observed rating patterns using a measurement model (dashed arrows between table and sample) to infer the location of persons on the continuum. In this simple illustration, the observed responses patterns suggest that Person C has the highest location on the continuum because the response pattern [1 1 1] suggests that this person is above the three cues. In this very simple example, the measurement model is called Guttman scaling. This book introduces and describes a variety of other measurement models that were created to aid researchers in linking observations to infer person locations on a continuum.

Figure 1.1 Visual Representation of Rating-Scale Methods Based on Guilford (1936)

As will be seen later in this chapter, Guilford’s concept of a continuum can be operationally defined in the form of a Wright map that can be used to represent a latent variable or construct. In earlier work, the Wright map has also been called a variable map, item map, and curriculum map (Wilson, 2011).

This chapter provides an overview and introduction to many of the themes that undergird this book on rater-invariant measurement. It lays the foundation for building a framework for invariant measurement with rating scales and raters. The following specific questions and issues are introduced in this chapter:

• What is invariance?
• What is measurement?
• What is invariant measurement?
• What is invariant measurement with raters?
• What are rating scales?
• What are rater-mediated Wright maps?
• Case study: Middle School Writing Assessment
• Summary and discussion

The questions and the underlying concepts are briefly illustrated with a small case study that examines the assessment of writing achievement among a group of middle-school students. The chapter concludes with a summary and discussion.

What Is Invariance?

Einstein, however, was not truly a relativist … beneath all of his theories, including relativity, was a quest for invariants … and the goal of science was to discover it.

(Isaacson, 2007, p. 3)

Invariance can be viewed as a fundamental concept in the physical and social sciences. As the subtitle of Nozick’s book (2001) on invariance suggests, the structure of the objective world can be viewed through the lens of the concept of invariance. A variety of definitions have been proposed for thinking about invariance. There is the commonsense notion that invariance is a property of attributes that includes relationships among other attributes that remain fixed within a stable and consistent framework for guiding research, theory, and practice. Within philosophy, invariance relates to objectivity and its twin concept of subjectivity, and it also has implications in the search for truths that endure beyond the details of a particular context. In addressing the concept of invariance in general, there is also an implicit concern with variance—if an attribute is determined not to be invariant, then it begs the question of how variant is it? As pointed out by Fisher (2008), “anomalous failures of invariance … [prompt] the search for explanations of consistent inconsistencies that might lead to discoveries of new variables” (p. 190). Nozick (2001) argues that almost all philosophers have “sought to establish permanent truth in an enduring framework of thought; these truths were supposed to be absolute, objective and universal … [and] to stand firm for all time” (p. 1). When the search for invariant “truths” fails, then the quest for invariance has still provided a useful philosophical framework for understanding the journey and discussing the aspects of relevant variance.

Turning from philosophy to science, the general concept of invariance also plays an important role in scientific endeavors. As indicated in the opening quote for this section, Einstein was not truly a relativist, and within the field of physics he considered labeling his ideas a theory of invariance (Isaacson, 2007). Stevens (1951) stressed that “the scientist is usually looking for invariance whether he knows it or not” (p. 20). Nozick (2001) points out that to understand something means that “we want to know the transformations it is invariant under and also the transformations it is variant under” (p. 78). Science can be viewed as a systematic examination and search for these invariances.

Invariance is important as a scientific concept because it reflects what researchers are seeking in order to provide a stable framework and structure for understanding the world around us. We recommend reading Nozick (2001) for a thoughtful discussion of invariance, and its implications for philosophy and the sciences. In this book, an important lesson for the reader is to recognize that invariance and variance are essentially two sides of the same coin. Scientists may seek invariance in the attributes that are being studied, but if invariance is not found it becomes important to identify sources of variance. It seems to us that frequently the importance of invariance as a goal for science is not recognized by researchers who tend to stress the noise, variance, and uncertainty in our activities rather than the messages conveyed in invariant relationships between attributes that we are seeking to discover. Everything is not invariant, but it is important for scientists to sort out what facets are stable and consistent from the facets that are ephemeral, unpatterned, and variant in the world around us.

The word invariance has been widely used in several different disciplines that range from mathematics and physics to philosophy. Each discipline has slightly different denotations and connotations of the term. In this book, invariance is viewed in a fairly commonsensical way as stability and consistency across different contexts that can be defined and studied from different perspectives. Fisher (2008) stresses the use of “the metaphysical status of invariance as the criterion hallmark of meaningfulness and existence, and so of validity,” (p. 190) across many scientific disciplines.

What Is Measurement?

MEASUREMENT consists in the assignment of numerals to things or properties. But not every assignment of numerals is measurement. A street is not measured when numerals are assigned to the houses in it; a dyer does not measure his colours when he assigns numerals to them in his catalogue. To understand in what circumstances the assignment of numerals is measurement, we must ask what is the significance of numerals as distinct from other symbols, such as hieroglyphics or the letters of our alphabet.

(Campbell & Jeffreys, 1938, p. 121, capitalization in the original)

Measurement, in its broadest sense, is defined as the assignment of numerals to objects or events according to rules.

(Stevens, 1946, p. 677)

There are a variety of different definitions of measurement that have been proposed over the years. Many of the definitions echo the quotes above regarding the assignment of numbers or numerals on the basis of rules. In many ways, the essence of this definition still stands, but it leaves open the question of how to define the rules. The rules define the conditions (requirements and assumptions) that must be met in order to achieve the important implications of invariant measurement for theory and practice. Some researchers have tried to define measurement in a theoretical and ecological vacuum. However, measurement at the end of the day is about the development of meaningful scores that locate persons on a continuum (latent variable or construct) that can be used to make decisions about each person. Measurement and the development of various instruments and scales occur within theoretical and purposive contexts that cannot be ignored. For example, Chang (2004) has described how measurement issues played an integral part in the invention of the concept of temperature. He offers an instructive example that captures many of the challenges researchers also face in developing scales in the social sciences.

As pointed out by the sociologist Paul Lazarsfeld, measurement theories are important because they define

problems of concept formation, of meaning, and of measurement necessarily fuse into each other … measurement, classification and concept formation in the behavioral sciences exhibit special difficulties. They can b...

Table of contents

• Cover
• Title
• Copyright
• Dedication
• CONTENTS
• Acknowledgments
• Preface
• PART I Introduction
• PART II Theories of Measurement and Judgment for Rating Scales
• PART III Foundational Areas for Rating Scales
• PART IV Technical Issues and IRT Models for Ratings
• PART V Practical Issues
• PART VI Final Word
• Glossary
• Index