Constructing Measures

An Item Response Modeling Approach

Mark Wilson

• Publisher: Routledge
• Year: 2004
• ISBN:
  9781135618049
• Edition: 1
• Topic: Education
• Subtopic: Education General

Constructing Measures introduces a way to understand the advantages and disadvantages of measurement instruments, how to use such instruments, and how to apply these methods to develop new instruments or adapt old ones. The book is organized around the steps taken while constructing an instrument. It opens with a summary of the constructive steps involved. Each step is then expanded on in the next four chapters. These chapters develop the "building blocks" that make up an instrument--the construct map, the design plan for the items, the outcome space, and the statistical measurement model. The next three chapters focus on quality control. They rely heavily on the calibrated construct map and review how to check if scores are operating consistently and how to evaluate the reliability and validity evidence. The book introduces a variety of item formats, including multiple-choice, open-ended, and performance items; projects; portfolios; Likert and Guttman items; behavioral observations; and interview protocols.Each chapter includes an overview of the key concepts, related resources for further investigation and exercises and activities. Some chapters feature appendices that describe parts of the instrument development process in more detail, numerical manipulations used in the text, and/or data results. A variety of examples from the behavioral and social sciences and education including achievement and performance testing; attitude measures; health measures, and general sociological scales, demonstrate the application of the material. An accompanying CD features control files, output, and a data set to allow readers to compute the text's exercises and create new analyses and case archives based on the book's examples so the reader can work through the entire development of an instrument. Constructing Measures is an ideal text or supplement in courses on item, test, or instrument development, measurement, item response theory, or rasch analysis taught in a variety of departments including education and psychology. The book also appeals to those who develop instruments, including industrial/organizational, educational, and school psychologists, health outcomes researchers, program evaluators, and sociological measurers. Knowledge of basic descriptive statistics and elementary regression is recommended.

Part I
A Constructive Approach to Measurement

Chapter 1
Construct Modeling: The “Four Building Blocks” Approach

1.0 CHAPTER OVERVIEW AND KEY CONCEPTS

construct modeling

the “four building blocks”

construct map

items design

outcome space

measurement model

This chapter begins with a description of what is meant by measurement in this book. The remainder of the chapter then outlines a framework, which I call construct modeling, for understanding how an instrument works by understanding how it is constructed. Construct modeling is a framework for developing an instrument by using each of four “building blocks” in turn. This chapter summarizes all four building blocks, and the following chapters describe each in detail. In this volume, the word instrument is defined as a technique of relating something we observe in the real world (sometimes called manifest or observed) to something we are measuring that only exists as part of a theory (sometimes called latent or unobserved). This is somewhat broader than the typical usage, which focuses on the most concrete manifestation of the instrument—the items or questions. Because part of the purpose of the book is to expose the less obvious aspects of measurement, this broader definition has been chosen. Examples of types and formats of instruments that can be seen as coming under the “construct mapping” framework are shown in this and the next few chapters. Generally, it is assumed that there is a respondent who is the object of measurement, and there is a measurer who seeks to measure something about the respondent. While reading the text, the reader should mainly see him or herself as the measurer, but it is always useful to assume the role of the respondent as well. The next four chapters explain each of the four building blocks in turn, giving much greater detail, many examples, and discussion of how to apply the ideas to instrument development.

1.1 WHAT IS MEASUREMENT?

In some accounts, measurement is defined as the assignment of numbers to categories of observations. The properties of the numbers become the properties of the measurement—nominal, ordinal, interval, ratio, and so on. (Stevens, 1946).¹ Assigning numbers to categories is indeed one feature of the account in this book; correspondingly, those numbers have certain properties. Yet that is only one aspect of the process of measurement—there are steps preceding the assignment of numbers that prepare the ground for measuring, and there are also steps after the assignment of numbers that (a) check that the assignment was successful, and (b) make use of the measurements.

The central purpose of measurement, as interpreted here, is to provide a reasonable and consistent way to summarize the responses that people make to express their achievements, attitudes, or personal points of view through instruments such as attitude scales, achievement tests, questionnaires, surveys, and psychological scales. That purpose invariably arises in a practical setting where the results are used to make some sort of decision. These instruments typically have a complex structure, with a string of questions or tasks related to the aims of the instrument. This particular structure is one reason that there is a need to establish measurement procedures. A simpler structure—say just a single question—would allow simpler procedures. However, there are good reasons that these instruments have this more complex structure, and those reasons are discussed in the following chapters.

The approach adopted here is predicated on the idea that there is a single underlying characteristic that an instrument is designed to measure. Many surveys, tests, and questionnaires are designed to measure multiple characteristics—here it is assumed that we can consider those characteristics one at a time so that the real survey or test is seen as being composed of several instruments, each measuring a single characteristic (although the instruments may overlap in terms of the items). This intention, which is later termed the construct, is established by the person who designs and develops the instrument. This person is called the measurer throughout this book. The instrument, then, is seen as a logical argument that the results can be interpreted to help make a decision as the measurer intended them to be. The chapters that follow describe a series of steps that can be used as the basis for such an argument. First, the argument is constructive—that is, it proceeds by constructing the instrument following a certain logic (this occupies the contents of chaps. 2–5). Then the argument is reflective, proceeding by gathering information on whether the instrument did indeed function as planned (this occupies the contents of chaps. 6–8). The book concludes with a discussion of next steps that a measurer might take. This lays the groundwork for later books.

In this book, the concept being explored is more like a verb, measuring, than a noun, measurement. There is no claim that the procedures explored here are the only way to measure—there are other approaches that one can adopt (several are discussed in chaps. 6 and 9). The aim is not to survey all such ways to measure, but to lay out one particular approach that the author has found successful over the last two decades in teaching measurement to students at the University of California, Berkeley, and consulting with people who want to develop instruments in a wide variety of areas.

1.2 THE CONSTRUCT MAP

An instrument is always something secondary: There is always a purpose for which an instrument is needed and a context in which it is going to be used (i.e., involving some sort of decision). This precipitates an idea or a concept that is the theoretical object of our interest in the respondent. Consistent with current usage, I call this the construct (see Messick, 1989, for an exhaustive analysis). A construct could be part of a theoretical model of a person’s cognition—such as their understanding of a certain set of concepts or their attitude toward something—or it could be some other psychological variable such as “need for achievement” or a personality variable such as a bipolar diagnosis. It could be from the domain of educational achievement, or it could be a health-related construct such as “Quality of Life” or a sociological construct such as “rurality” or migrants’ degree of assimilation. It could relate to a group rather than an individual person, such as a work group or sports team, or an institution such as a workplace, or it could be biological phenomena such as a forest’s ability to spread in a new environment. It could even be a complex inanimate object such as a volcano’s proclivity to erupt or the weathering of paint samples. There is a multitude of theories—the important thing here is to have one that provides motivation and structure for the construct to be measured.

The idea of a construct map is a more precise concept than construct. We assume that the construct we wish to measure has a particularly simple form—it extends from one extreme to another, from high to low; small to large, positive to negative, or strong to weak. There may be some complexity in what happens in between these extremes, but we are primarily interested in where a respondent stands on this range from one extreme to the other. In particular, there may be distinguishable qualitative levels between the extremes—these are important and useful in interpretation. At this point, it is still an idea, latent rather than manifest. Although qualitative levels are definable, we assume that the respondents can be at any point in between—that is, the underlying construct is continuous. In summary, a construct map can be said to be a unidimensional latent variable. Many constructs are more complex than this. For example, they may be multidimensional. This is not a barrier to the use of the methods described in this book—the most straightforward thing to do is tackle each dimension one at a time—that way they can each be seen as a construct map. There are also constructs that are quite different from those that can be well described by a construct map. For example, suppose the construct consists of two different groups, say those who are likely to immigrate and those who are not. This construct is not much like that of a construct map and, hence, is not likely to be well represented by one.

In this chapter, the four building blocks are illustrated with a recent example from educati...

Table of contents

• Contents
• Preface
• Part I A Constructive Approach to Measurement
• Part II The Four Building Blocks
• Part III Quality Control Methods
• Part IV A Beginning Rather Than a Conclusion
• References
• Author Index
• Subject Index