The Ambiguity of Teaching to the Test

Standards, Assessment, and Educational Reform

William A. Firestone, Roberta Y. Schorr, Lora F. Monfils

• Publisher: Routledge
• Year: 2004
• ISBN:
  9781135624200
• Edition: 1
• Topic: Éducation
• Subtopic: Éducation générale

Testing is one of the most controversial of all state and federal educational policies. The effects of testing are quite ambiguous. The same test may lead to different consequences in different circumstances, and teachers may use very different strategies to prepare students for tests. Although most experts agree that mandatory testing leads to teaching to the test, they disagree about whether it leads to meaningless drill, wasted time, de-professionalizing teachers, and demotivating students, or to more challenging and thoughtful curricula, more engaging teaching, increased student motivation, and increased accountability. To help sort through this ambiguity and provide a firmer basis for decisions, The Ambiguity of Teaching to the Test: Standards, Assessment, and Educational Reform offers a hard look at the effects of state testing, and thoroughly examines the ambiguity of test preparation and how test preparation practices are influenced by what teachers know and the leadership coming from the school and district. Drawing on data from a three-year study of New Jersey's testing policy in elementary mathematics and science, it helps to explain the variety of ways that teachers modify their teaching in response to state tests, raises important questions, and offers useful guidance on how state policymakers and local and district school administrators can implement policies that will improve educational equity and performance for all students. It also offers an in-depth analysis of classroom practices that should inform teachers and teacher educators whose goal is to meaningfully implement conceptually based teaching practices.This comprehensive look at the statewide variation in testing practice features:
*a data-based, non-ideological treatment of how testing affects teachers, in a field characterized by ideologically driven beliefs and by anecdotes;
*an extensive and well-integrated combination of qualitative and quantitative data sources that provide a statewide overview, as well as an in-depth analysis of teachers and classrooms;
*a careful analysis of the variety of forms of teaching to the test; and
*a multilevel exploration of how a variety of personal and leadership factors can influence teaching to the test.This is an important book for researchers, professionals, and students in educational testing, educational policy, educational administration, mathematics and science education, educational reform, and the politics and sociology of education. It will also prove useful for state policymakers, school and district leaders, and teacher educators and curriculum specialists who are making decisions about how to design and respond to new testing systems.

1
Introduction

William A. Firestone

Roberta Y. Schorr

One day in late spring in a classroom in New Jersey, Ms. Jones put the following problem on the overhead projector along with four possible answers: 56 × 24. She then told her children to turn off their calculators and said, “Let’s just look at the strategy here. You won’t have the calculator for this part. You’ll get paper and a pencil. You will have a choice on day of the test whether to have lined or unlined paper. I like lines ’cause I like things orderly. So knowing that, let’s look at what we have to multiply. We have to multiply by the ones digit. Which one is the ones digit?” The children then identified the six and the four as the being in the ones place and multiplied them together to get 24. At that point, she told the children they could rule out two answers because they didn’t end in four. She then passed out paper and had the children follow the conventional steps for multiplying 56 by 24. She then called on one student, Lindsey, who worked aloud on the problem, identifying the steps needed in order to get the solution.

At about the same time in another district, Ms. Smith passed out a bag of M&Ms to each student and asked each one to make a bar graph representing the number of candies of each color in the bag. The students spent about 30 minutes working with partners to sort and count the candies, construct graphs, and consider how they might begin to analyze their own data. Then Ms. Smith asked the students what statements they could make based on the data as displayed in their graphs. When the students reported to the class, one student, Michael, had calculated what turned out to be several different measures of central tendency for the number of M&Ms of each color. For instance, he had found that the mean number of candies of any given color was seven. He also shared that he had calculated what he called “the range” by finding the difference between the predicted total number of candies and the actual total number in his bag. Ms. Smith used this opportunity to build upon Michael’s idea, as well as to discuss what the standard mathematical meaning for “the range” is, and how it should be calculated. Other students described the percentage of candies that were particular colors and then used the percentages to make predictions about the likelihood of pulling out an M&M of a particular color from their bags.

Ms. Jones’ and Ms. Smith’s classrooms illustrate some of the issues surrounding “teaching to the test.” This is a term with many meanings, but the implication is, generally speaking, that teachers are doing something special to help students do well on a test, often without helping them to better understand the underlying subject matter. Teaching to the test is decried by testing experts because it artificially inflates test scores so that the measure (the test score) and the construct measured (the child’s knowledge of the subject tested) become progressively disconnected. It is abhorred by curriculum experts who believe that testing reduces the quality and quantity of what is taught as commercial test-prep materials are substituted for conventional materials. Students spend too much time learning techniques that help them do nothing more than cope with specific test formats (McNeil, 2000). Those concerned about equity point out that the worst teaching to the test often takes place in the nation’s urban schools, where high-quality, intellectually challenging teaching is most needed.

The vignette involving Mrs. Jones illustrates some of these fears about teaching to the test. She was preparing her students by using examples of problems that her math supervisor had downloaded from the state Web site because they were similar to those on the state test. This lesson was a preparation activity for the test, which was to be given in just a few weeks. Ms. Jones indicated that she felt that it was important to help children learn how to quickly eliminate incorrect answers for multiple-choice items in order to save time. Her teaching represents what we have come to call decontextualized test preparation because it is a special activity only loosely related to her regular lessons and focused on the test itself.

A more optimistic view of teaching to the test starts from the idea that American education is not sufficiently challenging for any of our students. The curriculum often emphasizes basic facts and skills, and new topics more relevant to the 21st century are sometimes not introduced until later on, if at all. Teaching is all too often organized in a way that treats students as passive receptacles to be filled with knowledge. Teachers drill children on conventional facts and algorithms without seeking to generate any intellectual interest or creative thought or understanding of the algorithms. Often, these tendencies are reinforced by tests that require factual information that can be easily recalled and tested and that fail to tap higher order skills.

In this view, new tests are a way to challenge the status quo and raise the instructional bar. If more thought-provoking tests are given, teachers are expected to rise to the occasion, and improve instruction. If the tests cover more relevant content, the curriculum can be updated. Often, those most identified with the reform movements want to use more authentic or performance- based assessments where children are asked to undertake activities that not only require them to repeat what they have learned but to use that information to construct something that shows that they have a deeper understanding of the subject tested and that they can connect what they know to other problems, including real-world situations. These performancebased assessments, it is argued, may provide more valid measures regarding genuine knowledge of a discipline. They also challenge teachers to teach in more intellectually engaging ways.

Ms. Smith’s teaching more closely reflects this view. The content involved graphing, statistics, and probability, topics that had not been prominent in the elementary curriculum before New Jersey introduced its new standards and tests. She also assigned more open-ended activities like making and reading graphs and drawing conclusions from them. Graphical representations, such as those generated by Ms. Smith’s students, do indeed often appear on the state test, and students are often asked to make conjectures and write conclusions about similar problems. Though Ms. Smith did not specifically refer to the test, she did feel that doing such activities could prepare students for it, but even more significantly, could help them to understand important content. Ms. Smith represents “embedded” test preparation because activities to prepare students for the test are integrated into her regular instruction, not a special, decontextualized activity.

Critics fear that state tests will reinforce rote teaching methods. Ms. Jones drilled children on the conventional algorithm for multiplying two-digit numbers by hand using a very didactic approach. She felt that the best way to help children prepare for the test was to help them memorize facts. In fact, earlier that period, when a child had forgotten a multiplication fact, she said “I’m a lion. I like to roar. Six times nine is …” The girl said, “fifty-four.” Advocates of reformed assessment hope that new items will lead to more engaging assignments. Ms. Smith gave a much more open-ended, inquiry-oriented task where children could determine what conclusions to draw, and Ms. Smith used their responses to stimulate the development of important mathematical ideas. She felt that it was important for children to use multiple representations. She therefore had them draw graphs, write number sentences, and talk about their mathematical thinking. This strategy is consistent with both National Council of Teachers of Mathematics (NCTM) standards and advances in cognitive science that suggest that understanding comes from building connections between various ideas, facts, representations, and operations (Hiebert & Carpenter, 1992; NCTM, 2000).

Ms. Jones and Ms. Smith illustrate that there are many forms of teaching to the test and many ways that state tests can influence teaching. But single cases can only illustrate possibilities. With so many states testing children in different ways, with different consequences for passing and failing those tests, it is important to understand (a) how teachers respond to different kinds of tests, (b) how other state education policies affect those responses, and (c) what local educators do to ensure that tests will improve instruction, not deaden it.

To explore these issues, we conducted a 3-year study of state testing in New Jersey. Our findings illustrate some of the problems with state tests, but they also suggest some of the ways that tests can be combined with other actions to improve teaching and learning. Our work confirms past studies showing that most elementary math and science teaching follows conventional, less challenging instructional models. However, we also describe the kinds of changes in practice—often just “add-ons” to conventional instruction— that teachers are adopting. We show that teaching consists of a variety of practices and that some are more intellectually challenging than others. The New Jersey story shows that tests with open-ended assessment items encourage more challenging forms of teaching to the test and of instruction in general, but that changing tests alone is not enough to bring about such changes. With the need for additional measures to change instruction in mind, we explore how school administrators support or undermine more constructive versions of teaching to the test through principal leadership and district professional development. Finally, we clarify how standards-based testing can help or harm educational equity by examining how student characteristics and a variety of policy variables affect achievement on the new tests over their 3-year history.

STATE TESTING AND THE STANDARDS MOVEMENT

Although New York’s Regents tests go back to the 19th century, most people date the beginning of modern state testing with the passage of Florida’s Education Accountability Act in 1971. Eleven years later, 36 states mandated some kind of testing program (Odden & Dougherty, 1982); by 2000, almost every state tested students at one or more grade levels, and testing was spreading from the most central subjects like language arts and mathematics to science and social studies (Editorial Projects in Education, 2001).

Several factors promoted this growth in state testing. One was the expansion of state government during the 1960s and 1970s. State departments of education grew more than threefold between 1957 and 1986 and the policy staff serving state legislatures more than doubled between 1968 and 1979. These larger agencies had more capacity to think about, monitor, and govern education (Firestone, 1990).

Another was the change in educational financing. Constitutionally, education is a state function, but it had been locally governed and funded throughout the 19th century with little state oversight. The state share of school funding increased gradually after World War I through the 1980s.

Since then, the proportions have been fairly consistent with state and local governments each contributing 42% with the balance coming from the federal government (National Center for Education Statistics, 2000a). As states paid more of the costs of schooling, they expected local districts to be more accountable for how these funds were spent. This was especially true when increased state funding was mandated by courts to equalize funding in rich and poor school districts.

A third reason was the concern throughout the postwar period about inadequate educational performance. An important milestone was publicity about dramatic declines in Scholastic Achievement Test (SAT) scores between 1963 and 1977. Another was the publication of A Nation at Risk (National Commission on Excellence in Education, 1983) and a spate of other commission reports in 1983, all decrying the poor achievement of American schools and offering a variety of remedies. Though some have challenged this overwhelmingly bleak view of the effectiveness of American schools (Berliner & Biddle, 1995), the consensus was that American schools were performing poorly, that something had to be done, and that a great deal of responsibility for leading the improvement effort rested with state government.

Meanwhile, state government began borrowing business ideas about efficient management. The first efforts included planning, programming, budgeting systems, and management by objectives. When these strategies did not work well in schools (Wolcott, 1977), states and business groups refined their approaches. They consistently pushed for specifying goals, measuring outputs to ascertain whether the government was spending money wisely, and holding educators and students accountable for their performance. The first recommendation of a recent report from the National Alliance of Business (2000), for instance, was to “enact a wide range of measures that give schools, educators, and students the strongest possible incentives to perform at high levels” (p. 1) and to link those incentives to “high, clearly defined, and publicly supported academic standards” (p. 1).

Over time, these business-based ideas, often refined by economists and political scientists, became the basis for current thinking about how the state should develop standards of educational performance and use them to design tests to hold schools and districts accountable. In rough outline, a standards-based accountability system was likely to include four elements:

• Content standards that set out the knowledge and skills children are expected to learn.
• Tests or assessments to measure those content standards.
• Student performance standards that define proficient performance in terms of the official assessments.
• Rewards provided to students or schools that meet or exceed the standards and punishments or remediation activities for those that do not.

A strong system would have all four elements. The theory of action behind such a system is that the formal sanctions linked to meeting standards motivate educators and students to learn what is tested. A weak system would certainly not have the last two elements and might not have the first—many states began testing without any guidance from standards. The theory then is that the publication of test scores will motivate improvement by appealing either to professional pride or indirectly to the public which will use political pressure to promote improvement (Firestone, 2003).

Meanwhile, the federal government was influenced by the same ideas about efficient management that were affecting thought at the state level. These ideas progressively influenced the main federal legislation affecting schools, the Elementary and Secondary Education Act (ESEA). Early requirements that ESEA be evaluated influenced the use of testing, stimulating school districts to test more (Linn, 2000). Revisions in 1994 required states to establish content standards, to implement assessments that assessed students against those standards by testing students in mathematics and reading at least once each in Grades 3–8, and to hold schools accountable for the achievement of all students (Goertz, 2002).

Led by governors and state legislators, state governments adopted and elaborated standards and accountability systems in the 1990s. In addition, they defined state standards, developed and refined tests, and considered the alignment between the two. Slowly, they moved toward stronger accountability systems by linking consequences to meeting (or failing to meet) performance standards. At the same time, along with this growing focus on tested outcomes, they de-emphasized, without totally removing, older forms of accountability (Adams & Kirst, 1999).

Every indication is that this movement to increase accountability will continue in the coming years. For instance, the reauthorization of the Elementary and Secondary Education Act of 2001, now referred to as “No Child Left Behind,” mandates that states increase their testing requirements substantially beyond what was required by earlier legislation. Specifically, states are required to conduct statewide assessments in reading and mathematics in every grade from third through eighth and assess students in science at least once in the elementary, middle, and high school years. Each child must complete the full assessment—no matrix sampling of items is allowed—and results must be broken out by race, income, and other categories. For schools that do not achieve proficiency or make adequate yearly progress toward proficiency,¹ penalties include a parental option to move their children to a different school. These requirements will increase the sheer quantity of state testing because very few states tested in all the grades specified by ESEA at the time the legislation was passed (Robelen, 2002).

DEFINING STANDARDS IN MATH AND SCIENCE

The movement to increase state testing and accountability comes largely from outside of education. Though it lauds high standards, it says relatively little about what those standards should be. In math and science especially, most of the work in defining standards has been led by subject matter specialists in government, schools, and universities.

This movement is in part a reaction to the remarkable sameness of American teaching in schools over the last hundred or more years. Cuban (1993) described the consistent pattern of teacher-centered teaching in which adults dominate whole-class discussion, make assignments, rely on textbooks, and treat knowledge as relatively fixed, and as something to be given to students. At all grade levels, but most particularly in the elementary grades, there have been some modifications in the way content is taught. For example, our research shows that more teachers are using small-group instruction, at least some of the time. However, the basic pattern of teacher-as-authority telling children is extraordinarily constant.

This pattern limits the intellectual activity of the students and the content that is actually understood. American math teachers, for instance, tend to focus on definitions and procedures with limited exploration of mathematical ideas, discussion involving proof and justification, or use of multiple media to help students see the connections between different representations of an idea (Stigler & Hiebert, 1999). Mrs. Smith’s use of M&Ms to analyze data is not typical of classroom activity. Many teachers may do the same or similar activities, but do so in a relatively traditional manner where children have little or no opportunity to explore ideas in a meaningful way. Moreover, Americans break topics into small, discrete units, which may make sense to an experienced mathematician, or educator, but works against helping students to understand the content or see the interconnections among mathematical ideas (W. H. Schmidt, McKnight, & Raizen, 1996). In science, topics are also broken into small units. Elementary students have few opportunities to design their own research problems, use scientific materials to see how ideas work concretely, or conduct any form of investigation—or in short, think like a scientist (Razze, 2001).

Each content area in the American elementary curriculum includes a wide range of topics. In science there is very little focus until high school. In mathematics, many topics are taught superficially one year, repeated the next, and again the following year. Although the intent may be to make the content increasingly complex, this rarely happens. The multitude of topics that are covered results in superficial attention to any one “big idea.” Worse yet, the focus is still primarily on numerical operations and procedures in a way that encourages facility with algorithms and operations without deep understanding (W. H. Schmidt et al., 1996). The NCTM suggests that although students should have fluency with algorithms and procedures, they must also learn mathematics with understanding and actively build knowledge from experience and prior knowledge (NCTM, 2000).

Attempts to reform American mathematics and science in the postwar world began with curriculum. This effort started with the introduction of the “new math” and science curricula between the mid-1950s and the mid- 1970s. These programs were intended to modernize the content covered, provide a variety of instructional media beyond books, and get children actively involved with scientific experiments. Though several curricula were distributed widely, the consensus is that classroom practice changed very little because of these programs (Welch, 1979).

Another effort began in the 1990s. The National Science Foundation again invested in developing research-based, field-tested materials such as Everyday Mathematics (University of Chicago), Investigations (Technology Education Research Center), Mathematics in Context (University of Wisconsin), and Connected Mathematics (Michigan State University). The intention of these programs was to provide materials that would introduce content in a more unified, coordinated manner. Like the earlier materials, these were very different from traditional conventional book series, and demanded greater content and pedagogic competency from teachers. Like their predecessors, these programs had some success in changing the content taught. In addition, commercial textbooks also began to incorporate these ideas on a broader scale.

In part because of the limitations of curriculum writing as a reform strategy, but also because of a growing interest in defining and mandating educational standards, professional associations began providing statements describing what such standards should include. The NCTM standards, which defined mathematical power as the “abilities to explore, conjectu...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• Preface
• 1 Introduction
• 2 Teaching Mathematics and Science
• 3 Teaching to the Test
• 4 Pressure and Support
• 5 The Principal, Test Preparation, and Educational Reform
• 6 The District and Test Preparation
• 7 Test Scores and Equity
• 8 Conclusion
• Appendix A: Methods
• Appendix B: Teacher Questionnaire
• Appendix C: Principal Interview Guide
• Appendix D: Teacher Interview Guide
• Appendix E: Interview Coding Scheme
• Appendix F: List of Codes for Classroom Observation Data
• References