Global Perspectives on Educational Testing

Keena Arbuthnot

• Publisher: Emerald Publishing Limited
• Year: 2017
• ISBN:
  9781787148963
• Topic: Bildung
• Subtopic: Bildungspolitik

This book provides a refined definition of standardized educational test fairness that can be utilized in multiple contexts to better understand the experiences and perspectives of diverse groups of test takers. Globally, there has been a significant influx in the use of and dependence on standardized tests to foster educational improvements. Standardized testing programs such as TIMSS (Trends in International Mathematics and Science Study), are designed to provide information about a country's global standing in multiple academic areas, ranking individuals and/or groups relative to the performance of others. The high-stakes nature of standardized tests has increased public concern and interest in issues related to test fairness. Since standardized test performance has a profound influence on multiple aspects of educational systems, it is imperative to better understand and examine issues of fairness.

CHAPTER 4

TAAF FRAMEWORK PHASE II: TEST AND SUBTEST PERFORMANCE

Chapter 3 focused on Phase I of the TAAF Framework, which included the selection of the countries to be included in the research and identification of country distinctions and similarities. This chapter covers the three steps included in Phase II of the TAAF Framework. The first step in Phase II is to conduct research that examines the differences in test and subtest (i.e., content domains and topic areas) performance for each of the four countries of Chinese Taipei, Finland, the United States, and Qatar, respectively. The second step examines the differences in test-wiseness skills among the four countries. The third and final step of Phase II focuses on understanding differences in the countries’ opportunities to learn the material covered on the TIMSS mathematics examination. Each study presents details about the measures that are used, as well as details about the appropriate analyses that are conducted. Additionally, the results from the studies are presented with discussion about the implications of the findings.

STUDY I: PERFORMANCE PATTERNS

The first study in Phase II of the TAAF Framework focuses on investigating the differences in performance patterns among the four countries of interest and includes an examination of the differences and the magnitude of the differences among the countries on the overall mathematics test as well as each of the content domains and topic areas. Details regarding the measures that were used in this study are defined as well as the analyses that were conducted. The results from this research are presented and discussed.

Measures

The TIMSS 2011 mathematics examination will be utilized in this study. The average scaled scores for the 4th and 8th grade TIMSS scales are used when reporting achievement levels for test takers. The scaled scores range in value from 0 to 1,000. The scale has a center point of 500 that is the mean of the overall achievement distribution. Additionally, every 100 point increase or decrease is equivalent to a 1 standard deviation change. Most of the scores are within 2 standard deviations of the mean.

The mathematics framework for the 2011 TIMSS mathematics assessment contains items that focus on two dimensions. First, items have a content dimension that reflects the domain (i.e., algebra or geometry) or topic area (i.e., fractions or whole numbers) that are assessed by TIMSS. The next is the cognitive dimension which focuses on the thinking processes that are used and assessed when taking the assessment, which shows a set of behaviors or types of strategies a test taker engages with on the mathematics assessment. The 4th and 8th grade content domains differ, because they reflect the different skills and difficulty that should be emphasized at each grade level. On the other hand, the cognitive domains are the same for the 4th and 8th grade test takers. Table 22 presents information about each content domain, and Table 23 presents details regarding the different cognitive domains for the 4th and 8th grade test takers.

In order to investigate the magnitude of the differences between countries, the released test items from the mathematics portion of the TIMSS 4th and 8th grade assessment from the 2011 administration will be used. The 4th grade mathematics section of the TIMSS assessment consisted of a total of 73 released test items, which included 36 multiple choice and 37 constructed response items. The 8th grade mathematics section of the TIMSS math assessment consisted of a total of 90 released test items, which included 48 multiple choice and 42 constructed response items. On constructed response items, test takers could receive either partial or full credit. However, for the purposes of this book, in order to dichotomously score the constructed response items, only those test takers who received full credit were coded as answering the item correctly. Consequently, test takers who received no credit or partial credit were coded as answering the item incorrectly. For the 4th grade examination, the items covered three content domains including the following: Data Display, Geometric Shapes & Measures and Numbers. The topic areas associated with each content domain are presented below.

Table 22. Definitions of Content Domains on the TIMSS Mathematics Test.

Domain	Definition
4th Grade
Number	Understanding of place value, ways of representing numbers, the relationships between numbers, number sense and computational fluency, the meanings of operations and how they relate to one another, and how to use numbers and operations
Geometric Shapes and Measures	Properties of geometrical figures, such as lengths of sides, sizes of angles, areas, and volumes; understanding how to identify and analyze the properties and characteristics of lines, angles, and a variety of geometric figures, including two- and three-dimensional shapes, and to provide explanations based on geometric relationships
Data Display	Reading and interpreting displays of data, understanding how to organize data and how to display it in graphs and charts to answer questions that prompted the data collection, and comparing characteristics of data and drawing conclusions based on data displays
8th Grade
Number	Understanding of numbers, ways of representing numbers, relationships among numbers, number systems, number sense and computational fluency, understand the meanings of operations and how they relate to one another, and be able to use numbers and operations to solve problems
Algebra	Recognizing and extending patterns, using algebraic symbols to represent mathematical situations, and developing fluency in producing equivalent expressions and solving linear equations
Geometry	Analyze the properties and characteristics of a variety of two- and three-dimensional geometric figures, including lengths of sides and sizes of angles, and to provide explanations based on geometric relationships; apply the Pythagorean Theorem to solve problems; using geometric properties and their relationships; geometric measurement, using measuring instruments accurately, estimating where appropriate, and selecting and using formulae for perimeters, areas, and volumes; understanding coordinate representations and using spatial visualization skills to move between two- and three-dimensional shapes and their representations, and using symmetry and applying transformation to analyze mathematical situations
Data and Chance	Knowing how to organize data that have been collected by oneself or others and how to display data in graphs and charts that will be useful in answering questions that prompted the data collection. This content domain includes understanding issues related to misinterpretation of data

Source: TIMSS data. Arbuthnot (2017).

Note: Definitions from Mullis, Martin, Ruddock, et al. (2009).

Table 23. Definitions of Cognitive Domains on the TIMSS Mathematics Test.

Domain	Definition
Knowing	Covers the facts, concepts, and procedures students need to know
Applying	Focuses on the ability of students to apply knowledge and conceptual understanding to solve problems or answer questions
Reasoning	Encompasses unfamiliar situations, complex contexts, and multistep problems

Source: TIMSS data. Arbuthnot (2017).

Note: Definitions from Mullis, Martin, Ruddock, et al. (2009).

Data Display

• Organizing and representing
• Reading and interpreting

Geometric Shapes and Measures

• Points, lines, and angles
• Two- and three-dimensional shapes

Number

• Fractions and decimals
• Number sentences with whole numbers
• Patterns and relationships
• Whole numbers

Similarly, for the 8th grade assessment, the following four content domains were covered: Algebra, Data and Chance, Geometry and Numbers. Each of the content domains includes two or more topic areas. The topic areas associated with each of the cognitive domains is presented below.

Algebra

• Algebraic expressions
• Equations, formulas, and functions
• Patterns

Data and Chance

• Chance
• Data interpretation
• Data organization

Geometry

• Geometric measurement
• Geometric shapes
• Location and movement

Numbers

• Fractions/decimals
• Integers
• Ratio/proportions
• Whole number

Although examining the overall test performance is important, it is also essential to examine the magnitude of the differences between all possible pairs of countries as well. This information provides more detailed information about the relative strengths and weaknesses of each of the countries.

Analysis

The data were analyzed to gain a foundational understanding of the performance patterns of each of the countries. First, an overview of the performance patterns for each of the countries is presented, including general performance scores for all of the TIMSS test administrations in which each country has been involved. The average scaled scores for each of the countries for each year they have participated in the TIMSS assessment is calculated and displayed. Additionally, a comparison of the average scaled scores for the 2011 administration of the TIMSS mathematics test is presented. Next, a detailed examination of the magnitude of the differences between and among each of the countries is analyzed. The p-value, or the proportion of test takers from each country that got each item correct, is utilized in order to investigate the magnitude of the differences between countries on each of the content domains and topic areas. Presented below are the analyses that were conducted in order to examine these differences.

Magnitude of the Differences between Countries

Step 1. Calculating Differences in p-Value. For each item, the difference in p-values between each pair of countries was calculated. The pairs of countries included: Chinese Taipei vs. Finland, Chinese Taipei vs. United States, Chinese Taipei vs. Qatar, Finland vs. United States, Finland vs. Qatar, and United States vs. Qatar. The value of the difference in p-values was reported for each pair of countries. For most pairs of countries there were both negative and positive differences in the p-value reported. For example, when analyzing the item-level performance of Chinese Taipei vs. Finland, one must subtract the Finnish p-v...

Table of contents

• Cover
• List of Tables
• International Assessments and Fairness Issues
• The Arbuthnot Assessment Fairness Framework
• TAAF Framework Phase I: Country Selection
• TAAF Framework Phase II: Test and Subtest Performance
• TAAF Framework Phase III: Item-Level Performance
• TAAF Framework Phase IV: Cultural Impact
• Rethinking Fairness and International Assessments
• References
• About the Author
• Index