Errors in Standardized Tests: A Systemic Problem
This paper documents a sizable collection of tsting errors--and the too-often absence of a resolution. As the authors point out, "No company can offer flawless products. Even highly reputable testing contractors that offer customers high-quality products and services produce tests that are susceptible to error. But while a patient dissatisfied with a diagnosis or treatment may seek a second or third opinion, for a child in a New York City school (and in dozens of other states and hundres of other cities and towns), there is only one opinion that counts--a single test score. If that is in error, a long time may elapse before the mistake is brought to light--if it ever is."
From the Introduction
[The heavy reliance of reformers on numbers--the quantification of performance--]is a facet of what Thomas Aquinas calls a "cultivated ignorance," ignorantia affectata. Garry Wills, writing in a different context, calls this ignorance "so useful that one protects it, keeps it from the light, in order to continue using it. . .this kind of ignorance [is] not exculpatory but inculpatory. . . a willed ignorance." Many proponents of high-stakes testing take a technological view: they choose to ignore the cumulative effects of test-based decisions, and view test takers as objects. Moreover, they ignore the fallibility of testing. Like any measurement tool that produces a number--whether a simple blood pressure reading or complex laboratory test--tests contain error. The widespread belief in their precision does not admit this inherent fallibility. . . .
This monograph is concerned with human errors, which differ from random measurement error in many ways. Human errors do not occur randomly; their presence is not known. These errors are of greater concern than random errors because they are capricious and bring with them unseen consequences. In contrast, measurement error is common to every test, and thus is expected: the amount of error is habitually calculated and disclosed, and therefore can be taken into account when interpreting test scores. . . .
Kathleen Rhodes & George Madaus
National Board on Educational Testing and Public Policy
INDEX OF RESEARCH THAT COUNTS