Orwell Award Announcement SusanOhanian.Org Home

Errors in Standardized Tests: A Systemic Problem

Susan Notes:
This paper documents a sizable collection of tsting errors--and the too-often absence of a resolution. As the authors point out, "No company can offer flawless products. Even highly reputable testing contractors that offer customers high-quality products and services produce tests that are susceptible to error. But while a patient dissatisfied with a diagnosis or treatment may seek a second or third opinion, for a child in a New York City school (and in dozens of other states and hundres of other cities and towns), there is only one opinion that counts--a single test score. If that is in error, a long time may elapse before the mistake is brought to light--if it ever is."

From the Introduction

[The heavy reliance of reformers on numbers--the quantification of performance--]is a facet of what Thomas Aquinas calls a "cultivated ignorance," ignorantia affectata. Garry Wills, writing in a different context, calls this ignorance "so useful that one protects it, keeps it from the light, in order to continue using it. . .this kind of ignorance [is] not exculpatory but inculpatory. . . a willed ignorance." Many proponents of high-stakes testing take a technological view: they choose to ignore the cumulative effects of test-based decisions, and view test takers as objects. Moreover, they ignore the fallibility of testing. Like any measurement tool that produces a number--whether a simple blood pressure reading or complex laboratory test--tests contain error. The widespread belief in their precision does not admit this inherent fallibility. . . .

This monograph is concerned with human errors, which differ from random measurement error in many ways. Human errors do not occur randomly; their presence is not known. These errors are of greater concern than random errors because they are capricious and bring with them unseen consequences. In contrast, measurement error is common to every test, and thus is expected: the amount of error is habitually calculated and disclosed, and therefore can be taken into account when interpreting test scores. . . .

— Kathleen Rhodes & George Madaus
National Board on Educational Testing and Public Policy
May 2003


This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.