Orwell Award Announcement SusanOhanian.Org Home

Schooling, Statistics, and Poverty:; Can We Measure School Improvement?

Susan Notes: Here's a link to a 42-page analysis of how school quality is assessed using either mean improvement of the school or longitudinal modeling (also known as value added). Raudenbush concludes methods based on mean proficiency are
scientifically indefensible, and although the longitudinal methods are better, they don't reveal school quality either.


Under No Child Left Behind legislation, schools are held accountable for making "adequate yearly progress." Presumably, a school progresses when its impact on students improves. Yet questions about impact are causal questions that are rarely framed explicitly in discussions of accountability. One causal question about school impact is of interest to parents: "Will my child learn more in School A or School B?" Such questions are different from questions of interest to district administrators: "Is the instructional program in School A better than that in School B?" Answering these two kinds of questions requires different kinds of evidence. In this paper, I consider these different notions of school impact, the corollary questions about school improvement, and the validity of causal inferences that can be derived from data available to school districts. I compare two competing approaches to measuring school quality and school improvement, the first based on school-mean proficiency, the second based on value added. Analyses of four data sets spanning elementary and high school years show that these two approaches produce pictures of school quality that are, at best, modestly convergent. Measures based on mean proficiency are shown to be scientifically indefensible for high-stakes decisions. In particular, they are biased against high-poverty schools during the elementary and high school years. The value-added approach, while illuminating, suffers inferential problems of its own. I conclude that measures of mean proficiency and value added, while providing potentially useful information to parents and educators, do not reveal direct evidence of the quality of school practice. To understand such quality requires several sources of evidence, with local test results augmented by expert judgment and a coherent national agenda for research and development in education.

— Stephen W. Raudenbush
William H. Angoff Memorial Lecture



This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.