Schooling, Statistics, and Poverty:; Can We Measure School Improvement?
Susan Notes: Here's a link to a 42-page analysis of how school quality is assessed using either mean improvement of the school or longitudinal modeling (also known as value added). Raudenbush concludes methods based on mean proficiency are
scientifically indefensible, and although the longitudinal methods are better, they don't reveal school quality either.
Under No Child Left Behind legislation, schools are held accountable for making "adequate yearly progress." Presumably, a school progresses when its impact on students improves. Yet questions about impact are causal questions that are rarely framed explicitly in discussions of accountability. One causal question about school impact is of interest to parents: "Will my child learn more in School A or School B?" Such questions are different from questions of interest to district administrators: "Is the instructional program in School A better than that in School B?" Answering these two kinds of questions requires different kinds of evidence. In this paper, I consider these different notions of school impact, the corollary questions about school improvement, and the validity of causal inferences that can be derived from data available to school districts. I compare two competing approaches to measuring school quality and school improvement, the first based on school-mean proficiency, the second based on value added. Analyses of four data sets spanning elementary and high school years show that these two approaches produce pictures of school quality that are, at best, modestly convergent. Measures based on mean proficiency are shown to be scientifically indefensible for high-stakes decisions. In particular, they are biased against high-poverty schools during the elementary and high school years. The value-added approach, while illuminating, suffers inferential problems of its own. I conclude that measures of mean proficiency and value added, while providing potentially useful information to parents and educators, do not reveal direct evidence of the quality of school practice. To understand such quality requires several sources of evidence, with local test results augmented by expert judgment and a coherent national agenda for research and development in education.
Stephen W. Raudenbush
William H. Angoff Memorial Lecture
INDEX OF RESEARCH THAT COUNTS