Every paper in the country should receive a letter with this conclusion offered by Ross and Mathison: The judgment about quality (or lack of it) in schools must be based on a critical, nuanced, multi-faceted analysis, not on single fallible indicators. And especially not on careless, unthoughtful reporting of these fallible indicators by the media. Just as schools should be accountable for their work, so too must the media be accountable for its work.]
Submitted to Louisville Courier-Journal but not published
We applaud the The Courier-Journal for its concern for the improvement of education in Kentucky. A literate and well-educated public is the foundation for an improved quality of life in the Commonwealth and absolutely necessary for a fully functioning democracy.
Our concern is that your support for educational reform has crossed the line into uncritical boosterism of policies that are harmful to children; that produce a diminished education; and that, despite their broad acceptance, are based on unsound principles and constitute educational malpractice. (See the statement of the American Evaluation Association Task Force on High Stakes Testing for a summary of how standardized testing is misused and links to the large and growing body of research, much of which illustrates the deleterious effects of high stakes testing, www.eval.org).
In your October 10 editorial ("KERA Restoring Hope"), you say "it is not possible to overstate the numbers" illustrating statewide gains on CATS and other standardized tests the past five years. In fact, it is possible to overstate the numbers and we contend that when educators, the media, and the public become fixated on "the numbers" that teaching and learning actually suffer. This is because the numbers are products of fallible instruments, which are laced with error and reflect (and serve to perpetuate) the inequities of race, gender, and wealth that mark our society.
In this same editorial you make a claim that is, in our judgment, irresponsible journalism--a claim that illustrates how an uncritical faith in an assessment system can turn reality on its head. Specifically, the editorial states:
"In fact, a scatter chart showing the distribution of Kentucky elementary schools by poverty and achievement shows no big correlation. We have poor and wealthy schools performing at high levels. We have poor and wealthy schools performing at low levels. There is no concentration of wealthy schools among the high performing."
This is an extremely strong statement, one that contradicts the findings of research on the relationship of poverty and achievement as measured by standardized tests. Unfortunately your claim is not supported by reference to evidence other than a "scatter chart" of unknown origin.
We examined the CATS data for Jefferson County Public Schools published in the October 8 edition of The Courier-Journal and calculated the correlation between individual schools' scores and the poverty level of their students (represented by the percentage of students receiving free and reduced lunch as reported in JCPS data books: www.jefferson.k12.ky.us/databook/index.htm). The correlations between wealth and CATS scores across the district at all levels are very high.
JCPS elementary schools (r = .72)
JCPS middle schools (r = .89)
JCPS high schools (r = .91)
Between 52% to 82% of the difference in CATS scores in JCPS can be explained by wealth. These findings are consistent with a recent study examining the impact of sociodemographic variables on Kentucky student achievement [Munoz M. A., & Dossett, D. (2001). Equity and excellence: The effect of school and sociodemographic variables on student achievement. Journal of School Leadership, 11(2), 120-134.] Munoz and Dossett's conclusions are consistent with earlier studies that have found that student poverty level remains the best predictor of academic achievement.
These findings do not mean that poor children are not as smart as rich children! Rather, what the repeatedly confirmed relationship between wealth and academic achievement indicates is that these tests are best at measuring the income level of a students' household.
The latest CATS scores, as reported in The Courier-Journal, indicate there are a small number of high poverty elementary schools with very high performance on the CATS. More, specifically there are three high-poverty elementary schools (e.g., free and reduced lunch rates of 50% or higher) in your list of "top ten" performing elementary schools in Kentucky. However, every school listed as a "bottom 10" performer by your paper (at the elementary, middle, and high school levels across the state) is a high poverty school, with free and reduced lunch rates of 47% to 97% (based on data from the JCPS data books and the Kentucky Department of Education's "MAX The Data Enterprise System").
The average free and reduced lunch rates of the reported "top ten" and "bottom ten" schools in Kentucky were:
"Top Ten" Schools% Students with Free & Reduced Lunch:
"Bottom Ten" Schools % Students with Free & Reduced Lunch:
Middle Schools »71.50%
The correlations between wealth and CATS scores for the reported top and bottom tens are:
Elementary schools (r = .81)
Middle schools (r = .71)
High schools (r = .93)
Your editorial also claims the gains on CATS, CTBS, etc are "real gains." We don't dispute the numbers per se, only what the numbers truly mean. The questions to ask are: What accounts for the distinctive performance of the three high-poverty schools in the top ten? Are these scores accurate reflections of authentic increased achievement? Are these scores an artifact of instruction narrowly aimed at improving CATS scores?
We don't presume to know the answers to these questions about the schools in question. What we believe the public deserves, however, is responsible journalism that investigates the meaning of test score increases (or decreases) as well as the secondary effects of education reform efforts, like KERA. For example, while you uncritically sing the praises of test score increases, recent research indicates the high school graduation rate in the Commonwealth has steadily dropped from 70.3% in 1994-95 to 64.2% in 2000-01 (see state by state comparisons of graduation rates in the September 2003 testimony of Dr. Walter Haney of the Center for the Study of Testing, Evaluation and Educational Policy at Boston College to the New York State Senate Education Committee on "Attrition of Students from New York Schools": www.timeoutfromtesting.org/testimonies_923.php
Testing in America's schools is an effort to treat teaching and learning in a simple and fair manner, but in a world where education is hugely complex with inequitable distribution of opportunity. The instruments and technology used for educational assessment have not and cannot live up to the expectations placed on them by politicians, policy makers, the media, or the public. The judgment about quality (or lack of it) in schools must be based on a critical, nuanced, multi-faceted analysis, not on single fallible indicators. And especially not on careless, unthoughtful reporting of these fallible indicators by the media. Just as schools should be accountable for their work, so too must the media be accountable for its work.
E. Wayne Ross & Sandra Mathison