Challenge Index Challenged
Ohanian Comment: Rotherham and Mead want Newsweek to use more sophisticated measures to select the country's best high schools. Who is served by such a list? I invite readers to comment on the damage done by this spurious system of rating "America's Best High Schools."
By Jay Mathews
Newsweek magazine plans to publish in a couple of months another edition of its "America's Best High Schools" list, based on a rating system called the Challenge Index I. It is a very odd and narrow measuring device, conceived a decade ago, and even though I think it is the best single statistical measure of public high schools, it takes a while to understand why that is. Thus the most common criticisms of it can be summed up in one word: "Huh?"
That is changing, as more people become accustomed to the idea of ranking schools based on participation in college-level tests and more experts judge the Newsweek list -- and a similar list of Washington-area schools in The Washington Post -- on what the Challenge Index tries to measure, rather the many important things about high schools that nobody can yet measure adequately, such as teacher quality and student motivation.
The most useful contribution to this debate so far is a 19-page article posted last month on www.educationsector.org by Andrew J. Rotherham, co-founder and co-director of the new Washington-based independent education think tank, Education Sector, and Sara Mead, the organization's senior policy analyst. Rotherham and Mead conclude that the 100 top schools identified by Newsweek cannot be accurately called the nation's best because many of them have very high drop-out rates and large gaps in test scores between white and minority students. They suggest Newsweek change the title of the list and use some of the test-score and drop-out data coming out of the No Child Left Behind Act to rate schools in different and more complex ways.
What I like best about the Education Sector piece, "Challenged Index: Why Newsweek's List of America's 100 Best High Schools Doesn't Make The Grade," is that it discusses other ways to rate high schools that may help improve learning for all students. The Challenge Index was born out of my frustration with the widespread notion that the best high schools are the ones with the fewest low-income and minority students, and the worst are theones with the most students in those categories. I sense Rotherham and Mead agree with me that such an assessment is not right, and that schools should be judged not on their average family-incomes but on how firmly and successfully each school works to raise achievement. Rotherham says he and Michael Goldstein, founder of the Media and Technology Charter High School, MATCH, in Boston, are working on their own top 100 list using the ideas in his article, and I look forward to that.
But I don't agree with the changes Rotherham and Mead suggest Newsweek make in the way it rates high schools because I think, despite the authors' good intentions, their ideas will lead to a list that recognizes only those schools in the nicest neighborhoods with the greenest lawns, the most educated parents and the fewest disadvantaged kids.
The Challenge Index rates schools by a very simple formula. It takes the number of Advanced Placement and International Baccalaureate tests taken by all students in May and divides that total by the number of seniors graduating from the school in June. The Newsweek list is actually much larger than the 100 schools that are ranked in the magazine. Last year, on the magazine's and The Post's Web sites the Newsweek researchers and I identified 1,069 schools that achieved a rating of at least 1.000, including all the public schools that had at least as many AP or IB tests as graduating seniors.
Rotherham and Mead acknowledge in their article, studded with charts, that "taking AP and IB tests is one key indicator of a good high school," but say "we believe that the Challenge Index is a seriously flawed measure of overall quality."
"A successful high school should show high levels of student achievement, graduate almost all of its students and not let any demographic subgroup suffer at the expense of others," they say. "Most national and local experts and policymakers share these values. To be sure, graduation rates and student achievement are hardly the only indicators of a school's quality. At a minimum, however, America's best high schools should be expected to meet these basic criteria.
"Yet our analysis shows that many schools on Newsweek's list do not meet these minimum standards. Using publicly available student performance data, we found that many schools on Newsweek's 2005 ranking have glaring achievement gaps and high dropout rates. By presenting them as America's best, Newsweek is misleading readers and slighting other schools that may in fact be better than those on Mathews' list. For example, the magazine ranks Eastside High School in Gainesville, Fla., as the third best high school in the nation, but only 12 percent of Eastside's black students were reading at grade-level in 2004. And Newsweek ranks Hillsborough High School in Tampa, Fla., as America's 10th best high school, but only 17 percent of black students and 26 percent of Hispanic students met the state's modest grade-level standards in 2004. While some students at Eastside or Hillsborough may be receiving a challenging education, it's clear that many are not. And Eastside and Hillsborough are not outliers. In fact, schools with substantial inequities in student achievement make up a significant proportion of Newsweek's list of best high schools."
What Newsweek should do instead, Rotherham and Mead say, is broaden its measuring system to include much more data, not just AP and IB participation rates. The magazine's editors, they say, should use "disaggregated data and graduation rates in their rankings and reward schools that do well with students from diverse backgrounds. As state data systems become more developed, information about course-taking patterns and other measures of college preparation -- as well as longitudinal data about performance -- could also be included."
Rotherham and Mead kindly showed me an advance copy of their article and included my lengthy response on their Web site. My principal complaint was that they were ruling out any recognition in the Newsweek list of several inner city schools that have made substantial improvements in the way they prepare students for college, but have yet to solve the achievement gap and drop-out problems. This is especially galling to me, I said, because one of the schools they would ban from the list with their new criteria is Garfield High in East Los Angeles, where I first learned that low-income students could achieve at a very high level if given enough time and encouragement to do so.
"There are a few inner city schools like Garfield that have produced unusually encouraging and resourceful teachers and impressive AP and IB participation rates," I said in my response to Rotherham and Mead, "but because the vast majority of their students are low-income, they have not made much progress yet on the dropout and achievement gap problems you properly identify. If I knew of any inner city public school with a majority of low-income students that had a significantly lower dropout rate and achievement gap than other schools with similar demographics, I would write about it and then follow your suggestion and look for a way to measure its achievement and rank other schools accordingly.
"But I have not found such a school, and I think that the dropout and achievement gap factors are so closely tied to average family income that no inner city school would ever get close to the top 100 list of schools that did the best by these measures. Indeed, that would not be a measure of how good the schools were but how well-off their students' parents were. When we find a way that inner city schools can significantly reduce their dropout rates, as Garfield significantly increased its AP participation rate, then we have something worth measuring. Until then, you have a measure with no point. Low-income schools will lose your game every time."
Here are a few more points made by Rotherham and Mead, and my quick reactions:
The Challenge Index is wrong to make the number of graduating seniors such an important part of the calculation, since that means schools with high dropout rates will look better than they would otherwise.
That is exactly right, and it was one of the reasons why I wanted to use the number of graduating seniors as a measure of relative school size. I don't think there is anything the staff of an inner-city school can do at the moment with the tools available to them to increase substantially their number of graduates, with the possible exception of ignoring graduation standards, which I don't think anyone would favor. Given all their other disadvantages when being compared to suburban schools, I thought it fair to allow inner city educators to measure their AP and IB participation rates against that smaller core of students devoted enough to their studies to graduate, rather than the larger number of nominally enrolled students who just show up occasionally to see their friends.
A school with a small number of students taking many AP and IB tests can look good on the Challenge Index, even though most students are not involved in the program.
I have discovered that such small cabals of frantic college test-takers almost never do their schools much good on the list. With very few exceptions, the schools that do best on the Newsweek list are those that try hardest to involve as many students as possible in AP and IB. Most high schools let only relatively few of their students take AP or IB courses and tests, which is the major reason why the 1,069 schools that made last year's Newsweek list represent only about 4 percent of U.S. public high schools.
Newsweek should not use the word "best" in describing these schools, because so many of them have high dropout rates and large achievement gaps.
Best has become a very elastic term in America. Your list of best film directors may rely on their Oscar totals; mine may depend on their box office receipts. This inspires useful arguments, which is what Newsweek's list is doing for high schools. A list of the top schools using the criteria Rotherham and Mead suggest is going to be a list of the most affluent high schools. I don't think that is as good a use of the word "best" as a list of which schools are trying hardest to get as many students as possible ready for college by taking college-level exams, since the research shows that students who do well on those exams are more likely to graduate from college.
Newsweek should use more sophisticated measures.
The more sophisticated the measures, the more difficult it will be for readers to understand what the magazine is doing, and the more difficult it will be to make valid comparisons between individual schools, especially when they are in different states. Rotherham and Mead suggest -- somewhat apologetically, because they are not dumb -- that Newsweek might try to use some of the data that the states use to determine which schools have achieved adequate yearly progress under No Child Left Behind. The sad truth, however, is that these data points and the state decisions themselves are often beyond reader comprehension, and can be very subjective. Each state uses different tests and different standards of proficiency, so schools in Texas cannot be ranked against schools in Illinois. Rotherham and Mead also acknowledge that the dropout statistics we have at the moment are a mess.
Doing it their way would also eliminate one of the Challenge Index's great strengths -- it is so simple readers can understand it and do the arithmetic for their own neighborhood high schools. Rotherham and Mead seem to prefer the method U.S. News & World Report uses in its college rankings -- a complex assortment of several different data points weighted in ways whose validity only experts can judge. Rotherham and Mead are experts, and so they are comfortable with that, but the editors of Newsweek and I are journalists who think our job is to serve readers with as much clarity as possible.
Rotherham is one of the most active, influential and knowledgeable education experts in the country. I often quote him in my stories. He served in the Clinton White House as an education adviser and now sits on the Virginia Board of Education. If Mark R. Warner, the former Virginia governor who gave Rotherham that appointment, succeeds in his quest for the White House, I would not be surprised to see him make Rotherham U.S. secretary of education. Mead, who crunched most of the numbers in their report, is also a fine researcher who did great work with a very complex subject.
I am flattered two such people took the time and effort to critique the Challenge Index with such care, and I hope they won't stop there. One point they make repeatedly is that the Newsweek list, because of its focus on AP and IB, is overlooking similar schools that are doing a better job than those on the list in reducing dropout rates and closing achievement gaps.
Here is my request. Please find those schools, and if you do, tell me which ones they are. I don't believe they exist, but if they do, it will be an important discovery and well worth a few more columns. I don't think Rotherham and Mead will be able to find even one open-enrollment public high school in the United States that is not already on the full Newsweek list and has significantly better dropout and achievement gap results than schools with similar portions of disadvantaged students that they say should not be on the list. I think that high AP or IB participation is a key indicator of a school that is moving in the right direction on other measures of quality, so I will be very surprised if there is an inner city school that doesn't reach the Challenge Index mark but does better than those that are on the list on graduation and achievement rates.
But if they do find such schools, I will be very glad, and eager to help them and Newsweek and The Post come up with an entirely new way of rating high schools, for the betterment of all of us, especially our children.