Orwell Award Announcement SusanOhanian.Org Home


NCLB Outrages

Juking the Stats 101: How supposed education reformers deceive with statistics

Remember these juking techniques the next time you see a story about an education miracle.

by Caroline Grannan,

After years of watching one "it's a miracle!" education reform after another somehow fail to achieve real change, I can't call myself an expert on the strategies known as "juking the stats"-- but I seem to have a better handle on this than a lot of education commentators and journalists, who too often seem susceptible to hype and flimflams.

So I was planning a post outlining some of the ways the alleged "reformers" maneuver to show supposed successes. But imagine my surprise when the Washington Pos ventured off message and did the story itself. Post management is deeply enmeshed in the D.C. civic leadership thatâs engaging in these very manipulations, so it's a somewhat bright spot in the depressing world of daily journalism to see the July 17 story, albeit low-key and eye-glazingly headlined: "Testing Tactics Helped Fuel D.C. School Gains/Tutoring, Eligibility Review Had Roles."

Oh, the phrase "juking the stats"? The iconic TV series "The Wire" probably didn't invent it, but it put it on the radar. Here's how "Wire" creator David Simon defined it in an interview with Bill Moyers:


You show me anything that depicts institutional progress in America, school test scores, crime stats, arrest reports, arrest stats, anything that a politician can run on, anything that somebody can get a promotion on. And as soon as you invent that statistical category, 50 people in that institution will be at work trying to figure out a way to make it look as if progress is actually occurring when actually no progress is.

So, here are the simple strategies Washington Post reporter Bill Turque listed as "fuel(ing)" those D.C. school gains under controversial D.C. schools chancellor Michelle Rhee, an icon to the "school reform" true believers.

* "improved statistical housekeeping"
* "intensive test preparation targeted to a narrow group of students on the cusp of proficient, or passing, scores" [and largely leaving other students â those far behind and most in need of help â neglected]
* Changing the reporting for students who didnât take the tests: Previously, "any student who did not take the test counting as having failed. Starting in 2008, students who did not take the test did not count as either a pass or a fail."
* "'cleaning the rosters' of students ineligible to take the tests â and also likely to pull the numbers down"

I had to read carefully for clarification on that last one. For high school grades, D.C. schools report only 10th-graders' test scores. But it often takes careful record-keeping to determine who's technically a 10th-grader based on credits earned. Quite a number of students who appear to be high school sophomores have actually blown enough classes to still be officially ninth-graders. Ensuring that students in that category aren't having their test scores reported among the 10th-grade scores keeps likely low-scorers' results out of the statistics. Does it mean the 10th-graders actually learned more in D.C. schools? Does it show that the quality of education improved? Obviously not.

As for the strategy of targeting those students who fall just short of the passing mark, the article explained:


One of Rhee's most widely discussed initiatives was "Saturday Scholars," a 13-week program for about 5,000 invited students whose academic records suggested that they were close to scoring at proficiency levels.
Critics call Saturday Scholars and programs like it "educational triage" that focuses disproportionate attention on students who require the least help.
Kerry Sylvia, a social studies teacher at Cardozo High School, said Saturday Scholars was less about serving children and more about making the adults who run the school system look good.


There are lots of other ways that those stats get juked, deliberately or just because of circumstances, the way things shake out. A partial-- and only partial-- list of examples:

* "Creaming": This is the outcome of any school application process that imposes hoops to jump through-- or even that requires any application process at all, as opposed to default assignment. Those processes screen out the uninformed, unmotivated, absent, low-functioning, overwhelmed and/or uncaring parents/guardians whose children are likely to pose the greatest challenge in the classroom and to achieve poorly. The more effort required for the process, the greater the creaming effect is likely to be.
* Attrition: Some schools shed students who aren't making it, for whatever reason. Lots of schools that serve low-income students have high turnover (because low-income families tend to have unstable lives), but most of those schools replace the students who leave with similarly high-mobility newcomers. Some don't. California KIPP schools are a known example. When the more-struggling students leave, the achievement is likely to rise, needless to say.
* Manipulation and selective use of figures: Plucking out one impressive-looking subcategory to report can mislead successfully. Back when controversial, for-profit Edison Schools was the education reform fad du jour and San Francisco's Edison Charter Academy (ECA) was a hot news story, Edison claimed big score increases specifically for ECA's African-American students. Those supposed gains were widely cited as though they were a sound basis for evaluating the school's overall achievement. But actually, ECA was experiencing high attrition of its African-American students and also was testing a startlingly low percentage of those remaining-- only about 75 percent of them, according to a research and information project, Parents Advocating School Accountability (which I co-founded), which documented the figures.
* Dishonesty: Edison Schools provided some stellar examples. One year it issued a report touting the supposed gains of its schools nationwide-- but actually excluded more than half its schools from the figures in the report. The unsuspecting mainstream press widely reported the dishonest figures without question.

I'm sorry this post is so cynical. It would be lovely to believe that every time we hear an encouraging story, trust and optimism is justified. But in reality, our current educational system creates huge incentive for dishonesty as we reward supposed successes and punish failures. It benefits no one-- least of all our students -- when we remain willfully naïve. One upbeat note is that the gloom-n-doom reports about American education are greatly exaggerated, as I've written before and will continue to report.

— Caroline Grannan
Examiner.com
2009-06-18
http://www.examiner.com/x-356-SF-Education-Examiner~y2009m7d18-Juking-the-Stats-101-How-supposed-education-reformers-lie-with-statistics?cid=examiner-


INDEX OF NCLB OUTRAGES


FAIR USE NOTICE
This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.