Orwell Award Announcement SusanOhanian.Org Home

Headline-Grabbing Charter School Study Doesn’t Hold Up To Scrutiny

Susan Notes:

Reviewer finds serious statistical flaws in research on NYC charter schools. The moral here is Beware of media headlines and editorials on education research, particularly beware of it when it damns public schools. Particularly when they say things like "The facts are in." We always need to ask "whose faces?"

Press Release

Contact: Sean Reardon, (650) 736-8517 (office); (617) 251-4782 (cell);

Kevin Welner, (303) 492-8370; kevin.welner@colorado.edu

Gary Miron, (269) 599-7965; gary.miron@wmich.edu
BOULDER, Colo. and TEMPE, Ariz.

November 12, 2009) â" A recent report on New York City charter schools found
achievement results at the charters to be better than comparison traditional schools. But
that report relies on a flawed statistical analysis, according to a new review.

The report is How New York City Charter Schools Affect Achievement and was written by Caroline Hoxby, Sonali Murarka, and Jenny Kang. When it was released in late
September, it was enthusiastically and uncritically embraced by charter advocates as well as media outlets. The Washington Post an editorial titled, "Charter Success. Poor
children learn. Teachers unions are not pleased." The editorial's first paragraph reads:
"Opponents of charter schools are going to have to come up with a new excuse:

They can't claim any longer that these non-traditional public schools donât
succeed. A rigorous new study of charter schools in New York City demolishes
the argument that charter schools outperform traditional public schools only
because they get the 'best students.' This evidence should spur states to change
policies that inhibit charter-school growth. It also should cause traditional
schools to emulate practices that produce these remarkable results."

The editorial argues throughout that the study provides unquestionable evidence evidence that charters result in improved student achievement. It ends, "Now the facts are in."

The New York Daily News was no less effusive: "It's official. From this day forward,
those who battle New Yorkâs charter school movement stand conclusively on notice that
they are fighting to block thousands of children from getting superior educations."

Because of the declared importance of the new report, we asked Professor Sean Reardon
to carefully examine the report's strengths and weaknesses for the Think Tank Review
Project and write a review that would help others use the study in a sensible way. Reardon, like the report's lead author Hoxby, is a professor at Stanford University. He is
an expert on research methodology.

The Hoxby report estimates the effects on student achievement of attending a New York
City charter school rather than a traditional public school. A key finding, repeated in
press reports throughout the U.S., compares the cumulative effect of attending a New
York City charter school for nine years (from kindergarten through eighth grade) to the
magnitude of average test score differences between students in Harlem and the wealthy
New York community of Scarsdale. The report estimates this cumulative effect at
roughly 66% of the "Scarsdale-Harlem gap" in English and roughly 86% of the gap in

In his review, Reardon observes that the report "has the potential to add usefully to the
growing body of evidence regarding the effectiveness of charter schools." New York
charter schools' use of randomized lotteries to admit students to charter schools offers the
possibility that the study of those schools can roughly approximate laboratory conditions.

But Reardon points out that that the report's key findings are grounded in an unsound
analysis--an inappropriate set of statistical models--and that the report's authors never
provide crucial information that would allow readers to more thoroughly evaluate "its
methods, results, or generalizability."

Reardon's review notes these shortcomings in the report:

  • In measuring the effects of charter schooling on students in grades 4 through 12,
    the study relies on statistical models that include test scores from the previous
    year, measured after the admission lotteries take place. Yet because of that timing,
    those scores could be affected by whether students attend a charter school. As a
    consequence, the statistical models "destroy the benefits of the randomization"
    that is a strength of the study's design. (The use of a different model makes the
    results for students in grades K-3 more credible, he notes.)

  • The report's claims regarding the cumulative effects of attending a New York
    City charter school from kindergarten through eighth grade are based on an inappropriate extrapolation.

  • It uses a weaker criterion for statistical significance than is conventionally used in
    social science research (0.05), referring to p-values of roughly 0.15 as "marginally
    statistically significant".

  • The report describes the variation in charter school effects across schools in a way
    that may distort the true distribution of effects by omitting many ineffective
    charter schools from the distribution.

  • Reardon explains that, as a result of the flaws in the report's statistical analysis, the report
    "likely overstates the effects of New York City charter schools on studentsâ cumulative
    achievement, though it is not possible -- given the information missing from the report -- to
    precisely quantify the extent of overestimation," This, as well as the lack of detailed information in the report to assess the extent of that bias, make it impossible for readers to know whether the reportâs estimated charter school effects are in fact valid.

    "Policymakers, educators, and parents should therefore not rely on these estimates until
    the bias issues have been fully investigated and the analysis has undergone rigorous peer

    According to Professor Kevin Welner, director of the University of Colorado at
    Boulderâs Education and the Public Interest Center (EPIC): "Readers of this review will
    understand that, while Hoxby's charter school study is a contribution, it has significant
    flaws and limitations. Unfortunately, the editorial reaction of otherwise-respectable
    media outlets trumpeted the New York City findings as the final and faultless word on
    charter school performance. In fact, the study used inappropriate methods that overstate
    the performance of the charter schools it studied."

    Welner notes that the Think Tank Review Project also recently reviewed another charter
    school study, released in June by Stanford's CREDO policy center. That study encompassed 65-70% of the nation's charter schools. "Our review pointed out a number limitations but also noted the relative strength and comprehensiveness of the data set, the solid analytic approaches of the CREDO researchers, and the important fact that the CREDO results were consistent with a large body of research showing charter schools overall to be performing no better than (and perhaps worse than) traditional public schools," Welner says.

    But he added that "the CREDO and Hoxby reports used different designs and covered different schools. They are not directly comparable, nor are we able to say which is 'better.' Neither report is definitive or without notable weaknesses."

    Welner concludes, "the important thing to understand is that if, after an appropriate
    reanalysis of the data, we still find that New York City's charter schools are in fact
    bucking the national trend, the sensible next step is for researchers to explore the causes
    rather than to jump to broad conclusions that fly in the face of the overall research base. It
    would be irresponsible to use the NYC results-- even if they were valid and reliable--to
    drive policy in places throughout the U.S. where charters are apparently underperforming
    their competition."

    Find Sean Reardon's review on the web at:

    Find the NYC report by Hoxby and her colleagues at:

    Sean F. Reardon
    Associate Professor of Education and (by courtesy) Sociology
    Stanford University
    (650) 736-8517 (office); (617) 251-4782 (cell)

    Kevin Welner, Professor and Director
    Education and the Public Interest Center
    University of Colorado at Boulder
    (303) 492-8370

    Gary Miron,
    Professor of Education
    Western Michigan University
    (269) 599-7965

    About the Think Tank Review Project
    The Think Tank Review Project, a collaborative project of the University of Colorado at Boulder Education and the Public Interest Center (EPIC) and the ASU Education Policy Research Unit (EPRU), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for
    Education Research and Practice.

    EPIC and EPRU collaborate to produce policy briefs in addition to think tank reviews.
    Our goal is to promote well-informed democratic deliberation about education policy by
    providing academic as well as non-academic audiences with useful information and high
    quality analyses.

    Visit EPIC and EPRU at http://www.educationanalysis.org,
    EPIC and EPRU are members of the Education Policy Alliance
    ( http://educationpolicyalliance.org).

    — Press Release
    Think Tank Review Project,, EPIC/EPRU


    This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.