Orwell Award Announcement SusanOhanian.Org Home

Conflicting Studies About Private Management Not Conclusive

Susan Notes: A review of two studies about Philadelphia calls for more study. The topic is important because researchers were looking at whether whether private management has improved elementary and middle school achievement in Philadelphia.

Education Policy Research Unit (EPRU) at ASU

Education and the Public Interest Center (EPIC) at CU-Boulder



In two studies of a Philadelphia reform with national implications, RAND researchers have presented better information than have Harvard researchers, but the RAND researchers should provide further analyses

Contact: Derek Briggs (303) 492-6320 (email) Derek.Briggs@Colorado.EDU

Kevin Welner (303) 492-8370 (email) kevin.welner@gmail.com

TEMPE, Ariz and BOULDER, Colo. (May 8, 2007) -- Two recent studies examining the restructuring of public schools in Philadelphia reach different conclusions about whether private management has improved elementary and middle school achievement. Because similar restructuring reform is promoted by the federal No Child Left Behind law, these studies have important national implications. Both studies have flaws, however, and neither study is yet sufficient to make definitive claims about the reformâs effects, a new review of the studies finds.

The two reports are âState Takeover, School Restructuring, Private Management, and Student Achievement in Philadelphia,â released by the RAND Corporation in partnership with Research for Action (RFA); and âSchool Reform in Philadelphia: A Comparison of Student Achievement at Privately-Managed Schools with Student Achievement in Other District Schools,â published by the Program on Education Policy and Governance (PEPG) at Harvard University.

The RAND-RFA report was written by Brian Gill, Ron Zimmer, Jolley Christman and Suzanne Blanc. The second report was written by Paul Peterson, who directs PEPG.

The reports were reviewed for the Think Tank Review Project by Derek Briggs, a professor at the University of Colorado at Boulder.

Both reports attempt to assess the impact of a school reform in Philadelphia. Starting in 2002, the cityâs school district restructured its 86 lowest-achieving elementary and middle schools. The most prominent approaches shifted school management to either the district or one of several private providers. The reports under review by professor Briggs examine the impact of private or district management on student achievement.

The reports both focus on math and reading test scores of students. The RAND-RFA paper concludes that private management had no impact on math or reading achievement, while district management had a positive effect on math achievement but none on reading. The PEPG paper, meanwhile, reached virtually the opposite conclusion. It found private management has had a positive effect on the performance of lower-scoring students in math and reading, while district management had no measurable effect.

âThe different findings from the two reports can largely be explained by the fact that PEPG did not have the same access to data as did RAND-RFA,â reviewer Derek Briggs writes in his summary. Largely because of that data gap, the PEPG report used different analytical methods than the RAND-RFA researchers, Briggs explains.

Briggs questions the RAND-RFA report because its authors, in choosing a particular statistical model for their analysis, never explain why they reject other approaches. He also states that report lacks a much-needed technical appendix as well as important âdescriptive statistics,â such as information on the movement of students in and out of the schools in the SDP during the time period being analyzed. He also calls into question some of the assumptions of the RAND-RFA modeling approach and suggests that the authors give those assumptions greater scrutiny.

Briggs criticizes the second, PEPG report on several grounds:

It uses baseline years of 2002 and 2003, which are after the reform already began, meaning that âany initial effects of the reform are not captured by the PEPG analysis.â

It focuses only on a subset of reform schools (those that include grades 5 through 8) and a subset of students (those who perform the worst on standardized achievement tests). It then compares these students in these unique types of schools to those in a set of non-restructured schools that do not necessarily encompass both grades 5 and 8. If switching from elementary to middle school, for example, hurts student achievement, âthis would lead to the overestimation of the effects Peterson has found for both privately managed and district-managed schools.â

The PEPG data analysis is not able to address issues of differential mobility â" it doesnât sort out whether high-achieving students are more likely or less likely to stay in a given type of restructured schools. As a result, the students whose test scores are measured in grade 8 may be very different than the students whose test scores were measured in grade 5.

Because the PEPG test score comparisons only include schools with students in both grades 5 and 8â"about 40 percent of privately managed schools, and 20 to 40 percent of all district-managed schoolsâ"it is unclear whether the effects he describes, even if they were correct for his subset, could be generalized to the unrepresented schools.

Briggs concludes: âThese two studies, read together, do contribute to our understanding of Philadelphiaâs experience. Yet, although the RAND-RFA study sheds more light on that experience than does the PEPG study, neither study offers a complete picture, and more research is needed before drawing any definitive conclusions.â

In particular, Briggs suggests that RAND could easily reconcile differences between its own conclusions and those of the PEPG report by conducting a new analysis of its data.

Find Derek Briggsâ review on the web at:


About the Think Tank Review Project

The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the ASU Education Policy Research Unit (EPRU) and CU-Boulderâs Education and the Public Interest Center (EPIC), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.

Kevin Welner, the project co-director, explains that the project is needed because, âdespite their garnering of media attention and their influence with many policy makers, reports released by private think tanks can be of very poor quality. Too many think tank reports are little more than ideological argumentation dressed up as research. We believe that the media, policy makers, and the public will greatly benefit from having qualified social scientists provide reviews of these documents in a timely fashion.â He adds, âwe don't consider our reviews to be the final word, nor is our goal to stop think tanksâ contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate.â


Derek Briggs, Professor

University of Colorado at Boulder

(303) 492-6320


Kevin Welner, Professor and Director

Education and the Public Interest Center

University of Colorado at Boulder

(303) 492-8370


— News Release
Education Policy Research Unit (EPRU) at ASU &


This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.