Success Metrics Questioned in School Program Funded by Goldman
Ohanian Comment: Should those of us who care about the survival of public education be glad that Penny Pritzker is now Obama's Secretary of Commerce and not continuing in her role of destroying schools in Chicago?
Why wasn't this a Front Page story in the Times?
When Goldman-Sachs decides to get into preK education, guard your children.
by Nathaniel Popper
It was, in the vernacular of corporate America, a win-win: a bond that paid for preschool for underprivileged children in Utah while also making money for investors.
Goldman Sachs announced last month that its investment in a Utah preschool program had helped 109 "at-risk" kindergartners avoid special education. The investment also resulted in a $260,000 payout for the Wall Street firm, the first of many payments that is expected from the investment.
Gov. Gary R. Herbert of Utah hailed the program as a model for a new way of financing public projects. Such so-called social impact bonds are a new kind of public-private partnership, promising financing from Wall Street and imposing a goal on local governments.
Yet since the Utah results were disclosed, questions have emerged about whether the program achieved the success that was claimed. Nine early-education experts who reviewed the program for The New York Times quickly identified a number of irregularities in how the programÃ¢€™s success was measured, which seem to have led Goldman and the state to significantly overstate the effect that the investment had achieved in helping young children avoid special education.
Goldman said its investment had helped almost 99 percent of the Utah children it was tracking avoid special education in kindergarten. The bank received a payment for each of those children.
The big problem, researchers say, is that even well-funded preschool programs -- and the Utah program was not well funded -- have been found to reduce the number of students needing special education by, at most, 50 percent. Most programs yield a reduction of closer to 10 or 20 percent.
The program's unusual success -- and the payments to Goldman that were in direct proportion to that success -- were based on what researchers say was a faulty assumption that many of the children in the program would have needed special education without the preschool, despite there being little evidence or previous research to indicate that this was the case.
"We're all happy if Goldman Sachs makes money as long as they are making it with smart investments that make a real difference," said Clive Belfield, an economics professor at Queens College in New York, who studies early childhood education. "Here they seem to have either performed a miracle, or these kids weren't in line for special education in the first place."
The concerns about the program are a reminder of how hard it is to properly structure public-private partnerships like social impact bonds, which depend on easily verifiable and commonly agreed-upon methods of measuring success for goals that can be hard to define, such as student success.
Finding such measurements is increasingly important as government programs face cutbacks, and public officials look to find private investors willing to address the funding gap. Social impact bonds have been described as one of the most promising ways to harness this money.
Indeed, these bonds will be the focus of a conference hosted by the Federal Reserve Bank of Philadelphia on Wednesday, when the mayor of Philadelphia, Michael Nutter, and a representative from the White House will speak, along with the Goldman executive who oversaw the Utah program.
But the criticism of the Utah program points to issues that could hinder the wider adoption of the concept of social impact bonds. Kenneth A. Dodge, a professor at the Sanford School of Public Policy at Duke, who has been an advocate for the pay-for-success model, said that if the model was to succeed it would have to be done differently than it was in Utah.
"It is a step in the right direction, but this is not the criteria I hope we hold ourselves to ultimately," he said.
More immediately, the apparent overstatement of the Utah program's results mean that the payments that Goldman -- and a philanthropic partner, the J.B. & M.K. Pritzker Family Foundation -- recently received from the state of Utah and the local United Way were probably also higher than they should have been.
A spokeswoman for Goldman, Leslie Shribman, deferred questions about the methods used to determine the payments it had received to Utah officials, who she said had developed the methodology.
"We were approached by community leaders in Utah to help address an important local priority," she said in a statement. "Social impact bonds allow the government to avoid paying for programs that don't work. In this case, targets were set publicly, impact was measured transparently, and the government paid for success based on those targets."
Brenda Van Gorder, director of preschool services at the local school district, said she was happy with the program because it had induced Goldman to pay for a program that the state would not have otherwise supported.
But she and the researcher who tested the local children, Mark Innocenti, acknowledged that they lacked certain basic data on what would have been expected to have happened to the students without the Goldman-funded preschool, and the difference in the performance of the students in the program versus other similar students who did not attend preschool.
"If we had the data -- if we could follow kids without intervention -- we would be able to come up with a clearer idea of what is happening," said Mr. Innocenti, a researcher at Utah State University. "Time for another study, I guess."
Early-childhood education experts said that the results from Utah should have been viewed skeptically from the start, just based on the amount of money being spent on the program.
Ms. Van Gorder said the preschool that the bank had paid for cost $1,700 a year for every student, or barely enough to cover the cost of part-time day care. Some of the children Goldman paid for were sent not to preschool but to a local day care center or Y.M.C.A., according to Ms. Van Gorder.
The preschools that have been found, in previous research, to reduce the future need for special education usually cost four or five times what Goldman spent in Utah.
"There are hundreds of studies of programs like this, and none of them find a large positive impact," said W. Steven Barnett, the director of the National Institute for Early Education Research at Rutgers University. "You have to spend real money to get results."
The rate of success being reported by Goldman -- and the success of the whole program -- is a product of the program's method of identifying a population of low-income 3- and 4-year-olds who were likely to need special education without preschool.
The school district tested each incoming preschool student using a picture and vocabulary test known as the P.P.V.T. Any child who, before entering preschool, received a score below 70, a very low score, was labeled likely to later need special education.
For Goldman, the children identified in this way were crucial to its investment, because the bank was paid for each at-risk child who ended up not needing special education after leaving the preschool program.
But early childhood experts said it made little sense to base Goldman's payouts on the assumption that all of the children who scored low on the P.P.V.T. test would end up in special education without the preschool.
At the most basic level, the P.P.V.T. is not usually a test used to screen for special education, particularly on its own, education experts said. What is more, non-English-speaking students have been shown to score very poorly on the test when it is administered in English, which is not a sign of any learning disability, but of a need for English instruction.
"To just assume that all these children would have gone to special education is kind of ridiculous," said Ellen S. Peisner-Feinberg, a senior scientist at the Frank Porter Graham Child Development Institute.
Mr. Innocenti, who administered the tests in Utah, said that from 30 to 50 percent of the children in the preschool program come from homes where English is not the only language. He said the school decided to test the children in English, despite the many non-English-speaking children, because the preschool program is conducted in English.
Before Goldman executives made the investment, they could see that the Utah school district's methodology was leading large numbers of children to be identified as at-risk, thus elevating the number of children whom the school district could later say were avoiding special education. From 2006 to 2009, 30 to 40 percent of the children in the preschool program scored below 70 on the P.P.V.T., even though typically just 3 percent of 4-year-olds score this low. Almost none of the children ended up needing special education.
When Goldman negotiated its investment, it adopted the school district's methodology as the basis for its payments. It also gave itself a generous leeway to be paid pack. As long as 50 percent of the children in the program avoid special education, Goldman will earn back its money and 5 percent interest Ã¢€” more than Utah would have paid if it had borrowed the money through the bond market. If the current rate of success continues, it will easily make more than that.
Ms. Peisner-Feinberg said that before these sorts of investment are considered again, investors and schools needed better ways to measure them.
"You have to be sure you have very rigorous ways of measuring the impact to make sure that it's legitimate in terms of the outcome you get," she said. "That didn't happen here."
New York Times Deal Book