Schools Know How to Lie with Statistics
Education officials across the nation are using the art of statistics to inflate the results of student testing, allowing hundreds of public schools to avoid stringent sanctions and an embarrassing failing label.
The controversial "margin-of-error" formulas used in 35 states give schools leeway--ranging from minor to dramatic--to meet math and reading standards that many parents and even some educators thought were set in stone, state and federal records show.
In Oregon, statistical latitude allowed one small high school to meet the state's requirement that 40 percent of students pass the reading test, though raw test scores showed only 28 percent of students passed the test.
In Maryland, an elementary school met standards with 31 percent of its black pupils passing a state math test--though a 41.4 percent passing rate was required.
Even in Illinois, which uses a different and far more conservative approach than other states, students at a handful of schools have been able to meet standards with a lower passing rate--37 percent--than the 40 percent the state requires.
The little-noticed formulas are beginning to surface as the No Child Left Behind Act, the most sweeping federal education reform in decades, puts pressure on states to boost student test scores.
Designed to ensure children of all backgrounds learn in school, the law has run into many obstacles, particularly in struggling urban districts such as Chicago. It has met widespread resistance from educators, who say it is difficult or impossible to meet the law's requirements, and it has frustrated parents who have failed to receive the promised benefits.
State and federal education officials say statistical leeway is needed to guard against errors in the testing process that could unfairly brand schools as failing under federal law if too few children pass state tests.
But some statisticians and testing experts are critical. There is no "theoretical basis" for applying the kinds of statistical models that most states are using to bolster test results, said Paul Sally, a University of Chicago mathematics professor.
"What they're trying to do is fake it, just plain and simple," Sally said.
Critics say the statistics mask the reality of how schools are performing and deny students benefits available under No Child Left Behind. Extra tutoring and transfers to better schools are offered to disadvantaged children only if their schools are labeled as failing to meet standards.
"In the long run, you're doing the kids a disservice," said Lawrence Aleamoni, a University of Arizona professor who teaches graduate-level statistics and advised Arizona on its student testing program.
The full effect of the formulas is not known, because many states are still analyzing test results. But there are signs that the impact could be dramatic.
Kansas education officials said 197 more schools would have failed to meet standards--double the number on the failing list--had statistical leeway not been allowed. Arkansas said another 25 schools would not have met standards. And the number of failing schools in Maryland would likely have doubled to more than 1,000 without the statistical leeway, said Deputy State Supt. Ronald Peiffer.
The formulas are buried inside state plans approved by the federal government to carry out No Child Left Behind reforms signed into law last year.
To understand the formulas, parents first have to understand No Child Left Behind.
The law requires annual student testing and sets a 12-year timetable for closing racial and economic achievement gaps in test scores. States develop their own tests and set passing rates required each year. Schools must make "adequate yearly progress" toward a requirement that all students pass the tests by 2013-14.
For the first time this year, schools are being judged not only on overall scores, but also on the performance of student groups, such as minority and low-income students who also must meet annual passing rates or their entire school will fail.
Sanctions are imposed on Title I schools--those that get federal dollars because of low-income students--if they consistently fail to make progress, including offering costly tutoring programs to disadvantaged students and allowing children to transfer to better schools.
Schools that don't get Title I dollars face other sanctions in their own states, in addition to the negative publicity that comes with a failing label.
The requirements are so stringent, many educators say, that they are impossible to meet.
"It's an unworkable system," said Tim Kurtz, head of assessment for New Hampshire's education department. "It will lead nowhere but identifying virtually every school in the country as failing."
Sympathetic to state concerns, the federal government approved proposals to use statistical formulas that allow leeway in evaluating test results, said Celia Sims, special assistant in the U.S. Department of Education's Office of Elementary and Secondary Education.
The argument was that test results aren't always perfect for a variety of reasons, Sims said. Groups of students tested every year are not the same, and not all grades are tested at every school, which means an entire elementary school could be labeled as failing if its 3rd- or 5th-graders don't do well.
In addition, some students might have a bad day on test day, which could skew results for the whole school or student group, particularly when very small schools and groups are tested.
To compensate, most states are using "confidence interval" formulas, a statistical approach that creates a band of acceptable passing rates based on size of the group tested, average test scores and the range of scores, among other factors.
Thirteen states use a 99 percent confidence interval, which means that school officials can be 99 percent sure that the true measure of students' performance lies somewhere in that band of acceptable scores.
However, 99 percent certainty requires a very broad range of possible passing rates, especially when testing small groups. Fourteen states use a 95 percent confidence interval--the more standard approach, according to experts. That allows a narrower band but still can make a big difference for schools trying to meet standards.
For instance, Maryland's 95 percent formula saved Riverview Elementary School in Baltimore County from the failing label it would have received because raw test scores for two student groups, blacks and special education pupils, fell short of standards.
In the case of the 58 black pupils tested in math, the band of acceptable passing rates ranged from 24.2 to 58.6 percent, even though the state standard was 41.4 percent in 2003. So the pupils' 31 percent passing rate sufficed.
A handful of states, such as Illinois, apply margin-of-error formulas other than confidence intervals, though they still provide some leeway for schools to meet standards.
The margin-of-error formulas are being used even though a number of provisions are in place to make it easier for schools to meet standards. Under a "safe harbor" provision, schools can meet standards if student groups show certain improvements in test scores from one year to the next, even if they don't meet passing requirements on state tests.
Schools also are allowed to average current test scores with scores from previous years to help meet standards.
Oregon, for example, uses two years of test scores to calculate passing rates. But it also uses a 99 percent margin of error formula.
Educators usually say what they are doing is similar to the margins of error seen in a random sampling of voters in a presidential poll. But critics believe that argument is flawed, saying that measuring the performance of a finite group of students who must take the test is not the same as estimating information about a large group from a random sample.
In addition, the statistical formulas can create a confusing picture for parents who may wonder what the real standard is, and whether their child's school passed or not.
Tom Fisher, a national testing expert who oversaw Florida's assessment system for more than two decades before retiring last year, said he always steered away from confidence intervals.
"I wanted pronouncements [on test results] to be exactly what they were," he said. "If you say the passing rate is 40 percent, it's 40 percent, not 37."
Florida is one of 15 states, plus the District of Columbia, that do not use the margin-of-error formulas in evaluating whether schools make adequate progress under No Child Left Behind. This year, nearly 90 percent of Florida public schools failed to meet test standards.
Illinois takes a conservative approach that allows only "borderline" schools to benefit from a margin-of-error formula, said Lynne Haeffele Curry, director of planning and performance at the State Department of Education.
Borderline schools, as a whole, already meet the state's required 40 percent passing rates on reading and math tests, as well as other key standards, such as test-participation requirements.
For those schools, a 37 percent passing rate will be allowed for individual student groups tested, such as minority and disadvantaged children. Curry calls it a "benefit of doubt" plan for those schools that just barely fail to make the grade.
Four Chicago elementary schools have met standards this year with the help of that formula--Octavio Paz Charter School, George Rogers Clark, Murphy and Zapata Academy. Those may be the only schools that benefit, Illinois education officials said, though statewide test results are still being reviewed.
For example, 44 percent of Zapata pupils passed the state reading test and 58 percent passed the math test--above the state standards. The school fell short in only one category of pupils, limited English proficient children in 3rd grade, who had a passing rate of 37 percent on the reading test. For that reason, Zapata was able to make adequate yearly progress under the margin of error plan.
Zapata and the other schools were initially told they had failed to meet standards and had to offer parents transfers to better schools.
But after hundreds of letters went out to parents in August, state officials called back to say the schools had met state requirements after all, catching them by surprise. Zapata Principal Christina Gonzalez still can't explain what happened. "Even educators had no idea" that such a statistical formula existed, she said.
Unlike the formulas used elsewhere, Illinois' approach is a legitimate and "very serious effort" to come up with a fair way to treat certain schools, said Northwestern University statistics professor Bruce Spencer.
"They are giving a 3 percent break, which actually is very fair but not too generous for schools," he said.
Diane Rado & Darnell Little
Schools toying with test results Some states meet standards with art of statistics
INDEX OF NCLB OUTRAGES