Software's Benefits On Tests In Doubt: Study Says Tools Don't Raise Scores
Note the label: President Bush's No Child Left Behind law. Bush's war. Bush's law.
By Amit R. Paley
Educational software, a $2 billion-a-year industry that has become the darling of school systems across the country, has no significant impact on student performance, according to a study by the U.S. Department of Education.
The long-awaited report amounts to a rebuke of educational technology, a business whose growth has been spurred by schools desperate for ways to meet the testing mandates of President Bush's No Child Left Behind law.
The technology -- ranging from snazzy video-game-like programs played on Sony PlayStations to more rigorous drilling exercises used on computers -- has been embraced by low-performing schools as an easy way to boost student test scores.
But the industry has also been plagued by doubts over the technology's effectiveness as well as high-profile bribery scandals, including one that led to the resignation of the Prince George's County schools chief in 2005.
The study, released last night, is expected to further inflame the debate about education technology on Capitol Hill as lawmakers consider whether to renew No Child Left Behind this year.
"We are concerned that the technology that we have today isn't being utilized as effectively as it can be to raise student achievement," said Katherine McLane, spokeswoman for the Department of Education.
Industry officials played down the study and attributed most of the problems to poor training and execution of the programs in classrooms. Mark Schneiderman, director of education policy at the Software and Information Industry Association, said that other research trials have proven that the technology works, although he said that those trials were not as large or rigorous as the federal government's.
"This may sound flip or like we're making excuses, but the fact is that technology is only one part of it, and the implementation of the technology is critical to success," said Schneiderman, whose group represents 150 companies that produce educational software. "We need to take every study with a grain of a salt and look at the overall body of work."
The study, mandated by Congress when it passed No Child Left Behind in 2002, evaluated 15 reading and math products used by 9,424 students in 132 schools across the country during the 2004-05 school year. It is the largest study that has compared students who received the technology with those who did not, as measured by their scores on standardized tests. There were no statistically significant differences between students who used software and those who did not.
In classrooms, the programs -- such as "iLearn Math" and "Achieve Now" -- are used in different ways, depending on teachers. Some educators use the software as a supplemental tool to drill students in particular lessons; others use it instead of textbooks to teach entire lessons.
Backers say the technology better engages students by giving them individualized instruction and prepares them for a technology-filled world. Schools use the software to teach almost every subject, although the federal study looked only at math and reading programs.
In the Washington region, the debate over educational software raged most prominently in Prince George's, where Superintendent Andre J. Hornsby resigned and was indicted on suspicion of arranging for the school system to buy $1 million worth of software from LeapFrog SchoolHouse, where his then-girlfriend was a saleswoman. The indictment says that he demanded and received kickbacks. The schools have not made any major software program purchases since.
County Superintendent John E. Deasy said the programs aren't magic bullets. "No technology adds value by itself," he said. "Just employing software is not likely to lift test scores for students."
Nationally, perhaps no school system better represents the fears of industry supporters than that of Los Angeles, which spent $50 million in 2001 to buy Waterford Early Reading, distributed by software giant Pearson Digital Learning.
Ronni Ephraim, a chief instructional officer for the district, said the company gave presentations that described how successful the program was for other schools. Los Angeles school administrators soon began praising it.
"Teachers loved it. Kids loved it," Ephraim said. "Waterford gave us data from their tests that showed it was working. Everyone said, 'Oh my God! The kids are doing so well.' "
But a school district evaluation found that students using Waterford were not scoring better on standardized tests than those not using it. "I'm so embarrassed to admit this," Ephraim said, "but when we heard the results we said, 'This can't be true.' "
The Los Angeles system dropped the program from its regular classes but sometimes uses it for individual students. Ephraim said she blamed the school system, because teachers were not prepared or properly trained to use the technology.
Nonetheless, some experts said the software holds promise. Elliot Soloway, professor of educational technology at the University of Michigan, said that teachers need to be better trained and that administrators need to wait more than one year to see results. He said he worried that the study would scare off school districts.
"This is the last thing that we need now," he said. "It is the poor kids who will suffer, because it is their schools who will not get technology because of this study."
To persuade companies to participate in the study, researchers promised not to report the performances of particular programs. Among the businesses whose products were in the study were LeapFrog SchoolHouse, PLATO Learning, Scholastic Inc. and Pearson. (The Washington Post Co. owns Kaplan, a test preparation company that sells education software. Kaplan applied to be in the study but was not included.)
Although some of the companies are now criticizing the report, many were initially eager to be studied and praised researchers.
"We are proud to be the largest commercial supporter of this important study of the effectiveness of using technology in the classroom," said John Murray, president and chief executive of PLATO, in 2004.
Amit R. Paley
INDEX OF NCLB OUTRAGES