McGraw-Hill Testing: Here We Go Again: the worst scoring miscalculation in the 19-year history of the annual test
Ohanian Comment: Momma, don't put your baby's future into the tentacles of McGraw-Hill. We've seen this question before: Have expectations extended beyond the capacity to deliver? Translated, this means, "Can the testing industry deliver on the massive amount of testing required by NCLB? We can all hope that they can't.
Note that only 4th, 6th, and 8th graders were being tested at this time. What happens when the NCLB mandates for testing in all grades kicks in?
INDIANAPOLIS -- In cavernous rooms lined with rows of computers, the fate of Connecticut's multimillion-dollar student testing program depends - literally - on whether Indiana workers like Ethel Trice-Sanders know the score.
One of the nation's largest testing companies, racing to fix errors in its initial scoring report, has hired an army of workers to rescore student writing on this year's Connecticut Mastery Test.
"This supplements my retirement," said Trice-Sanders, 74, one of hundreds of temporary employees selected for their ability to maintain rigid accuracy as they reduce each written answer to a single number.
No less than CTB/McGraw Hill's new seven-year, $48 million contract with Connecticut is on the line as the California-based company tries to recover from the worst scoring miscalculation in the 19-year history of the annual test.
A flawed initial report by the company not only forced Connecticut to postpone its customary January release of scores, but also raised questions about whether the company - and the entire testing industry - can handle the onslaught of new test requirements under President Bush's school reform law, the No Child Left Behind Act.
"There's a lot at stake here," said Barbara Q. Beaudin, acting chief of the state Department of Education's bureau of student assessment. "Can we do it as a state? Can the industry do it? Have expectations extended beyond the capacity to deliver?"
Some experts contend that the glitch is a warning of more problems to come for an overburdened industry.
"It's a terrible problem. It's happening all over the United States with greater consequences for students and teachers," said Thomas Haladyna, a testing specialist at Arizona State University.
Connecticut asked CTB/McGraw-Hill to rescore this year's exams after the company, in the first year of its contract, reported scores that were mysteriously lower than those of a year ago. About 125,000 fourth-, sixth- and eighth-graders took the exam in September.
The problem was traced to questions that required essays or other written answers - an increasingly common task on many state achievement tests and soon even on the SAT college entrance exam.
Educators believe written answers are crucial in demonstrating student skills but, unlike multiple-choice answers, they are far more difficult, time-consuming and costly to grade.
Connecticut - which tests students in reading, writing and mathematics - was one of the earliest states to require open-ended written answers on parts of its exam.
"If we can't score this correctly," said Beaudin, "that will bring into question whether this higher level of testing we're doing can be done on a broader level."
That question will be answered here at CTB/McGraw-Hill's Indianapolis center, a sprawling complex consisting of converted strip mall stores. It is the largest of the company's four scoring centers, a bustling factory that processes hundreds of thousands of tests from across the nation.
Beaudin was in Indianapolis earlier this month with a contingent of Connecticut and local school testing officials to monitor the rescoring effort.
The company has mounted a massive effort, bringing in more than 900 scorers who are working two shifts and weekends to meet a series of deadlines. The first reports are scheduled to be sent to districts by the end of this month, with a final report due June 4.
The scorers, many of them veterans of earlier projects, are recruited by Kelly Services, a job placement agency with an office located at the center. A bachelor's degree is a minimum requirement for the job. The scorers start at $10.50 an hour and range from recent college graduates to moonlighting teachers to retirees such as Trice-Sanders.
By the end of her eight-hour shift, Trice-Sanders estimates she will finish 300 papers, each consisting of the same block of written answers from one small section of the test.
Trice-Sanders, a grandmother and former nutritionist, calls the work "my brain aerobics." She reads answers that have been scanned into a computer and pop up on her screen in each student's own handwriting.
"We always focus on accuracy first," Trice-Sanders said, referring to strict guidelines that require scorers to adhere to a point scale designed to take the guesswork out of grading student writing.
State officials believe that faulty monitoring of how scorers followed those guidelines is at the heart of what went wrong in the first round of scoring.
CTB/McGraw Hill took over from another company, and the transition led to inconsistencies and misunderstanding about the guidelines, CTB/McGraw-Hill officials say.
"There is an art to scoring," said Bud Hall, CTB/McGraw-Hill's director of hand-scoring. "There is subjectivity. ... Our work is to remove as much of that variable as possible."
The company insists it has tightened its procedures on the Connecticut exam.
It puts employees through hours of training on Connecticut's scoring blueprint - which includes strict point scales for each written answer.
In a cluster of cubicles, dozens of trainees wearing headsets listen to instructions as they read a sample essay, looking for elements such as organization, fluency and detail.
"We are starting to get more specific detail" in this sample, trainer Pam Spear tells them, explaining that the essay would merit a score "in the mid-5-point range" on a 6-point scale.
The company rejects scorers who cannot consistently rate papers accurately.
On the writing exam, every essay is read by at least two scorers. The company also assigns team leaders to rescore random papers to make sure scorers stick to the guidelines. A group of supervisors performs similar rechecks on the team leaders.
"You develop a rhythm after a while. You internalize the guidelines," said Judy Dye, a veteran team leader who helps realign scorers who, after rating hundreds of essays, sometimes "drift away from where they belong."
Straining The System
Company officials say they are confident the quality checks will iron out earlier problems, but as recently as last week, they still were finding errant scoring patterns on sixth-grade writing tests.
State officials are taking a wait-and-see stance.
"I won't be entirely convinced until I see they've met the deadlines," state Education Commissioner Betty J. Sternberg said.
The state hired CTB/McGraw-Hill last year, ending a long relationship with Harcourt Assessment, a Texas company that had scored Connecticut's mastery test since it was first given in 1985. In recent years, state Department of Education officials had grown increasingly dissatisfied with Harcourt's work.
"They were consistently late, inaccurate," said Abigail Hughes, who oversaw the testing program for the state. "These kinds of problems occur with all kinds of [test] companies ... a lack of quality control."
Within months, there were signs that the new company, too, was off to a rocky start.
"When can we expect to get the scoring tables ... sent to us for approval?" asked a Dec. 4 e-mail from the state Department of Education, one of a series of increasingly urgent messages to the company about procedural delays.
A Boston College study last year found that scoring mistakes have occurred at several testing companies. One such error in 1999 by CTB/McGraw-Hill resulted in incorrect scores for a quarter-million students in Tennessee, Indiana, Wisconsin, Nevada, South Carolina and New York City, the study found.
Because of another testing company's error, 50 Minnesota students were temporarily denied high school diplomas in 2000, the report said.
Georgia officials limited the administration of their test last year after a testing company inadvertently revealed some test questions before the exam was given.
"In Connecticut, we've been criticized by testing companies [saying], `Why do you have to double-check everything?'" Hughes said. "We've seen those cases in Georgia, New York, Maryland, where they were wrong. Unfortunately, it seems to be the nature of the industry."
Much of the nation's school testing is done by only a handful of major companies, which now must gear up for a heavier workload. Under the federal No Child Left Behind Act, most states are expanding existing test programs or adding new ones. In Connecticut, the mastery test program will add grades 3, 5 and 7, doubling its size by 2006.
"Given No Child Left Behind ... and given the fact that 50 [states] are trying to get vendors to do this ... it's straining the system," said George F. Madaus, one of the authors of the Boston College study.
Michael Kean, a spokesman for CTB/McGraw-Hill, disagrees. Although schools are adding new tests to comply with the federal law, they are also dropping older tests, he said. "It's largely a trade-off."
He said CTB/McGraw-Hill has been expanding for more than a decade and can handle new demands. It currently holds contracts in 22 states, he said.
Of more immediate concern is the Connecticut mix-up. Unless the company can resolve the scoring problem, some educators fear there could be pressure to reshape the test itself - by backing away from essays or other written answers.
That would not surprise 68-year-old Lucille Heinzmann.
The retired American Express employee has worked on several scoring projects for CTB/McGraw-Hill but found the Connecticut test too difficult to grade.
"They took me off that project. My team leader and I couldn't agree on a number of scores," said Heinzmann, who described the scoring guidelines as "loose and inconsistent."
Nevertheless, after her brief stint, she had formed her own opinion about Connecticut's students.
"I've done Indiana, I've done Colorado, I've done Missouri," she said. "Connecticut students spell better. Some of them write a lot better."
A discussion of this story with Courant Staff Writer Robert Frahm is scheduled to be shown on New England Cable News each half-hour Monday between 9 a.m. and noon.
Robert A. Frahm
Feverishly Fixing An Error
INDEX OF NCLB OUTRAGES