Orwell Award Announcement SusanOhanian.Org Home

NCLB Outrages

Unusable protractor error joins many issues with test

One of the sponsor ads on the site offers protractors for sale.

Don't you wonder about a state superintendent of public instruction who can make this statement:

The assessment itself is very valid and very reliable, but the delivery system has brought the data we will be generating into question. --Jim McBride

Maybe this comes of 26 years in the Air Force and doing what you're told instead of raising hell.

By Jackie Borchardt

While proctoring the state's standardized math exam last week, Fred Maguire, a teacher at Kelly Walsh High School, noticed something: The protractors were unusable.

Protractors have cross-hairs to indicate where to begin measuring angles. The mark was missing on the half-circle protractors supplied by the testing company, NCS Pearson. Used incorrectly, the protractors could yield a measurement off by a few degrees -- enough to be counted as a wrong answer.

Maguire made the discovery too late: 90 percent of the students already had completed the math portion of the test. Students take different forms of the test, so while some used the faulty protractors, some didn't use them at all. The rest of the students marked the line on their protractors, but previously affected students did not retake the test.

"Even if you know how to use a protractor, a student could be confused, and it takes time away from the test," Maguire said.

The protractor issue is yet another problem in this year's Proficiency Assessments for Wyoming Students, or PAWS. Students and educators around the state have been dealing with troubles in taking the test, leading some to question whether the results will be valid.

Errors, mistakes, snafus

The protractor error wasn't even on the radar for the state Department of Education, which has been discussing what to do about the PAWS problems daily with NCS Pearson, said Mary Kay Hill, director of administration for the department.

"We will make sure we account for all of the issues with Pearson in the administration of this assessment," Hill said. "Our focus is that kids who are trying to take the assessment are trying to take it in as good as an environment as possible."

The department hasn't made any decisions but hopes to have options by next week, Hill said.

PAWS tests students in third through eighth and 11th grades in reading, math and writing. Students in fourth, eighth and 11th grades are also tested in science. The tests are used to measure "adequate yearly progress" under the No Child Left Behind Act. Schools that do not make AYP are subject to "corrective action."

Teachers and building administrators worried about how technical problems will affect student scores and whether the building would make AYP, said Sandy Cherry, assistant principal at Kelly Walsh.

"Even though the students were patient and very good, we know it has an effect on the validity of the test and the questions," Cherry said.

School districts worried about the regular testing period, but testing company NCS Pearson sent technical help and assured all would be fine. Every district experienced problems with the online portion -- frozen screens, lost answers, slow load times -- but most of the problems have been worked out over the past few weeks, said Jim McBride, state superintendent of public instruction.

"The assessment itself is very valid and very reliable, but the delivery system has brought the data we will be generating into question," McBride said.

The state extended the five-week testing window by four days, and about half of the state's 48 districts have finished and submitted the tests, according to the department.

"We are very concerned about the accuracy of the data we will be getting form PAWS this year, and I'm asking the department to figure out a way forward," McBride said.

Pearson problems

Wyoming's problems are the newest in a long history of problems with NCS Pearson, said Robert Schaeffer, public education director of the National Center for Fair and Open Testing. When No Child Left Behind mandated annual testing in several grades, the testing market doubled and tripled and stretched large testing companies thin. The testing industry has become a game of musical chairs -- companies take on new states after being kicked out of others, according to Schaeffer.

"Very high-stakes decisions about kids, teachers, schools and systems are being made on the system of flawed indicators," he said. "It's one of the fundamental errors of No Child Left Behind."

States have redone tests, Schaeffer said, but there's not a precedent for how to deal with such widespread difficulty.

Wyoming began online testing portions of the annual assessment in 2006 with Harcourt Assessments. The department signed on with NCS Pearson when it purchased Harcourt in 2007.

In 2008, errors were found in calculating AYP and reporting writing scores. Despite past problems, the department signed a four-year, $40 million contract with NCS Pearson in 2009.

Problems with Harcourt were different -- the contract lacked dates, deliverables and specific timelines, McBride said. The department plans to seek damages for those parts of the contract with NCS Pearson.

"We're pleased with the assessment," McBride said. "What's problematic is the platform -- it simply wasn't adequately tested, and we were the guinea pigs."

About 95 percent of the bugs in the TestNav7 platform have been worked out during PAWS testing, McBride said. When other states begin using the platform this month, they have Wyoming to thank for what should be a smoother ride.

Reach education reporter Jackie Borchardt at (307) 266-0593 or at jackie.borchardt@trib.com. Read her education blog at tribtown.trib.com/reportcard

— Jackie Borchardt
Star Tribune


This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.