Orwell Award Announcement SusanOhanian.Org Home


NCLB Outrages

No Single Explanation For Md. Test Score Bump

Researchers say there are many ways, intentionally or inadvertently, to skew a test: Replace difficult questions with simple ones. Assign more weight to easier questions to exaggerate small gains by weak students. Lower the passing score, so a student who gets half the questions right is judged a success.

And note that the real point gets slipped in toward the end: The call for a national test.


By Daniel de Vise

Maryland educators this month celebrated a major jump in test scores, with achievement gaps narrowing and pass rates rising six percentage points in reading and four points in math. Then skeptics crashed the party.

The revelation that this year's Maryland School Assessments were a half-hour shorter than last year's raised suspicions among researchers who thought the scores were too good to be true. Here, some thought, was the smoking pencil.

The episode illustrates a basic disagreement within the education community over why scores are rising across the nation since the 2002 enactment of No Child Left Behind, which sets a goal of 100 percent proficiency by 2014: Are kids getting smarter, or are tests getting easier?

"The Congress has told governments and state school officials that all children must be magically proficient by 2014," said Bruce Fuller, a professor of education and public policy at the University of California at Berkeley. "They're finding ways to make sure everybody creeps toward universal proficiency."

Maryland officials removed a section of multiple-choice questions from state reading and math tests this year, shortening each from roughly three hours to two and a half. They did not publicly announce the change, although the 24 school-system superintendents were apprised in a June 2007 memo.

State Superintendent of Schools Nancy S. Grasmick said she did not even recall the change when she made the results public July 15. State officials contend that the revision had no bearing on results because only a few deleted questions counted toward student scores, and those were replaced.

"The 2008 results are absolutely comparable to every previous year back through 2003," said Ronald A. Peiffer, deputy state superintendent.

In the No Child Left Behind era, it is unusual for questions to be raised about state test results, based on changes to an exam that had not been publicly announced.

Researchers say there are many ways, intentionally or inadvertently, to skew a test: Replace difficult questions with simple ones. Assign more weight to easier questions to exaggerate small gains by weak students. Lower the passing score, so a student who gets half the questions right is judged a success.

Illinois, Missouri and Arizona all have publicly lowered passing scores on their tests, yielding higher pass rates. California officials shuffled the order of questions on the third-grade reading test two years ago, out of concern that the first question students saw was overly complex.

Virginia's Board of Education eased passing scores on several history and social studies tests in 2001. A subsequent Washington Post analysis found that the changes were responsible for about half of the increase in schools meeting state accreditation standards in 2002. State education spokesman Charles Pyle said there has been no easing of the tests since then. One significant change in 2006 actually made some tests harder, he said.

D.C. education officials say their testing system, introduced in 2006, has never been altered in a way that could make it easier. Scores in the District rose notably this year.

Recent reports from the Thomas B. Fordham Institute and Fuller's group, Policy Analysis for California Education, have concluded that most recent gains on state tests are illusory, reflecting better test-taking skills or lower standards rather than increased knowledge. Another study, from the Center on Education Policy, concluded that the gains seemed genuine but did not necessarily reflect greater learning.

The reports compared state results with other tests, such as the National Assessment of Educational Progress and Measures of Academic Progress, which suggest that academic skills are improving at a slower rate, if at all.

"When the arrows don't point in the same direction, you have to at least ask yourself what's going on here," said Chester E. Finn Jr., Fordham Institute president.

Maryland's controversy came as schools reveled in a sense of collective achievement after six years of testing. The overall pass rate on the exams, given in grades 3 through 8, reached 82 percent in reading and 76 percent in math this year, more than 20 points higher than proficiency rates in 2003.

Until this year, Maryland's tests comprised two distinct sections. One was a series of multiple-choice and written items -- about 35 in reading and twice that many in math -- that measured students against state standards. The other was a series of purely multiple-choice items culled from commercial tests, administered to yield nationally normalized percentile scores for Maryland students. The state deleted the second section, judging it irrelevant because the questions were not derived from Maryland's curriculum.

An independent panel of psychometricians validated the revised test, affirming that it was neither easier nor harder than last year's, although the group spent a few minutes in a conference call pondering unusually large gains in certain grades. (The share of fifth-grade students rated "advanced" in reading, the highest of three performance levels, rose an unprecedented 18 points.)

Some panel members say the changes might have contributed to the higher scores. How a student performs on a test item depends partly on what comes before and after, factors that could affect concentration or confidence.

"So we have to ask . . . 'What are the effects of context on student scores?' " William Schafer, a University of Maryland professor emeritus who is on the panel, wrote in an e-mail. "Not much is known in the public literature about that question."

When Grasmick and Peiffer reviewed the scores a week before their release, neither questioned the gains. The results made sense, Peiffer said: They were driven by large increases in historically low-performing Prince George's County and Baltimore, systems with dynamic leaders and well-documented reforms.

"It's not as if these are results we weren't expecting," Peiffer said.

Nonetheless, some in the research community are pushing for an overhaul of the national testing apparatus. A simple fix: Require states to announce any change that might affect scores. A more radical solution: a national test, immune to state manipulation.

"I think most people are trying to do the right thing," said Jack Jennings, president of the Center on Education Policy. "But the pressure to get results is enormous, and some people fail. Some people sin."

— Daniel de Vise
Washington Post
2008-07-27


INDEX OF NCLB OUTRAGES


FAIR USE NOTICE
This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.