Orwell Award Announcement SusanOhanian.Org Home

NCLB Outrages

Do state tests make the grade?

By Pauline Vu

Itâs hard to overestimate the importance of standardized tests in public schools today. Grade advancement, high school diplomas, teacher bonuses, principalsâ jobs and school reputations can all hinge on whether a student picks the right answer.

So who creates the tests that carry so much weight?

Much of the work is done by five giants: CTB/McGraw-Hill, Educational Testing Service, Harcourt Assessment, Pearson Educational Measurement and Riverside Publishing. Together, the companies own about 90 percent of the state-testing business, which has become a $1.1 billion industry since passage of the federal No Child Left Behind Act in 2001. The law, which took effect in January 2002, requires states to give annual reading and math tests to third- through eighth-graders, and to test students in those subjects once again in high school.

Working with state educators, the big five â or big four, once Pearsonâs planned acquisition of Harcourt takes place â create and score the tests. But the explosion of testing and changes in the types of tests states administer have left the companies scrambling to keep up.

Also, differences in state standards that are used to create the tests and the reluctance of some states to spend money for high-quality, challenging tests have caused a great disparity in testing from state to state.

For example, a look at various fourth-grade reading tests shows wide differences. Texasâ 2006 reading test[pdf file] is entirely multiple choice. Ohioâs 2005 test includes several short-answer questions, such as asking for the main conflict in a passage; in another section, students fill out a cause-and-effect chart for a certain problem. Massachusettsâ 2007 test was arguably the most rigorous: Students had to answer four long open-response questions.

Some states have fewer questions that test writing skills. A main reason for that is money. Gary Cook, Wisconsinâs former testing director, said it could cost a thousand times more to score an essay question than a multiple-choice question. In 2005, the Editorial Projects in Education Research Center reported that 15 states relied entirely on multiple-choice questions in their reading and math tests. Some of these states gave separate writing tests in certain grades. (On Jan. 9, the center reported that 12 states use only multiple-choice questions on their math and reading tests.)

âPeople who donât have their heads stuck in the instruction donât realize itâs not cheap to do this really well,â Cook said of test-making. âAnd right now, I donât know many legislatures that are very open to spending money or raising taxes to develop these kinds of instruments.â

Last year, the federal government gave states $407.6 million to help pay for testing. But states have said that falls short. In January, a federal appeals court revived a lawsuit that charges the federal government does not provide enough money for states and districts to meet the lawâs requirements.

In the 2006-07 school year, Virginia spent in state and federal money about $11.92 per test, while Washington state spent about $17.74 per test. But these amounts also include the cost of additional tests the states administered that were not required by No Child Left Behind.

South Carolina doesnât even calculate a âper pupilâ number because not every student is tested. Despite these disparate ways of measuring spending, one thing is known: Of the total education budget combining federal, state and local money, less than one quarter of 1 percent goes to testing.

âStates are not putting any more resources into the testing infrastructure, and as a result, we are getting testing on the cheap, and that is working against No Child Left Behindâs efforts to produce high-quality assessments that promote higher standards,â said Thomas Toch, the co-director of Education Sector, a nonpartisan think tank. âIf weâre going to make tests the driver of quality in public education, then we need to invest to ensure that we get tests that are up to that task.â

On the whole, however, state spending on testing has shot up since George W. Bushâs education plan became the law of the land. In early 2001, a year before No Child Left Behind was enacted, states collectively spent almost $423 million on standardized tests, according to a Stateline.org report. During the 2007- 08 school year, states will spend almost $1.1 billion on these tests, according to Eduventures Inc., an education industry research firm.

The costs have been driven up by the sheer volume of testing required by the law. In 2005-06, when states had to have math and reading tests in place for all the required grades for the first time, about 45 million tests were administered throughout the country, 11.4 million more than the previous year. This year, states are required to add a science test, which is expected to add another 11 million tests to the total.

One side effect of the greater demand for tests is a shortage of educational professionals with a jawbreaking job title: psychometrician. Their task is to oversee test creation, administration and scoring. Competition for them is fierce among the test companies, and theyâre often lured away from one company by another.

âA psychometrician has to make sure that algorithm is absolutely correct so that a student who is just barely going to pass, doesnât just barely fail,â said Dr. Michael Bunch, a psychologist and senior vice president of research and development at North Carolina-based Measurement Inc., which has contracts with a dozen states.

The increased emphasis on testing has caused other pressures as well. States are now testing later in the academic year to squeeze in more teaching. And, states want scores back faster than ever before.

Not surprisingly, this has caused some high-profile errors. In 2004, CTB/McGraw-Hill had to re-score thousands of Connecticut tests after scores came in mysteriously low because of the grading on the writing section. In 2006, the late distribution of tests to Illinois by Harcourt meant a long delay in getting the scores back to the state. Last year, American Institutes for Research had to re-grade 98,000 Hawaii tests after teachers found that some students who submitted blank test books received scores anyway.

Probably the biggest impact of the No Child Left Behind law has been on the kind of tests states are giving, which has changed dramatically. The law, which imposed so much federal intrusion into local classrooms, passed only with a compromise: States would be allowed to create the tests.

The result was states switched from standardized tests that compare how their students stack up against students across the country â the Stanford Achievement Test is a prominent example â to those based on each stateâs specific standards. But this system allows the difficulty of the tests to vary widely from state to state, resulting in some states producing easier tests that measure lower-level skills.

A report released in October by the conservative Thomas J. Fordham Foundation studied testing in 26 states by comparing studentsâ proficiency on two tests â their state tests and the Measures of Academic Progress (MAP) test, administered by the Northwest Evaluation Association (NWEA).

The results were stark: Differences in student performance from state to state were significant. According to NWEAâs analysis, to be considered proficient readers in Wisconsin, for example, fourth-graders needed to answer questions about as difficult as one that asked them to note a few differences between cats and dogs. But fourth-graders in Massachusetts faced more difficult questions such as those about a written passage by Russian author Leo Tolstoy.

Wisconsin Department of Public Instruction spokesman Patrick J. Gasper called the Fordham study flawed and pointed out that the two tests compared in the report have different goals and measure different skills.

âTheir report attempts to draw conclusions by comparing two different types of tests that are scored on two different scales,â Gasper said. âThe purpose and types of tests being compared are different.â

Test difficulty can be measured in a variety of ways. One is by looking at how tough the questions are. Again, a review of various fourth-grade reading tests shows that reading passages on Massachusettsâ test are lengthier and appear to be more detailed and use more difficult vocabulary than the passages on Ohioâs and Texasâ tests. One question on Ohioâs test simply asks students to identify who is speaking in the passage.

But that is an imprecise way to determine a testâs difficulty, said John Cronin, a research specialist at NWEA and one of the Fordham studyâs lead authors. Several factors could make a reading question difficult, such as the length of the passage, how straightforward the question is and the quality of the wrong answers in multiple-choice questions. Test creators can determine how difficult a question is only after they see how it has performed in field tests with real students.

And even if the questions themselves are challenging, a testâs difficulty can be misleading if the state sets a low cut score, or the number of questions a student must answer correctly to be deemed proficient. The Fordham Foundation report found that statesâ definition of âproficientâ based on the cut score was far from consistent. Cronin said setting low cut scores would be like saying a high school baseball player is proficient only if he can hit against Major League pitcher Roger Clemens â an admittedly high performance standard â but then requiring batters to get only one hit out of 50 pitches to meet that standard.

âInstead of a .300 (batting average), theyâre aiming for .020,â Cronin said. âItâs a very rigorous test, facing a very difficult pitcher, but the standard of performance is very low.â

These test disparities and the wide variety of tests has fueled calls for national standards.

âYou may be deemed proficient in North Dakota math or Wisconsin math, but that doesnât mean youâre proficient, really, in math,â said Fordham spokesman Jeffrey Kuhner. âIf you want to prevent this varied discrepancy from state to state, rather than dumbing down our standards we believe that the way to do it is to have an across-the-board national test and national standards.â

International comparisons show that students in several countries have caught up to â and passed â American students. The results from the 2006 Program for International Student Assessment, which compares how 15-year-olds in 57 countries perform, showed U.S. students were 29th in science and 35th in math, behind counties like Estonia, Slovenia and Latvia.

As Congress works to authorize No Child Left Behind, several experts are promoting national standards, but the concept seems unlikely to advance in the face of staunch opposition by local-control advocates. When Fordham released an analysis showing how national standards could come about, it was titled, âTo Dream the Impossible Dream.â

A bipartisan discussion draft proposing changes to NCLB released by key U.S. House members â most notably Rep. George Miller (D-Calif.), the chairman of the House Committee on Education and Labor and one of the prime movers of the original No Child Left Behind bill â would not require national standards or a national test. The draft, released in August, would give states incentives to work with universities and the business community to develop more rigorous standards and tests based on those standards.

âTo be successful, our system of accountability must encourage states to set high standards,â Miller said in September at a House hearing. âLowering the bar so more students can reach it is a sham.â

— Pauline Vu


This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.