NAEP Achievement Levels: Inappropriate Statistics Unethically Used
Susan Notes: Bracey explains why, if "NAEP were a private company, it would be sued for false advertising. And it would lose.
On June 19, The U. S. Department of Education will release the latest reading results from the National Assessment of Educational Progress (NAEP). People should know in advance of that release that if NAEP were a private company, it would be sued for false advertising. And it would lose.
NAEP could stay out of court if it simply reported student scores. For its first 25 years, NAEP did just that.
It asked questions it thought most people would know, questions maybe half the people would know and questions few people would know. And it reported the numbers. It was purely descriptive and nonjudgmental.
But beginning in 1988, though, NAEP shifted towards being prescriptive, to specifying what students should know. To this end, it developed what it called achievement levels: below basic, basic, proficient and advanced. These levels are a disaster. They were initially tinged with an ideological bent that sought to sustain the sense of crisis produced by "A Nation At Risk." They have never been accepted by testing professionals as valid or meaningful, but school critics often use them to bash public education. It is these
achievement levels that would land a private NAEP in court.
The achievement levels paint a grim picture of American education and the Bush administration
emphasizes the levels and the portrait they provide of American public schools. The home page of the U. S. Department of Education currently states that "Just 32% of fourth graders read proficiently." Bush reading czar, Reid Lyon claimed (falsely), that the results of a recent international study in reading showed that only about 30% of American fourth grader can't read at all. In his May 5, 2003 column for the Wall Street Journal, former Delaware Governor, Pete DuPont also claimed baldly that "When the NAEP test scores were released in 2001, only 32% of American fourth-graders could read proficiently or better." All of these judgments are rendered solely on the basis of the NAEP achievement levels.
When people with no ideological axes to grind have studied the achievement levels, they have invariably rejected them. The General Accounting Office, the Center for Research in Evaluation, Standards and Student Testing, the National Academy of Education and the National Academy of Sciences (NAS) have all concluded that the achievement levels simply do not work.
NAS had this to say: "NAEP's current achievement level setting procedures remain fundamentally flawed. The judgment tasks are difficult and confusing; raters' judgments of different item types are internally
inconsistent; appropriate validity evidence for the cut scores is lacking; and the process has produced unreasonable results."
Whew. Back to the drawing board for the achievement levels, right? We won't let the U. S. Department of Education publish such misleading numbers, right? Wrong. Incredibly, the achievement levels can be used in NAEP reports as long as the Commissioner of Education Statistics states that they have only
"developmental status," whatever that means. Pete DuPont does mention any such status. Neither does Reid Lyon or the Department of Education. Nor do critics who use the achievement levels to pummel public schools. They all treat the levels as platinum standards. You have to delve deep into the formal NAEP reports to find any discussion about their "developmental" nature.
What about NAS' conclusion that that the levels produce "unreasonable results?" By this the NAS means that NAEP results don't accord with anything else. For example, in the most recent international study of reading, only three of 35 nations (Sweden, Holland and England) significantly outscored American kids. Only three. The National Academy of Sciences observed that "when American students performed very well on a 1991 international reading assessment, these results were discounted because they were contradicted against the flawed NAEP reading achievement levels the following year. American kids finished second in this study and our best readers outscored the best readers in the country with the highest average score (Finland). Yet, NAEP, DuPont, Lyon and others claim only 32% of our fourth graders read proficiently.
But overall results like the preceding are also misleading. The U. S. has a larger proportion of kids in poverty than any other developed nation. Indeed, no other developed nation even comes close. In the most recent international comparison, the figure for the U. S. was 22%. For second-ranked Australia, 14%. For Sweden, 3%. The impact of poverty on reading scores is stunning.
In the 2003 reading study, the international average score of the 35 nations was 500 and the top-ranked Swedes garnered a score of 562 while American students attained 543. But look at American scores by poverty level:
Percent of students
In a school Score
Less than 10 589
10 to 25% 567
25 to 50% 551
America overall 543
50 to 75% 518
75% and up 485
Kids in low poverty schools outscored the top country. Kids in U. S. schools with 25 to 50 percent poverty scored 551. If these students constituted a nation, it would rank fourth in the world. The top three categories, world class all, make up 58% of all students so we're not talking small numbers here.
These results are wholly consistent with results from other international studies such as the Third International Mathematics and Science Study (TIMSS) and 2001's Program of International Student Assessment (PISA). They, too, reveal the NAEP achievement levels as "unreasonable." For instance, in TIMSS, American 4th graders were above average in math and third in the world in science among students from 26 nations. And yet, on the NAEP math and science assessments from the same year, only 21% and 29%, respectively, scored proficient or better.
How can one reconcile such fine finishes in international comparisons with such dreary results on NAEP?
One can't. How can a group of kids only 29% of whom are proficient in science rank third in the world in science? They can't, but it is the NAEP achievement levels that are off. Although NAEP can't be sued, the U. S. Department of Education should be ashamed to report the NAEP achievement levels. For the media to then slavishly parrot the Department's flawed statistics is equally reprehensible.
Using NAEP achievement levels was bad enough before the No Child Left Behind (NCLB) law. NCLB makes things worse. It requires that all children test at the proficient level or better by 2014. According to the law, each state defines "proficient" but no one thinks that provision will last because "proficient" means
many different things in different states. NAEP, which NCLB makes mandatory for the states, will be the common test, the sole means for determining "proficient."
In his presidential address to the American Educational Research Association in April, Robert Linn of the University of Colorado pointed out that if we continue to raise NAEP at the same rate as in the last decade, we can anticipate meeting the NCLB proficiency requirements by 2060 in 4th grade, 2067 in 8th grade, and 2169 in 12th grade. Linn is among those who have deemed the NAEP achievement levels "unreasonable."
The Department of Education should convene a panel to make needed changes in the achievement levels and, if the panel fails to find valid alternatives, which is likely, abandon them entirely. In the meantime, they should not be used. At all.
Gerald W. Bracey
NAEP Achievement Levels: Inappropriate Statistics Unethically Used
INDEX OF RESEARCH THAT COUNTS