Part 2: How Does NAEP Label a Reader

Susan Notes:

This research provides the inside dope on media headlines screaming, "NAEP Finds 71% of 4th graders score below the proficient level." This is important because corporate politicos are pushing for NAEP to become the national test.

Down the Rabbit Hole with the Reading Passages

Of the eight items NAEP posts as samples from 4th grade reading tests over the past decade, topics include an American female astronaut on Mir, crab hunting, wombats, and life in the American Colonies. Two items, a West African tale and a pourquoi story from William Bennett's edited collection The Moral Compass,; are in the folklore genre. There are two stories about rural children and their dogs. Highlights magazine holds the copyright on half of the items. NAEP provides online copies of the items and an analysis of the questions and responses. NAEP's Content Classification purports to explain what is being measured: Reading for Literary Experience, Forming a General Understanding, Developing Interpretation, Making Reader/Text Connections, Examining Content and Structure. (see Appendix)

NAEP asserts, "To reach the goal of approximating actual reading experiences, NAEP reading passages are typical of those read by students every day. The passages are taken from authentic texts found in the environments of students in grades 4, 8, and 12. Selected passages are original sources, not simplifications, paraphrases, abridgments, or modernizations."

http://www.nagb.org/pubs/reading_Framework/ch2.html [no longer available at this NAEP address]

Selections typical of those read by students every day? Ask yourself which children have subscriptions to Highlights ($26.04 a year) and read selections from William Bennett's The Moral Compass($35 new but available from Amazon.com used books for $.39). Ask yourself about children's access to these materials: California, to name one state education system, gave up on the idea of staffing schools with professional librarians decades ago, in many cases giving up on the idea of even libraries, and across the nation public libraries, when they can even remain open, are cutting back on hours and book budgets. For documentation on library disparity, take a look at Prof. Stephen Krashen's research, as discussed on Jim Trelease's website:

http://www.trelease-on-reading.com/rah_chpt6_p1.html [no longer available at Trelease. See http://www.sdkrashen.com]

NAEP declares: "The selected passages for each grade are developmentally and topically appropriate. In addition, passages are intact and meet criteria for genre, language, interest, and organization. These criteria include curricular considerations, appropriateness of the topic, style, perspective, and fairness. Documents are genuine and related to tasks that are appropriate for the grade level and experiences of the students being assessed."

http://www.nagb.org/publications/frameworks/r_framework_05/ch2.html

Fourth graders may indeed be interested in many things, even crab hunting and wombats. But a government that subjects children to testing that will be trumpeted as indicating the quality of their schools, should surely include a few items that break out of the nostalgia/rural/agrarian genre, stories and articles closer to their own backgrounds and experiences. One can make the argument that reading about astronauts, crab hunting, wombats, life in the American Colonies, and rural children and their dogs might not be appropriate in topic, interest, perspective, or fairness. For those who think a West African folktale satisfies an ethnic consideration, consider the specialized language and format of such a tale. Teachers do very well teaching this genre and the rest of the nostalgia genre, too, but what is appropriate for a classroom discussion doesn't give students full opportunity on a test, when they must do the whole thing solo.

According to NAEP, "Passages range in difficulty from those that could be read by the least proficient readers (e.g., about second-grade level in a fourth-grade class) to those that could be read by only the most proficient readers (e.g., possibly eighth-grade level in a fourth-grade class), as determined by teachers in specific grades." So when NAEP issues their proclamations about proficiency levels, 71 percent below proficiency nationwide, where do these eighth-grade level passages fit in? After all, fourth graders not proficient on eight-grade level passages is hardly surprising. But this little detail seems to escape media scrutiny. Always, always we must ask: When we look at NAEP proficiency levels, what are we seeing?

NAEP notes that one of their categories, Examining content and structure, requires "critically evaluating, comparing and contrasting, and understanding the effect of such features as irony, humor, and organization." Indeed. One can ask how many members of the National Assessment Governing Board (NAGB), recognize irony in literature. Or in life. That this peculiar group of people has been put in charge of reading assessment for the nation is, to say the very least, odd.

For fourth graders, the most tested quality is Reading for Literary Experience; 55% of the questions are in this domain, which NAEP describes thusly: "The reader brings his or her experiences and knowledge to the text in such activities as anticipating events, picturing settings, predicting consequences, analyzing actions, and considering the language of literary works. The reader thinks about the perspective of the authors and characters and considers the language and story structure when reading for literary experience. Various types of texts are associated with reading for literary experience, including novels, short stories, poems, plays, legends, biographies, myths, and folktales." (emphasis added)

It seems more than obvious to note that children who vacation at the beach, visit museums, and whose home and public libraries are well supplied with a variety of books will bring more experiences to passages about crab hunting and American colonists than kids who live in inner city high rises or in rural isolation. And as schools pursue the relentless path of test prep to satisfy NCLB requirements on state tests, children of poverty are shortchanged even more. They don't have cultural experiences testmakers value and measure, their Reading First schools rob them of intellectual experiences, and tax dollars no longer support public libraries.

NAEP breaks down data on each question in every way you might imagine: sex, region of country, race/ethnicity, public nonpublic (5 categories), national school lunch eligibility, location (central city, urban fringe, rural). So you can find out that more girls than boys correctly answered the questions of where wombats live. Since students in the Northeast, South, and Midwest outscored students in the West on the question Where do wobats live, Standardistos in the West can form a wombat habitat deficit course of study and NCTE and IRA can apply to the Carnegie Foundation for money on wombat-habitat-deficit coaching models.

Our professional associations have a lot to answer for.

The Passages NOTE: All the released reading passages are online at the NAEP website, but one must access them through the NAEP site. You can assess story texts through the questions site. Once you pull up a question, click on View Reading Passage. Here is the question site:

http://tinyurl.com/ba426 [no longer available at NAEP]

"How the Brazilian Beetles Got Their Coats" retold by Elsie Eels (from The Moral Compass, edited and with commentary by William J. Bennett, Simon & Schuster. 1995) offers a small example of how background knowledge and experience informs the text. It would be interesting to ask children to explain the meaning of "set" in this sentence: In Brazil the beetles have such beautifully colored, hard-shelled coats upon their backs that they are sometimes set in pins and necklaces like precious stones. . . . Children of relative affluence are more likely to know about "setting" jewels, particularly when "set" is used differently a few paragraphs later: The bright green and gold parrot set the royal palm tree at the top of the cliff as the goal of the race.

And if a fourth grader does understand the meaning, we can guarantee he will be grossed out by the concept.

But I admit that for me this "set" variance was small potatoes compared with green mangoes with golden flushes on their cheeks lying on the ground under the mango trees. I don't know anything about mangoes and learning they have cheeks stopped me cold in my reading. My husband didn't know about mango cheeks either, dismissing my question with "You know what they mean." But neither one of us quite knows what a mango looks like. An Internet search reveals I can buy frozen mango cheeks and cook a coconut, mango, and lychee terrine. I even found a picture of a mango cheek. My point here is that as a reader this term shut down my reading for about 15 minutes. Test takers can't afford to stop and ponder.

NAEP gave "Full comprehension" to just 35% of children answering this constructed response probe: Choose someone you know, have read about, or have seen in a movie or television show who is like the rat or the beetle. Explain how that person or character is like the rat or beetle.

Full Comprehension: "I am like the beetle because I am well-mannered and not greedy. My friend is like the rat because she thinks she is the best at everything."

Scorer's Comment: The student compares him/herself and a friend to both the rat and the beetle, using the appropriate character traits as the basis of comparison.

Evidence of little or No Comprehension: "The hare is like the rat."

Scorer's Comment: Response mentions a character that is like the rat but does not explain the likeness with a text-based characteristic.

Now why would this child have chosen a lead character from The Tortoise and the Hare if he had little or no comprehension of the text at hand? Why didn't he compare the beetle or rat to a character in The Velveteen Rabbit or Captain Underpants?

NAEP rated 71% of the responses Unacceptable on this question: Do you think that the beetle is being sneaky by not telling the rat she has wings? Use information from the story to explain why or why not?

Unacceptable: "Yes I do because she is not telling the rat she has wings."

Scorer's Commentary: Response provides an opinion but the support merely repeats the question.

Sometimes scorers reward students for repeating the question; sometimes they don't. Note that the student who declares herself well-mannered gets full credit but the one who finds a story heroine sneaky does not.

A frequently-used prompt asks readers to suggest a different ending to the story. These two answers for "The River" by Yetti Frenkel received only partial credit: "Minny and/or Cory might have drowned" and "The dog might have drowned," because, in the scorer's words: Both responses provide an appropriate opinion of how the story might have ended differently, but neither supports the opinion with evidence from the story. Thus, although children's answers may exhibit a certain lack of sophistication (or maybe they exhibit the virtue of Robert Frost's matter-of-fact hard floor), even the scorer acknowledges they don't exhibit lack of understanding of the reading selection. And so, once again, people viewing NAEP results on the reading proficiency of American fourth graders need to be aware of just what is being tested. It isn't reading comprehension but the ability to pull off a sometimes sophisticated, sometimes formulaic integration of textual elements into one's response. Many young responders don't repeat elements from the text because they take the text for granted. It's sitting there: why repeat it? A fourth grader who writes "The dog might have drowned" flunks. If he's written "The dog might have drowned because the kids couldn't reach him" he gets full credit.

Similarly, when the prompt asks "Suggest a new and different title for the story and explain why it would be a good title," the student who summarized the events of the story in his title "How the Brazilian Beetle Won the Race" and offered this justification--"because I like it"-- received only partial credit because, according to the scorer, "the response offers no support for why it would be a good title." Since when is a writer's gut feeling not allowed? And again, what is being tested? Surely not whether the student understood what he read. It is clear he did. When a student fails to get full credit for this answer, someone has to admit that there's another agenda here. What's being tested is not reading comprehension but English major-type ability to supply textual evidence to support right answers. Anyone who teaches knows that children's writing sophistication lags behind their reading level. NAEP ignores this, choosing to label lack of writing erudition as deficient reading comprehension. The NAEP mantra seems to be: If you can't--or choose not to--write about it in full detail, you don't know it.

When children were asked, Suggest another title for the story [The River]. Use examples from the story to explain why it would be a good title, only 19% of respondents received full credit. The scorer gave only partial credit for "'Danger on the Ice' because there is danger on the ice," complaining that this is a circular explanation. But full credit was given to "'The Icy River' because the dog got stuck in the icy river." Go figure. And remember: this kind of arbitrary and capricious judgment determines NAEP proficiency ratings, those ratings broadcast in media headlines across the nation: NAEP Finds 71% of 4th graders score below the proficient level. For New Mexico it was 80%. Does this mean their 4th graders can't read, or does it mean they weren't given credit for their alternate titles for "The River?" This is a critical question, a question ignored by:
a) The Media
b) Educators
c) Researchers
d) Professors
d) Think Tanks
e) All of the Above

In another constructed response probe for "The River," children are asked, "Do you think this story was exciting? Use an example from the story to explain why or why not." Children who answered "Yes, because every good story has some danger"and "Yes, because I like rescue stories" were given only partial credit. Here is the scorer's reasoning: Both responses provide an opinion and support it with a general reference to the story. The first response claims that the story has some danger, but does not use examples to describe what is dangerous. The second response provides a general reference to the genre of the story that is not explained in terms of story events.

Are we looking for pre-trial lawyer skills here or evidence of whether children understood the story? Young readers who write "because every good story has some danger" are taking it as a given that the scorer knows that they know the story has danger in it. Shouldn't the word because be worth something here? Fourth graders aren't used to building an evidentiary case to prove their reading savvy, and there are two reasons why schools should not go into the business of training them to do so:

1) It would mean a whole lot less time spent on reading.

2) It would kill the joy of reading for reading's sake.

On Thin Ice

Another constructed response probe asked, What was one of the most important lessons that Cory and Elisa learned from their experience? Here are samples of responses that received full credit and those that didn't:

Evidence of Full Comprehension:
1)"Cory and Elisa learned that they must not give up hope and keep on trying." 2) "One of the most important lessons they learned was that by working together they could save Minnie."

One doesn't have to read many answers labeled "Evidence of Full Comprehension" to recognize which 10-year-olds have been trained to repeat the question in their answer and which haven't. Teachers, even knowing this is what the Standardistos want, should be very reluctant to deliver it. Teaching in our times means following Henry David Thoreau: Resist much. A surefire method of killing any joy in reading for children is to require them to grind out these answers for everything they read. Too many of the Full Comprehension answerers march along like Stepford byproducts, leading the reader to start checking penmanship: most of the "Full Comprehension" answers sound like they were written by the same person. Teachers need to come to grips with the reality that sometimes "full credit" from the testing marauders comes at too high a price.

Evidence of Partial or Surface Comprehension:

1) "Never walk on thin ice." 2) "Keep an eye on the dog."

Scorer's Commentary: Both responses identify a superficial lesson that conveys an understanding of a trivial aspect of the story.

Trivial aspect of the story! Indeed. For children who live near bodies of water that freeze in winter, the lesson about walking on ice isn't trivial at all. Besides that, I must admit I feel a kinship to kids who insist on avoiding all frippery and cutting to the chase. They remind me of my first year of teaching--when on a test I asked students what lesson they learned from Silas Marner. I gave full credit to the student who wrote, "Hide your money in a safe place," feeling she'd put me in my place. And I confess I don't remember any of the other student answers. Standardistos will label such answers naive and incomplete; the rest of us, recognizing there are many levels of comprehension, must be willing to give credit to students who go for such basics. It would do testocrats well to remember that only in the school world do English major-type answers hold sway. The real world is more pragmatic. And since the world needs pragmatists as well as philosophers, it does us all great harm to label pragmatists as failures.

The plot of "The Box in the Barn" by Barbara Eckfeld Conner involves a young boy who lets a surprise puppy for his sister's birthday out of a box, and the puppy runs off. He's very worried because he'd been told not to touch the box. Fourth graders are asked, If the box had been empty when Jason opened it at the party, what would Jason most likely have said? Give examples from the story that support your answer.

This student response was rated unacceptable: "I think he would say, 'I didn't do it.' That's what most people would say." The student responder even included quotation marks for the direct quote.

Scorer's Commentary: The response doesn't use information from the story to predict Jason's reaction. It provides dialogue that does not demonstrate understanding of Jason's honest character.

The fact is that the reader has gone beyond the story to make an observation about human nature. The scorer's reaction seems especially bizarre when one considers another response that received full credit: "Maybe he might have just kept what he had done a secret. Because he had been trying to act normal." Here, the scorer says, The response uses information from the story to predict how Jason might have reacted if the box had been empty. The response predicts that Jason might have kept it a secret, based on the part of the story when he pretended to act like everything was fine when the party began.

Can we tell from these scorer judgments that one child is a more proficient reader than the other? Again, NAEP's declarations about non-proficiency are worse than suspect.

Another consideration must be whether the children don't feel oppressed by the incessant writing tasks. This quite insipid story, 1029 words long, is weighed down by 12 questions, 7 of which require constructed responses providing evidence from the story:

Why did Jason think everyone would be angry with him when they found the puppy missing?

If the box had been empty when Jason opened it at the party, what would Jason most likely have said? Give examples from the story that support your answer.

Why do you think Jason�s father explained how he found the puppy?

Describe how Jason might have felt if the box had been empty when it was opened at the party. Explain why he might have felt that way?

Why did Jason not answer when his mom called him for dinner?

From when Jason got up in the morning until he went to bed that night, his feelings changed as different things happened. Describe three different feelings that Jason had and explain what made him have those feelings.

What did Jason's father think that Jason had learned? Tell what in the story makes you think so.

Whew! Someone take pity on these fourth graders required to deconstruct so many motives and responses to one limp story.

Of Fares and Fishes: Grade 8 Reading Passages

The 8th grade test introduces a new category, Reading to Perform a Task. The sample-- Tiny print filling two pages is indeed as indecipherable as any fare guide I have ever seen. Certainly, students' ability to scan for information, a useful skill, and maybe it's even a relief from the relentless literary probes. But even with this practical task students are required to write mini-essays to constructed response probes such as What is the guide to Metro's Fares and Passes supposed to help you do? And a literary response is expected.

These two are judged Partial or Surface Comprehension:

1) "The guide to Metro's Fares & Passes is supposed to help you get around when visiting the city."
2) "It's supposed to help me by answering all the questions I have about Metro's fares and passes. It's an informative guide to help the customer."

Scorer: The first response provides a general description of what the Metro guide can help you do, demonstrating only partial understanding of its purpose. The second response goes further by indicating that the Metro guide can help answer questions in general, but does not specify what those questions are.

Little or No Comprehension:
"It's to help you with Metro's fares."

Scorer: The response merely repeats several words from the question and demonstrates no understanding of what the Metro guide can help you do.

I can't see that the response labeled Full Comprehension is that much different from either Partial or Little Comprehension, just more verbose: "The Guide to Metro's Fares & Passes is supposed to help you understand how to get around the city by knowing what the fares are, what is needed to travel between Metrobus and Metrorail, and help people enjoy their visit more."

Scorer: The response demonstrates understanding of the Metro guide's intended purpose: providing readers with fare and travel information.

Obviously, what a fare guide does for you depends on what you're looking for, providing at the same time lots of information most people don't care about. I would congratulate all responders for not being overwhelmed by the massive amount of detail in the brochure: Metrobus school fares, children ride free, Metrobus fares, Metrorail fares, Metro fare replacement policy, Transfers, and on and on and on. To cut through all this detail and come up with a summary statement is an accomplishment. But if you wanted to judge comprehension of such a text, wouldn't a multiple choice question have been better?

Another item from the 8th grade test, "The Fish" by Elizabeth Bishop, offers yet another example of relentless essay questions which are so similar to one another. Since NAEP samples different parts of the test in different schools, one can only hope nobody had to answer all these questions.

I caught a tremendous fish
and held him beside the boat
half out of water, with my hook
fast in a corner of his mouth.
5 He didn't fight.
He hadn't fought at all.
He hung a grunting weight,
battered and venerable
and homely. Here and there
10 his brown skin hung in strips
like ancient wallpaper,
and its pattern of darker brown shapes like full-blown roses
15 stained and lost through age.
He was speckled with barnacles,
fine rosettes of lime,
and infested<
with tiny white sea-lice,
20 and underneath two or three
rags of green weed hung down.
While his gills were breathing in
the terrible oxygen
--the frightening gills,
25 fresh and crisp with blood,
that can cut so badly--
I thought of the coarse white flesh
packed in like feathers,
the big bones and the little bones,
30 the dramatic reds and blacks
of his shiny entrails,
and the pink swim-bladder
like a big peony.
I looked into his eyes
35 which were far larger than mine
but shallower, and yellowed,
the irises backed and packed
with tarnished tinfoil
seen through the lenses
40 of old scratched isinglass.
They shifted a little, but not
to return my stare.
--It was more like the tipping
of an object toward the light.
45 I admired his sullen face,
the mechanism of his jaw,
and then I saw
that from his lower lip
--if you could call it a lip--
50 grim, wet, and weaponlike,
hung five old pieces of fish-line,
or four and a wire leader
with the swivel still attached,
with all their five big hooks
55 grown firmly in his mouth.
A green line, frayed at the end
where he broke it, two heavier lines,
and a fine black thread
still crimped from the strain and snap
60 when it broke and he got away.
Like medals with their ribbons
frayed and wavering,
a five-haired beard of wisdom
trailing from his aching jaw.
65 I stared and stared
and victory filled up
the little rented boat,
from the pool of bilge
where oil had spread a rainbow
70 around the rusted engine
to the bailer rusted orange,
the sun-cracked thwarts,
the oarlocks on their strings,
the gunnels--until everything
75 was rainbow, rainbow, rainbow!
And I let the fish go.
"The Fish" from The Complete Poems 1927-1979 by Elizabeth Bishop. Copyright (c) 1979, 1983 by Alice Helen Methfessel. Reprinted by permission of Farrar, Straus and Giroux.

It is troubling to note how formulaic the answers receiving Full Comphrension credit are. Of course it's too much to ask that a respondent's "voice" come through on a test, but the NAEP correctors discourage this anyway. Students who draw on their own personal experience are penalized for not referring to the poem. In some cases answers labeled No Comprehension might actually be indicative of a deep, personal response to the poem. One wouldn't know without talking to the students. Talking with children about poetry usually raises more wonderful questions than answers, but NAEP is looking only for answers that fit their formula.

Modest Proposal: Lock up the Governing Board in a room and make them answer all these questions:

1. Describe your reaction to the poem. Explain what it is about the poem that caused you to react that way.

Performance Results: No Comprehension: 16% Partial comprehension: 38% Full Comprehension: 44% Omitted: 2%

Full Comprehension: "I liked the poem because the poet, Elizabeth Bishop, beautifully and accurately described the catching and letting go of a fish."

Scorer: The response provides a reaction to the poem and explains in specific detail ("beautifully and accurately described the catching and letting go of the fish") what it was about the poem that caused the reaction.

Partial Comphrehension: "The fish wanted to get let go and the boy saw the rainbow and let him go."

Scorer: The response provides a reaction to the poem, but the explanation offered makes only a general reference to the writing style of the poem.

Little or No Comprehension:
2) "Happy because it's a nice and happy poem."

Scorer: Both responses provide a reaction to the poem, but the explanations offered make only a general reference to the writing style of the poem.

2. Which of the following best describes the person speaking in the poem?
A) Thoughtful and observant
B. Tired and victorious
C. Grim and sullen
D. Frightened and lonely

Performance Results: Correct: 82% Incorrect: 18%

3. Why does the person let the fish go? What in the poem makes you think so?

Performance Results:
No comprehension: 25% Partial comprehension: 45% Full comprehension: 29% Omitted: 1%

Full comprehension:
"I think he let it go because it was tired and old and it was caught many many times. What made me think that was the five hooks in its mouth and the fact it didn't fight."

Little or no comprehension:
1) "I think she let it go because she's an animal lover, and she's just one of those people who won't eat fish when they catch them or like to fish just for fun."

2) "I think she let him go because he was making everything colors of the rainbow."

Scorer: The first response provides a personal opinion that has no relation to the content of the poem. The second response attempts to provide evidence from the poem, but it is illogical reasoning as to why the person let the fish go.

4. Choose an image from the poem and explain what it means.
No comprehension: 28% Partial comprehension: 34% Full comprehension: 31% Omitted: 6% Off task: 1%

Little or no comprehension:
"The rainbow means it is very colorful."

Scorer: Response refers to the plot of the poem, rather than a particular image.

What?????! The plot of the poem????

5. Explain how the language used in this poem is different from the language used in everyday speech. Use a line or phrase from the poem as an example.

Performance Results:
No comprehension: 39% Partial comprehension: 28% Full comprehension: 26% Omitted: 6% Off task: 1%

Full Comprehension:
"The dramatic reds and blacks from his shiny entrails. He uses a lot of adjectives."

Scorer: The response provides an example from the poem and an explanation of how the language in the poem differs from everyday speech because of the author's use of literary devices.

Partial Comprehension: "Fresh and crisp with blood." Very different from everyday speech. Most people would of said, "Very slimy and bloody." (quotes in original)

Scorer:The response rephrases a line from the poem into everyday speech, but does not provide an explanation of how the language differs from everyday speech.

One can only wonder what the scorer thinks "Most people would have said" means.

6. When the poet says 'Like medals with their ribbons frayed and wavering' (lines 61 & 62), she is referring to:
A) victory
B) fishhooks
C) trophies
D) fish scales

Performance Results: Correct: 53% Incorrect : 46% Omitted: 1%

7. From the time she catches the fish until she lets it go, the speaker's feelings about the fish change. Tell how her feelings change and what causes them to change. Explain your answer by referring specifically to the poem.

Performance Results:
Unsatisfactory: 6% Partial: 36% Essential: 38% Extensive: 14% Omitted: 5% Off task: 1%

The student replies labeled "Extensive" are just that: these kids are on track for Ph.D.s in English.

Essential: "When she first catches the fish, she is happy about it. When she reels it in, she begins to examine and notice the condition of the fish. She sees his battered skin, his cut gills, his discolored eyes, and the other hooks attached to his jaw. She begins to feel sorry for the fish, who stares blankly at her, presenting no feelings. She sees the rainbow colored oil around the boat and lets the fish go."

Scorer: The response provides information from two points in the poem that cause the speaker's feelings to change and identify appropriate feelings associated with those points, but this response fails to extend the feelings into a larger realization of why the speaker decides to let the fish go.

Partial:
1) "She changes her feelings when she looks into its eyes and at the fishhook."
2) "They change from her feeling happy for the fish to feeling bad and scared for the fish."

Scorer: The first response provides a point in the poem for what may have caused the speaker's feelings to change, but doesn't explain or identify the feelings. The second response provides ways the speaker's feelings changed, but doesn't refer to any points in the poem that may have caused those changes.

8. Reread the lines beginning with "I admired" (line 45) and ending with "aching jaw" (line 64). What do these lines tell you about the fish's experience?

Performance Results:
No comprehension: 21% Partial comprehension: 48% Full comprehension: 29% Omitted: 1%

Partial:
1) "This tells me the fish was in pain and he wasn't very happy."
2) "That it has got caught a lot of times."

Scorer: Both responses provide an understanding of a part of the fish's experience.

Little or no Comprehension:

1) "She thought proudly of the fish. She knew how it felt to have an aching jaw."
2) "the hooks."

Scorer: The first response does not reflect an understanding of the fish's experience, but rather attempts to refer to the author's experience. The second response does not merit partial credit because "the hooks" is not enough to reflect an understanding of the fish's experience.

9. Describe how the speaker's decision to let the fish go is related to something you have experienced, read, or seen.

Performance Results:
No comprehension: 28% Partial comprehension: 42% Full comprehension: 29% Omitted: 0%

Full Comprehension:
1) "I have seen this reaction before in many inspirational movies and I have done so too by letting a captured butterfly fly freely." 2) "This relates to when I caught a lightning bug and felt sorry for it and let it go."

Scorer: The first response provides a basis of comparison ("realized that everything has feelings") and relates it to a similar experience (letting a butterfly go). The second response provides a basis of comparison ("I felt sorry for it") and relates it to a similar experience (letting a lightning bug go).


Partial or Surface Comprehension:
1) "When I go fishing I feel that if a fish is caught it should be put back."
2) "I have been fishing and me and my sister and uncle had to let the fish go. . . ."

Scorer: Both responses provide a connection to the poem with a situation that is similar to the one in the poem, but it is only the surface level action of catching and releasing a fish.

Interesting. This must be the NAEP version of elegant variation: a child gets full credit by comparing the poem to her own catching of a butterfly but not for her own catching of a fish.

One might also suspect some class bias here.

How Did This Happen?
And so, reading comprehension is assessed by writing polish. And writing proficiency is judged by a narrow formula: providing examples from the texts is a lot safer than pulling them from your own experiences. If any of this bothers you, write the members of the Governing Board. After all, as mandated by NAEP law, the Governing Board reviews the items both before and after field testing for appropriateness and bias.

Remember the House that Jack Built? A "comprehensive national process" conducted under the auspices of the National Assessment Governing Board (NAGB) produced a reading framework, which is a "broadly accepted outline of what hundreds of educators, curriculum experts, policymakers, and members of the general public thought the assessment should test." This framework describes the goals of the assessment, "providing the theoretical basis for the assessment and the directions for what kinds of exercises should be included in the assessment, how those exercises should be designed, and how student responses should be scored."

http://nces.ed.gov/nationsreportcard/reading/whatmeasure.asp

After the completion of the framework, the NAEP Reading Committee worked with measurement specialists to create the assessment questions and scoring criteria according to the framework's specifications. "All exercises and scoring criteria were carefully reviewed to ensure that the assessment met the requirements of the reading framework."

http://nces.ed.gov/nationsreportcard/reading/howdevelop.asp

The 2005 NAEP Reading Committee

Peter Afflerbach, University of Maryland; Rosalinda Barrera, University of Illinois at Urbana-Champaign; Susan Biggam, Vermont Reads Institute; Violet Harris, University of Illinois at Urbana-Champaign; Tamara Jetton, James Madison University; Janet Jones Berry Elementary School; Judith Langer, The State University of New York at Albany; Patricia McGonegal, Mount Mansfield High School; Jane Miller; The Lawrence Barnes School; Patricia Parrish; Sumrall Attendance Center; Jack Pikulski, University of Delaware; Timothy Shanahan, University of Illinois; Alfred Tatum; University of Maryland; Suzanne Wade; University of Utah; Karen Wixson, University of Michigan

As you read all this, keep in mind that for all this hoopla and committees layered upon committees, NAEP tested 165,000 fourth graders, 159,000 8th graders, and 12,000 12th graders. We can be grateful it wasn't more and we should be very wary of NCLB's corporatized political push to make NAEP the test of record.

Appendix

These charts come from the NAEP website and don't reproduce well online here. But some important points can be derived from them.

Distribution of Reading Questions 2005 Target and actual percentage distribution of questions, by context for reading, grades 4, 8, and 12: 2005

Grade 4 Grade 8 Grade 12
Target Actual Target Actual Target Actual
Reading for literary experience
55 51 40 29 35 23 Reading for information 45 49 40 40 45 50
Reading to perform a task
20 31 20 27
Reading to perform a task was not assessed at grade 4.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Reading Assessment.

This means that 8th graders have to interpret a mass transit schedule and 12th graders are interrogated about Instructions for the IRS Form 1040EZ.

Performance Data provides National Performance Results, percentages of correct, incorrect, and omitted answers. Open-ended questions categorize student responses as little or no comprehension, partial/surface comprehension, full comprehension, omitted, off task. But occasionally NAEP rates open-ended answers as unsatisfactory, partial, essential, extensive. Sample student answers are provided in each category along with scorers' brief remarks.

Distribution of Reading Questions 2005 Target and actual percentage distribution of questions, by context for reading, grades 4, 8, and 12: 2005

Grade 4 Grade 8 Grade 12
Target Actual Target Actual Target Actual
Reading for literary experience
55 51 40 29 35 23 Reading for information
45 49 40 40 45 50 Reading to perform a task
20 31 20 27

Reading to perform a task was not assessed at grade 4.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Reading Assessment. The proportion of items related to each context for reading changes from grade to grade to reflect the changing demands made of students as they mature.
________________________________________
Target and actual percentage distribution of student time, by aspect of reading, grades 4, 8, and 12: 2005

Grade 4 Grade 8 Grade 12<br> Target Actual Target Actual Target Actual
Forming a general understanding/developing interpretation
60 68 55 59 50 56 Making reader/text connections
15 14 15 17 15 14 Examining content and structure
25 17 30 24 35 29

NOTE: Actual percentages are based on the classifications agreed upon by NAEP's Instrument Development Panel. It is recognized that making discrete classifications for these categories is difficult and that independent efforts to classify NAEP questions have led to different results.

Percentages may not add to 100 due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Reading Assessment.

The table above shows the percentage of time students in a given grade would spend on NAEP items that measure each aspect of reading (general understanding and developing interpretation are combined) if they responded to all items in the NAEP reading assessment.
Aspects of reading
Forming a General Understanding

To form a general understanding, the reader must consider the text as a whole and provide a global understanding of it. Students may be asked, for example, to demonstrate a general understanding by giving the topic of a passage, explaining the purpose of an article, or reflecting on the theme of a story. Tasks and questions that measure this aspect of reading include the following:

Write a paragraph telling what the story/poem is about.

Which of the following is the best statement of the theme of the story?

Write a paragraph telling what this article generally tells you.

What is this text supposed to help you do?

What would you tell someone about the main character?

Exhibit 7. Sample NAEP Items, by Element of Literary Text and Aspect of Reading

Element of Literary Text Aspect of Reading
Forming a General Understanding Developing Interpretation Making Reader/Text Connections
Examining Content and Structure
Theme What is the moral in the story? Use evidence from the story in your response. How does the setting help to illustrate the theme of the story? Do you think the lesson in this story is true today? Why or why not? Explain what makes this story a fable.
Major characters What was the major character's opinion of _____? What causes the main character to do _____? Use evidence from the story in your response. How do you think the character's actions might be different today? Support your response with evidence from the story. How does the author's description of _____ help explain the character's actions?
Major events Write a short summary of the major events in the story. What happens after _____? How do you think the story would have ended if _____ had not happened? How do the first events help you predict the ending?
Problem How does _____ make the problem worse? Use evidence from the text to support your response. How did _____ help solve the problem? How does the problem in the story compare with another story you have read? Include evidence from the text and another story. Why does the author explain the problem in the first part of the story? Explain with evidence from the story.
Vocabulary Which words describe what the story is mostly about? Use evidence from the text to support your response. Which words let you know that time has gone by? Explain with evidence from the story. Explain the double meaning of _____. Tell which meaning better explains the major ideas in the passage. Why does the author use the words _____ to describe how _____ feels?


Exhibit 8. Sample NAEP Items, by Element of Informational Text and Aspect of Reading

Element of Informational Text Aspect of Reading
Forming a General Understanding Developing Interpretation Making Reader/Text Connections Examining Content and Structure
Central purpose What might be the author's message in this article? How does the author support the message? Do you agree with the author's message? Give evidence from the text. Based on what you read, what might be the reason the author wrote this?
Major ideas Give a summary of the major ideas. How does the big idea in the first section relate to the big idea in the last section? Who might need or want this information? Use details from the text in your answer. What did the author do to present information clearly?
Supporting ideas Identify ideas that most closely relate to the topic. Give evidence from the text to support your choice. How does the author show you that the main idea is important? Which details about the _____ help you to have a clear image of the topic? Explain why you chose them. What information did the author have to know before writing the article.
Adjunct aids The chart in this article is mostly used to _____? How does the information in the chart support the information in the article? Why did the author include the picture with the chart? Explain using what you know and information from the text. What is the significance of the map to the article? Explain.
Vocabulary Which words describe what the passage is mostly about? Use the evidence from the text to support your choice. Which words do you think mean the same as the title? Tell why you think so. Explain the double meaning of _____. Tell which meaning better explains the major ideas in the passage. Why did the author give a definition of _____ in paragraph 2?

Exhibit 9. Sample NAEP Items, by Element of Practical Text and Aspect of Reading

Element of Practical Text Aspect of Reading
Forming a General Understanding Developing Interpretation Making Reader/Text Connections Examining Content and Structure
Central purpose What is the purpose of the document? Use evidence from the document in your response. Do these directions list all the materials you need? Give a reason for your answer. Do you think the directions would be the same for _____? Use evidence from the text. Is this mainly for readers familiar with the activity? Explain why or why not.
Key information Tell how this information would be useful. Use evidence from the document. Why is it important to do _____ before _____? Use evidence from the document in your response. Compare the directions to a set of directions you have used in the past. Which set is easier to follow? Tell why. Name one step that is important in order to follow the directions. Explain why this step is important.
Key organizing features Tell what you need to complete the steps. Which features indicate this? What happens after _____? Explain why this is an important step? When might you use these steps? Support your answer with evidence from the text. Explain how the author organized this document. Use support from the text.
Key graphics What is the significance of the graphics to the total set of directions? Use support from the text in your response. Why does the second picture show _____? Which additional graphics would you add to make the directions clear? Use support from the text. Why is there boldface print at the front of each section? Vocabulary Which words tell what the document is mostly about? Use evidence from the text to support your response. Use the context to tell the meaning of _____. Explain why the phrase _____ is useful in these directions. Why does the author use the words _____ to present the last step of the directions?


Finally, as mandated by NAEP law, the Governing Board reviews the items both before and after field testing for appropriateness and bias.

How NAGB Works
NAGB conducts its work and carries out its responsibilities through five standing committees. The full Board and its committees meet quarterly. Additional business is conducted through teleconferences and other in-person meetings, as necessary. Committees monitor external contracts; prepare and recommend procedures for reporting and disseminating NAEP results; review and recommend test content for NAEP; and recommend policies to the full Board to guide other NAEP activities. 2004-2005

Committee on Standards, Design and Methodology
Chair:
Vice Chair: Michael Ward
Mark Reckase
Barbara Byrd-Bennett
Jeb Bush
John Easton
Andrew Porter
Darvin Winick
Susan Loomis (staff)

Reporting and Dissemination Committee Chair:
Vice Chair: John Stevens
David Gordon
Dwight Evans
Keith King
Luis Ramos
Mary Frances Taymans
Thomas Vilsack
Eileen Weiser
Larry Feinberg (staff)

Assessment Development Committee Chair:
Vice Chair: Amanda Avallone
Kathi King
Francie Alexander
David Alukonis
Carl Cohn
Shirley Dickson
Sheila Ford
Kim Kozbial-Hess
Oscar A. Troncoso
Mary Crovo (staff)

Nominations Committee
Chair: Eileen Weiser
Francie Alexander
Amanda Avallone
B

— Susan Ohanian
Substance, November 2005

---

INDEX OF RESEARCH THAT COUNTS