Parents Question Fairness of Misleading Exams
Ohanian Comment: This article was part of a May 2004 series "Flunking the Test" that won a first-place award for education writing in the 71st annual National Headliner Awards. And it is well worth reading again.
What a hoot! The reporters asked John Glenn to choose the right answer to a standardized test question about him. But if this politician supports the tests, scholars who know something about them, don't. Ultimately, the young test-takers fell into traps laid for them by the test makers because they did what people usually do when they read - they drew on things they knew about the world to help them better understand the test's short, often incomplete reading passages.
This is a great article to share with parents.
Fourth-graders Steve McPherson and Kayla Childs got the answer right, but the question - Why did John Glenn join the Navy after college? - troubled their parents. The correct answer is A: "He wanted to be a fighter pilot."
But Steve's mom also thinks a fourth-grader would be tempted by answer C: "He wanted to be the first man in space." And Steve's father had another problem: Wasn't Glenn, the former astronaut and Ohio senator, a Marine pilot? "If you ask a question in black and white, the answer should be black and white," said Todd Childs, Kayla's father.
In the wake of the 2002 federal No Child Left Behind Act, questions like this are key to guiding state officials, lawmakers and the public as they make judgments about the academic skill level of young children, the quality of instruction by their teachers and the success or failure of their schools. But critics and some researchers say today's tests do not provide the type of information about skills needed to draw fair conclusions about the students, their teachers or their schools. They say the tests are cheating children.
Glenn question classic `test trap'
The Dayton Daily News presented actual sample questions to Steve McPherson and Kayla Childs and their parents to test whether they would fall for the traps laid for students on achievement tests. Steve, who attends Beavercreek's Parkwood Elementary, and Kayla, a student at Dayton's Lincoln Elementary School, earn good grades, but struggle with standardized tests. Both got mostly right answers, but to their parents, some of the wrong answers were disturbingly plausible. The Ohio Department of Education contends students should be able to "infer" from the sentence, "after college, (Glenn) joined the Navy and became a fighter pilot," that Glenn joined the Navy because he wanted to become a fighter pilot.
Jan Crandell, director of the Ohio Department of Education's Office of Assessment, said the question tests "the ability to infer and predict," which is part of the state's expectations for third-grade readers. But to Todd Childs, the word "infer" seems a lot like "guess." He wonders how this question demonstrates anything about his daughter's reading skills.
"There are many different answers you could give," he said. "To become an astronaut, if that's what he wanted to be, a kid could think that you'd have to join the Navy." Steve's mom, Mari McPherson, also thinks the question invites students to draw on what they knew about Glenn before they ever read the passage. If kids know Glenn first and foremost as an astronaut, she thinks they could plausibly predict that joining the Navy was an early step in Glenn's quest to be the first man in space.
"Every fourth-grader in Ohio would have had extensive lessons about John Glenn," she said. "They may have trouble separating what they recall learning in class from what is in the passage."
John Katzman, founder of the Princeton Review, a test preparation company, said the question includes a classic test trap answer. "You know a lot of people will bite at it because they know John Glenn is an astronaut," he said. John Glenn himself said the answers Steve and Kayla chose are right, but he can understand why people can find some elements of the question confusing.
Glenn said he did enlist because he wanted to be a fighter pilot, but his goal all along was to be a Marine pilot. While it is "technically true" that he joined the Navy, it was only for training and he never served in that military branch. "I never got a commission in the Navy," he said. "But all Marine pilots go through the Navy training."
Glenn supports the No Child Left Behind Act, although his chief complaint about it is that the federal money promised in the law has not come through. "I have no objection to the testing requirement,' Glenn said. "No Child Left Behind raised a lot of hopes, but the money was not there to make it real. It raised hopes, but dashed them again."
Good readers can still flunk
Trick questions and other flaws in standardized reading tests are so pervasive that two Columbia University researchers say the tests should be stopped entirely. "I don't think the public has gotten a hold of how insidious this thing is," said Clifford Hill. "We found kids understood what they were reading, but they still didn't succeed on the tests." Hill and Eric Larsen chose to study the once widely used Gates-MacGinitie third-grade reading test because it was "a respected test we felt was responsively done," Hill said. Their goal was to help test-makers find new ways to accurately measure reading skills. "We thought maybe we could get some insights and make recommendations," Hill said. "But the more we went at it, the more we saw this is such a flawed genre that it needs to be scrapped."
Hill and Larsen interviewed students about their answers and how they picked them - especially how they picked wrong answers. They found that many of the students who picked wrong answers were good readers.
Ultimately, the young test-takers fell into traps laid for them by the test makers because they did what people usually do when they read - they drew on things they knew about the world to help them better understand the test's short, often incomplete reading passages.
The tests invite the kids to make inferences because the passages are incomplete," Hill said. "When they make inferences that are plausible they get penalized."
William G. Harris, executive director of an industry trade group called Association of Test Publishers, said there may be problems with some questions, but standardized tests overall are sound.
"You can always find bad examples," he said. "But if you're asking are we doing what states and educators are asking, we are. If tests are reliable and valid and consistent, that's all we can expect a test to do."
Hill, who co-authored the book Children and Reading Tests with Larsen, said the problem is the test makers do not consider how flaws in their processes encourage misleading or confusing questions. "It leads them to produce items that are misleading," he said. "It looks innocent to the test makers. They don't ask themselves any of these tough questions."
Take the example of Alice and the fawn. The Gates-MacGinitie test creators adapted this brief passage from Lewis Carroll's book Through the Looking Glass:
"The fawn looked at Alice with its large, gentle eyes. It didn't seem at all frightened. 'Here then! Here then!' Alice said, as she held out her hand and tried to stroke it. It moved back a little and then stood looking at her."
One question asked "How did the fawn's eyes look?" with the choices sad, tired, gentle
. The correct answer is "gentle." The test-makers thought this should be easy. After all, the text describes the fawn as having "large, gentle eyes" and specifically states that the fawn "didn't seem at all frightened."
But Hill and Larsen found many students were sure the fawn was not only frightened, but injured. These children drew on their outside knowledge of wild animals like fawns - that they fear people.
They told the researchers the fawn would have run away unless it was hurt. They found evidence for this natural-sounding explanation in the description of the eyes as "large" - which some interpreted as wide with fear - and the fact that it moved back when Alice tried to pet it.
With a short passage, students are all but forced to use their outside knowledge to understand the context of what is going on, Hill said. Then test creators put wrong answers that make sense based on what children already know and distract them from the desired right answer.
"The material is incomplete," Hill said. "They build tests where one distractor encourages the kids to make an inference that can't be justified by a strict interpretation of the text, but can be justified based on experience outside the test."
Beating the test
Understanding what a standardized test rewards and what it punishes are keys to a high score. And as some entrepreneurs have discovered, these skills can be taught.
In the early 1980s, one of John Katzman's tutors had trouble with a student. The high-schooler, taking Katzman's Princeton Review course to prepare for the SAT college admissions exam, always seemed to pick the wrong answer after narrowing down her choices to two.
In frustration, the tutor told her to decide which of the two choices she thought was right, but then put the other choice down on the score sheet. It turned out to be a revolutionary idea that became a staple of Princeton Review's program.
"He realized it wasn't random that she was always picking the wrong one," Katzman said. "She chose her answer because it was an attractive answer." To Katzman, who founded the Princeton Review in 1981 while a student at Princeton University, the best test preparation isn't drilling children in grammar and mathematical formulas. It's teaching them the rules of the game they are about to play.
"You have to game it," Katzman said. "You have to be good at puzzles." One of the games Princeton Review teaches is how to spot the "Joe Bloggs" answer. Joe Bloggs represents the "average Joe student" - a child who won't put much thought into a question, but will quickly pick the first answer that looks attractive.
That is a deadly strategy for a standardized test, which is filled with wrong answers designed to entice test takers by sounding plausible - the "distractors."
Princeton Review teaches children to rule out obviously wrong answers quickly and narrow the choices to the two most likely. They are then trained to figure out which of the two is most likely the "Joe Bloggs" choice and encouraged to re-examine the other choice to determine whether it could be correct.
Out of this world logic
In 1998, John Glenn boarded the shuttle Discovery and became the oldest man in space - an event that made headlines around the world. That flight also became the perfect Joe Bloggs answer on Ohio's third-grade reading test.
The question was, "What was the name of the space capsule in which John Glenn orbited the Earth?" The answer is A: Friendship 7, the small spacecraft Glenn piloted in 1962.
But Beavercreek parent Stephen McPherson points out that even the parents of some of today's third-graders were not born at the time of that flight. More people today would recall Glenn's 1998 space trip. And sure enough, Discovery is option C: a Joe Bloggs answer.
A 9-year-old under time pressure could easily fall for that trap, McPherson said. And the question is asked in such a way that one word could change the right answer. After all, Glenn also orbited the Earth in Discovery.
McPherson noted that technically correct can still be wrong. "If they would have put "spaceship" in here instead of capsule, there could have been two correct answers," he said.
Mark Fisher and Scott Elliott
Dayton Daily News