Accountability and the Slippery Language of Public Relations
Part 2, Building Support for Data-Based Decisions, offers a wide variety of data collection and management. As it happens, Chapter 12, "School System Strategies for Supporting Data Use" by Amanda Datnow and Vicki Park leads off with the same paragraph from Education Week5 that I used (2009) in a recent article for Language Arts.6
Talk about the eye of the beholder! The authors view this scenario as something to be desired. I am appalled by the casual assumptions presented in this positivist, technocratic view of teacher decision-making. Such "planning" assumes:
One could go on. Datnow and Park decry practitioners who "make decisions based on
intuition, gut instinct, or fads," but they and many other authors in this book take the content of standardized tests as a given. They assume that this content is linked to state
standards and extreme focus on this content benefits students.
For seven years, I've run a website documenting the outrages offered by standardized tests, documenting in chilling detail how schools have become test prep boot camps for test questions that are just plain loony. Here's an item from a test given to every fourth grader in New York State. After reading a passage about how pretzels are made, fourth-graders were asked:
If you think you know the answer to this question, try looking up pretzel in the
encyclopedia. Nothing. In an article about bagels from the Dec. 31, 2003, New York
Times, I did find a pretzel mention: "It wouldn't be Philadelphia without soft pretzels." More searching produced one sentence that might qualify as "historical": Old-time pretzel makers dipped the pretzels into a lye solution. I found it in my kitchen among my
cookbooks, A World of Breads by Dolores Casella.7
Many will see this test item as benign enough, but take a moment and consider: Everything about this item is emblematic of the hubris embedded in standardized tests. Item writers with no connection to real children in real classrooms, or even, it seems, any connection to real pretzel history, invent inaccurate and devious and just plain stupid material. No wonder they insist on keeping tests secret. The teacher who is informed by computer printout that Johnny, whose mother may have just torn a pretzel recipe out of the newspaper, missed this answer, won't know a thing about his reading comprehension. How can poring over student answers to such questions in the name of making "data-based decisions" possibly improve educational practice? OR "promote the ongoing adaptation of the organization."
And there's much worse. Mark Fisher and Scott Elliott won first-place award for education writing in the 71st annual National Headliner Awards for their series "Flunking the Test" in the Dayton Daily News By deconstructing a test question about John Glenn, they show that the more you know about the subject of a test question, the more likely you are to get it wrong. The journalists interviewed Clifford Hill, co-author of Children and Reading Tests,9 a book that blows the lid off the ability of standardized tests to measure something called comprehension. Using methods of discourse analysis, the authors examine not only representative material from elementary school reading tests but also children's responses to that material. In short, they talk to children about why they chose the answers they did, and with this information, the reader is suddenly made aware of another universe, the one inhabited by children. One cannot read the student explanations without realizing how biased toward a "foreign: perspective the reading tests are. The convincing quality of the children's "wrong" answers show us how far the testers miss the boat. The book is particularly attentive to the role of culture in shaping children's understanding of what they read.
Massachusetts tenth graders were asked to read a passage from Steinbeck's The Grapes of Wrath and then to choose the correct answer to this question:
As though Nobel Prize writers write to provide children with grammar lessons. On Massachusetts tests featuring poetry, children are invariably interrogated about parts of speech. After reading Ruth Krauss's The Carrot Seed, Mississippi second graders were asked:
Ask yourself how appropriate it is that a Latino high schooler in Los Angeles taking the California Exit Exam--to determine whether he gets a diploma--is interrogated on the work of Gretel Ehrlich, known as the Whitman of Wyoming. To prove reading proficiency, Florida high schoolers were tested on a poetic, almost surrealistic, account of tracking moose in Alaska. I wrote the essayist, who termed the test questions "very weird." He wrote me, "I could hardly believe what I read." In New York, high schoolers hoping to graduate had to decipher an essay by Roger Ascham. You remember Ascham, 16th century essayist who wrote about archery. A key word in understanding the passage on the test is listed as obsolete in the Oxford English Dictionary.
Ah, so many bad questions, but so little space.
With Arne Duncan and crew roaming the country and promoting the specter of national standards and a national test, I must mention NAEP. Of the eight items NAEP posts as samples from fourth-grade reading tests over the past decade, Highlights magazine holds the copyright on half the items. Topics include an American female astronaut on Mir, crab hunting, wombats, and life in the American Colonies. Two items, a West African tale and a pour quoi story from William Bennett's edited collection The Moral Compass, are in the folklore genre. There are two stories about rural children and their dogs. NAEP explains that test items are "taken from authentic texts found in the environments of students." Ask yourself about student access to such materials. Teachers are forbidden to bring in books not in the scripted lessons shipped in from publishing conglomerates approved by NCLB. California schools gave up on the idea of staffing schools with professional librarians long ago (in many cases, giving up on even libraries themselves). Oh, and a subscription to Highlights costs $29.64.
Think about it: Students perform poorly when they're asked to do the wrong things. Please push all that scientific data aside for a moment and stop and think about this truism.
Instead of putting the screws to teachers and students, we'd do well to regulate the
testing industry. As FairTest once pointed out, we pay more attention to what goes into our pet food than what goes into our standardized tests.
Datna and Peak describe their study of four school systems "to capture the details of data-driven instructional decision making." This study was funded by NewSchools Venture fund and the Gates and Hewlett Foundations. As it happens, they chose Garden Grove and Aldine, both honored by the Broad Prize for Urban Education. Garden Grove won the award in 2004 and Aldine has been declared a finalist numerous times. Both are infamous for their allegiance to scripted curriculum. What the author refers to as "high level of engagement in data-driven decision making" translates as allegiance to the pacing guide, where the teacher is told, "You're going to follow it, and it's non-negotiable."
In Chapter 11, "Building Data-Driven District Systems: Examples from Three Award-Winning Urban Systems," Heather Zavadsky, whose vita includes managing the Broad Prize in Urban Education, describes Aldine in some detail. She notes that "one of Aldeine's most impressive features is the frequency with which the district monitors student progress on an ongoing basis." She also points out that "Teachers are routinely observed by principals, department chairs, and their peers who have been trained to use a structured observations and walk-throughs." Those walk-throughs are inspections, making sure teachers are on script. Education Week put it this way11: "Aldine is striking because the district leaves so little to chance when it comes to student success. . . . It takes little time for a visitor to Aldine to see the heavy emphasis that is placed on preparation for state tests. Computer programs scroll through lists of practice questions. Printers spit out scores for teachers to review. Timed practice quizzes help students prepare for the real thing."
Leaving so little to chance. . . . I hear from teachers across the country who complain of so-called literacy coaches trolling the hallways to make sure everybody is on the same page of the mandated scripted curriculum. The inspectors are on the alert for "illegal" picture books and chapter books that teachers might have sneaked in to supplement the curriculum. Only books specified by the publishing conglomerate supplying the mandated "scientific" curriculum are allowed.
Years ago, then-New York Times metro reporter Michael Winerip wrote a fascinating piece about a member of the Association of Professional Organizers, people paid top dollar for organizing other peoples. closets. Winerip asked to see the professional organizer's own closet and reported that for every piece of clothing, this woman keeps a note card of matching accessories: With her green suit, she always wears her green shoes, amber pin, and beige pocketbook. "I never have to think about anything, it's great," she said.
Much as we may admire tidiness, and be tempted by the Lorelei lure of all those sleek solutions receptacles at http://www.holdeverything.com offering a beautifully organized future, teaching isn't neat and tidy and predictable. We can't always wear the green shoes with the amber pin. Likewise, we have to clean our own closets; we can't hire somebody to invent and arrange our curriculum--or accept the one shipped in from a publishing conglomerate. As NCLB schools embed themselves in some sort of data fung schway and test prep becomes the imperative for clearing classrooms of intellectual clutter, some of us must stand up for clutter. The teacher who allows form to triumph over substance loses her core.
In Chapter 10 "Principal Leadership, Data, and school Improvement," Michael A. Copland, Michael S. Knapp, and Julia A. Swinnerton actually mention that teachers' perceptions, or classroom-based assessment, have "as important a role in improving teaching and learning as their quantitative counterparts." The authors make the important point that certain data points (e.g. disaggregated state math test scores) may provide an awareness of a given situation (e.g. performance gap between seventh-grade boys and girls), but the data do not necessarily indicate how educators should address the issue at hand. In this example, assessment data certainly inform conversation about possible actions, but they do not necessarily "drive" decisions or provide information about how best to address the issue of low performance. They observe that data may not lead to action but may raise more questions. They discuss a culture of inquiry, not a system of plugging in answers. And they point out that attempting to use data within cycles of inquiry is likely to be messy.
Part 3, Data-Based Applications, discusses "stakeholders" using data, offering some cautionary notes and lots of encouragement. I find it interesting that with all the talk of technology for Twenty-First Century Learners, there is no mention of IBM's Reinventing Education initiative. Here's how a January 2001 Education Week story about
this initiative opened:
Billed by IBM as "a commitment to corporate citizenship," IBM modestly claimed to be "solving education's toughest problems with solutions that draw on advanced information technologies and the best minds IBM can apply. Our programs pave the way for systematic reform in school systems nationwide."14
Provoked by University of North Carolina sociology professor Roslyn Mickelson's compelling account of IBM's mission and methods in "International Business Machinations,"15 I studied how teachers used (or didn't use) their district's IBM Reinventing Education grants, recounting what I learned in some detail in Why Is Corporate America Bashing Our Public Schools?16 The West Virginia Learning Village featured juried lesson plans written by teachers. These lessons featured a close link between state standards and classroom curriculum. Even though these standards loom over every lesson, surely the teachers know that students aren't reading Anne Frank's diary to learn how to use the computer or to trace the surprise ending in a story. But the template demands plugging in standards. Apparently, the more, the better. Never mind that the list of standards is jarring and even offensive to the work of literature under consideration: The Diary of Anne Frank becomes the delivery system for workplace skills.
Throughout, the Internet acts as handmaiden to exaggerate the trivial in the lesson templates. A Synonyms and Antonyms lesson plan that is studiously, even maniacally, aligned to state standards is thirty-three pages long-and that's just the bare-bones description, not including all the assignments and assessments. One of the rules of the template is that for every activity, there must be an assessment. This means that when Standardistos rule, you get a template of assignment-assessment, assignment-assessment, and teachers are trapped into doing only those activities that can deliver an assessment. Eighth graders do all this work with synonyms and antonyms to meet West Virginia English Language Arts Standard 8.50: "use writing strategies to write for audiences including peers, teachers, and employers." IBM orates, "Teaching to standards, teachers can use IBM Learning Village to create lesson plans and activities that "linked to standards and are correlated to the mandated curriculum. . . . Our facilitators offer Professional Development programs to train educators on productivity, efficiency and enhancing the learning environment for students."17
Such technological links to state standards are the natural outcome of the imperative delivered by the National Alliance of Business in "'Knowledge Supply Chain': Managing K-80 Learning,"18 where teachers are defined as knowledge suppliers and schools as the knowledge supply chain:
needs and have them put all the partners together to deliver the people I need.
One thing the Standardistos haven't figured out is that a teacher can teach and teach and teach. That doesn't mean the students will learn it--not at that moment, anyway. Mihaly Csikszentmihalyi, Kevin Rathunde, and Samuel Whalen led some interesting research19 in Chicago. Using an Experience Sampling Method, they hooked up 9th and 10th graders in accelerated or advanced classes to beepers and beeped them during the school day, asking them to answer questions in a provided booklet, questions which asked what they were thinking about at that time. As it happens, while a teacher lectured 27 students on Genghis Khan's invasion of China and the conquest of Bejing in 1215, only two of those students mentioned China: One was thinking about a meal he'd had the night before in a Chinese restaurant and the other wondered why the men used to wear pigtails. Nobody mention Genghis Khan.
No teacher would be surprised. The only surprise is that researchers cared about what students are thinking about. Next thing you know they'll be asking teachers. The authors note that getting students interested in a subject is one of the least important goals. "Despite our relatively heavy investment in education as a nation, we still do not seem to realize that teaching which does not consider the students' priorities is useless. It is wasteful to teach someone who is not interested and so is not motivated."20
As much as I decry the lessons they produced, I wince at what happened to the West Virginia teachers' labors. In 1999, West Virginia teachers posted 128 juried lessons. Ten years later, when one tries to access them, this message pops up:
This site was taken offline on June 30, 2008.
The server software has reached 'end of life' and is no longer supported.
IBM has moved on. Any veteran teacher can tell you that this is the way hotshot reform
I appreciate Jane Hemsley-Brown's concern as she opens Chapter 17, "Using Evidence to Support Administrative Decisions," by acknowledging "the gap between the researcher's world and the practitioner's work." But overall, this book gives me the sense that this is a one-way street: I acknowledge that "research literature is generally not part of a practitioner's library." But teachers know when they're being treated as just so much rutabaga.
I am reminded of a Kurt Vonnegut story where a young man is admiring the centerfold of some girlie magazine. He shows it to an older man and says, "Look at that woman!" "Son, that's not a woman," the older man says, "that's a photograph." I would say the same to Standardistas who rely on skills charts and standardized test scores for their notion of children: "People, those aren't children, those are numbers." Confucius said, "I show one corner, and if a man cannot find the other three, I am not going to repeat myself." I always regarded teaching as finding those other three corners, but these days the Federal government insists on sending out the whole prefab building to NCLB schools. Right now, the prospects look dim for convincing the U. S. Department of Education that data and knowledge are not the same thing. They're not even kissing kin.
I admit that I find one of the most positive revelations in this book is that many schools continue to collect data without using it. I am enough of an optimist to think this means they are relying on other information. Although the corporate-political alliance won't believe this, teaching is, much more like a Chinese lyric painting than a bus schedule. You can't chart a kid's learning like the daily temperature. No matter how many tests you inflict on him.
In Chapter 22, "Evidential Reasoning and Decision Support in Assessment of Teacher Practice" Arthur M. Recesso and Sally J. Zepeda acknowledge that "Teaching is a complex series of events and it is difficult to explain what it means to enact effective teaching or standards-based practices. . . ." They then present "an instantiation of methods and a tool in the context of assessment of teacher practices." I remember my first teacher evaluation. At that time in New York City, a teacher grade--A, B, C, D, F--arrived with multiple carbon copies that were shipped off to various offices after the teacher had signed it. My department chair at a high school larger than my home town gave me a C. Acknowledging that I was deficient in several areas, he commented that I was quick to incorporate practical suggestions into my routine and then he added something extraordinary: He wrote that I went out of my way to help students with special needs, that I had a good heart, and I was going to develop into an excellent teacher. Imagine that: Somewhere in the vaults of the New York City educational bureaucracy there's a document testifying to the importance of a good heart. My evaluator didn't assess "the extent to which a standard of teaching is present or absent." He insisted a good heart counted for something.
Theodore J. Kowalski, one of the book's editors, insists that "it has become imperative that administrators and teachers view data-based decision making as a professional responsibility separate from political convictions." I respectfully and vehemently disagree. Educators don't like to admit it, but teaching is a political act. Putting the teachers of urban children receiving free/reduced lunch into scripted curriculum while funding suburban teachers to hold data reflection retreats are political acts. To their credit, Bruce S. Cooper, John Sureau, and Stephen Coffin, open Chapter 23, "Data: The DNA of Politically Based Decision Making in Education," with this assertion, "The politics and processes of making decisions in education are particularly complex, and at times, utterly confusing." They continue, acknowledging that the politics of data-driven decision making "has been virtually ignored by researchers and political leaders."
I'm bothered by such common phrases as "The use of educational data to make decisions and foster improvement." The crucial questions are ignored: Where did this data come from? How good is it? Who's using it? Hows it being used? How will it help us? And so forth. The researchers remain distant and disinterested in individual children who produce the data, but they approach the numbers like a religious rite of purification. I cheered when I heard David Berliner's assertion on C-Span,21 during an interview about Collateral Damage: How High-Stakes Testing Corrupts America's Schools:
Throughout the book's 502 pages, I worried over the absence of anecdotes. I'm the kind of person who wants to know how all these charts and graphs and declarations about transparency affected a single teacher or student. And my worry leads me to a modest proposal. I recommend that every researcher who isn't able to get in classrooms often, not able to talk to--and listen to--individual teachers and students, rent the movie OT: Our Town, a quite remarkable account of Dominguez High School's first theatrical production in over 20 years. Who would ever guess that an update of Thornton Wilder's classic set in Grover's Corner could deliver such educational and social value to a tough and violent school and its tough and violent neighborhood in Compton, California?
As the kids, who are voluntarily showing up after school, mill around the cafeteria, which is the only school space for putting on a play, lines not memorized, jabbering, joshing--busy worrying about everything but Thornton Wilder, a "walkthrough" evaluator would rarely see anything that looked like "on task" behavior or a teacher "delivering" anything that looked like "instruction." Most of the time the viewer has a hard time finding the teacher in the melee. It takes a very good teacherly eye to figure out what she's doing. I was thrilled by this depiction of the subtle ways a teacher works her magic on kids who hate school, kids who have lots of things to worry about besides memorizing their lines. As I watch the film, I'm reminded of Deborah Meier's observation: "Teaching is mostly listening, and learning is mostly telling." This book under review is telling. All the way telling.
Watching a teacher working with students who had life stories of abandonment, suicide and parental waywardness and producing something they could be proud of showed me a whole lot more about those students than anything stored in the school data bank. Wendell Berry, Kentucky poet, farmer, and teacher, writes often of the accountability of words and of deeds faithful to words. If we "stand by our words," insists Berry, then we must speak in specifics about this child and this curriculum. When we are unable to stand by our words, we fall back on the dictates of Standardistos, resorting to the slippery language of public relations, which means abandoning our students to political abstractions.
1 Biesta, Gert (2007). Why 'what works won't work: evidence-based practice and the democratic deficit in educational research. Educational Theory, 57(1), 1-22.
2 Hightower, Jim (1997). There's Nothing in the Middle of the Road but Yellow Stripes and Dead
Armadillos: A Work of Political Subversion. New York: Harper Collins, p. 68 .
3 Groopman, Jerome M. D. (2007) How Doctors Think. Boston: Houghton Mifflin, p. 24.
4 Ibid. (2007) Mental Malpractice, New York Times, July 7, 2007.
accessed May 14, 2009
5 Hoff, David Hoff (2006). Delving into data. Education Week, 25(5), 12-14, 20-22.
6 Ohanian, Susan (2009). On Assessment, Accountability, and Other Things That Go Bump in the Night. Language Arts, May.
7 Casella, D. (1966). A World of Breads. New York: David White, p. 277.
8 Fisher, Mark and Scott Elliott (2005) Flunking the Test, Dayton Daily News, March 11.
Accessed May 26, 2009
9 Hill, Clifford and Eric Larsen (2000). Children and Reading Tests Stamford, CT: Ablex.
10 Press Release (2001). Testing Industry Not Competent to Handle Bush Exam Expansion Plan.
May 20, 2001.
http://www.fairtest.org/testing-industry-not-competent-handle-bush-exam-ex accessed May 18, 2009
11 Johnston, Robert C. (2000) In a Texas District, Test Scores For Minority Students Have Soared, Education Week. April 5
12 Winerip, Michael (1989). Our Towns; A Neat Woman Finds Her Calling In Creating Order.
New York Times, March 14.
13Trotter, Andrew (2001) IBM "Attracts praise for 'reinventing education,'" Education Week, Jan. 10. 2001.
http://www.edweek.org/ew/articles/2001/01/10/16ibm.h20.html14 from www.ibm.com/ibm/ibmgives accessed 6/12/2003 accessed 5/15/2009 content changed.
15 Mickelson, Roslyn (1999). International Business Machinations: "A Case Study of Corporate Involvement in Local Educational Reform," Teachers College Record 100  476-512.
16 Emery, Kathy and Susan Ohanian (2004). Why Is Corporate America Bashing Our Public
Schools? Portsmouth, NH: Heinemann. 67-68, 190-195.
No longer available. Try
http://www.ibm.com/ibm/ibmgives/grant/education/programs/reinventing/wfl.shtml accessed 5/21,109
18 Conference Board 2002 Business and Education Conference during a session, The Business
Role in PreK-16 Learning: Aligning the Knowledge Supply Chain. The phrase, knowledge supply chain, also is used by the National Alliance of Business in Work America, a newsletter published in May 1998.
Knowledge Supply Chain: Managing K-80 Learning," WorkAmerica, May 1998.
19 Csikszentmihalyi , Mihaly Kevin Rathunde, and Samuel Whalen (1993). Talented Teenagers:
The Roots of Success and Failure. Cambridge, England: Cambridge University Press.
20 Ibid. 9
21 Berliner, David. C-Span interview, April 28, 2007. Cited in Susan Ohanian, (2008) When Childhood Collides with NCLB. Brandon, Vermont: Society for the Study of Education, 55
22 Berry, Wendell (1983). Standing by Words. San Francisco: North Point Press. pp. 24-63.
About the Reviewer
Susan Ohanian is a longtime teacher and prolific author on education topics. She has maintained a website of resistance since the passage of NCLB and does not see any reason to take it down.
Copyright is retained by the first or sole author, who grants right of first publication to the Education Review.
FAIR USE NOTICE
This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.