Orwell Award Announcement SusanOhanian.Org Home


Commentaries

Standardized students: The Problems with Writing for Tests Instead of People

Posted: 2005-11-02

The author raises interesting questions about training students to game the system, and his remarks about computer-graded tests raise a fascinating/disturbing/important question: Who's the audience? What is the point of writing for a computer? One can easily extend this and ask, What is the point of going to school at all when your teacher has become nothing more than a delivery agent for a standardized curriculum designed to game the system?


from Journal of Adolescent & Adult Literacy 49: October 2005

pp. 152-158



Bronwyn T.Williams teaches at the University of Louisville in Kentucky. He is editor of the Literacy and Identity Department.





I?m not usually a huge fan of bumper stickers, even

the ones I agree with, because of the way they

shout out simplistic positions on complex issues.

I still remember, though, when I saw the one that

read ?Standardized Testing Produces

Standardized Students.? I smiled and nodded my

head a bit in agreement. Simplistic as that phrase

may be, over the years it comes back to me when I

am involved in conversations about testing that

seem bound in reductive and simplistic arguments

about standards, rigor, and accountability.

I don?t use the phrase because I don?t think using

a bumper-sticker argument I agree with makes it

superior to my opponents? bumper-sticker arguments.

But I do try my best to nudge arguments

about testing toward a more complex consideration

of the myriad implications of the concept of

standardization.



Some levels of standardization I rely on. I

take great comfort that there are identifiable standards

for inspecting elevators, for example. I also

appreciate that most medical doctors go through

some kind of standard training in human anatomy.

And I believe that we teachers have a responsibility

to assess whether students are learning the concepts,

ideas, and ways of thinking that we believe

are important and also to hold ourselves and our

pedagogies accountable if we are failing to reach

most of our students.



Like many of my colleagues, however, I am

not convinced that literacy assessment is best

achieved through standardized tests given to huge

groups of students in high-stakes situations.

More and more, it seems as if the point of literacy

education?of all education?is becoming standardized

assessment and rankings rather than

learning. Standardized assessment differs from

assessment that attempts to determine whether

students are learning what we are trying to teach.

No one I know is against the latter.What recent

trends in standardized assessment emphasize,

however, is not learning but the comparison and

ranking of failure. Incessant testing regimes, such

as the infamous No Child Left Behind (2002) law

in the United States (known ruefully among

many teachers as No Teacher Left Standing), focus

on broad comparisons of students, with little

regard to their differences, and severe punishments

for schools and teachers who fail to meet

the ?standards.?As long as students meet the

standards, what they may actually be learning

seems to be beside the point.



The fervor for this kind of standardized

testing reinforces the kind of ranking games that

are a particular enthusiasm of Americans and are

certainly not unknown in other countries.We?re

willing and eager to rank anything?from the 100

greatest movies to the 250 best cities in which to

raise children. Never mind that the criteria for

such rankings are hazy at best; if we can?t put a

number on it and rank it, then what good is it?

Numbers seem scientific and technological. So we

test and test and test, oblivious or resistant to the

possibility that standardized literacy testing often

produces numbers with about as much utility or

connection to reality as ranking songs on the old

American Bandstand television program. ?It

sounds good, has a good beat, and I can dance to

it. I give it an 87.?



Testing as punishment

My concerns about the increasingly pervasive reliance

on standardized testing in literacy education

are about more than questionable methods

of assessment and measurement, however. I am

also deeply troubled about the implications for issues

of literacy and identity.What effect does the

unrelenting emphasis on standardized literacy

testing have on students? perception of the purposes

and possibilities of literacy? By extension,

what effect does such testing have on their perception

of the possibilities for themselves as readers

and writers?



Many concerns about identity and standardized

testing have been framed in terms of

race and social class and have been well documented

and well argued by others (McNeil, 2000;

Murphy, 1997; Ohanian, 1999; Orfield &

Kornhaber, 2001). These teachers and researchers

have argued that standardized testing works not

from a set of objective standards somehow as

constant as the North Star but from a set of cultural

conceptions about literacy that are neither

objective nor static. Students whose race or social

class is not part of the dominant culture often

face more complex challenges in meeting the

standards of that dominant culture.Much of the

impulse behind standardized tests and their illusion

of objectivity seems to be a drive to punish,

ridicule, and marginalize those who already feel

punished, ridiculed, and marginalized by the institutions

of education. At the same time the

standardized tests from the dominant culture reassure

its members about the quality of their educational

institutions as well as their children.

Using literacy tests to reinforce dominant privileges

and exclude others is nothing new; all we

have to do is remember the literacy tests used to

reject immigrants to the United States or to keep

African Americans off U.S. voting rolls in the last

century. The standardized testing movement today

is just better able to cloak such motives within

the rationale of ?not leaving children behind.?



I do have another concern about these

methods of literacy assessment in terms of student

identity.Most teachers have stories of bright

students who ?don?t test well.? I?ve seen such students

at every level, from middle school to university,

and all of them could do innovative,

creative, fascinating work on a project or a paper.

But for reasons stretching from learning disabilities

to personality traits and cognitive ways of

processing and communicating information, they

could not score well on timed, standardized tests.

Nevertheless, there are students who blossom in

such test-taking situations?students who understand

the rhetorical demands and structures of

standardized exams, and whose minds organize

and recall certain kinds of information quickly

and efficiently.



I see one of each kind of student when I

look at my twin adolescent sons, who were born

just 15 minutes apart and raised in the same circumstances.

One son excels at taking standardized

tests of all kinds by understanding the

rhetorical structure of the questions and the institutional

demands of the exam. The other, though

in some ways a more powerful writer and just as

strong a student in school, has always found standardized

tests rigid and bewildering. If all I knew

about them were the results of their annual

standardized tests, I would no doubt rejoice in

how the school system was succeeding to educate

one while worrying over its failure to reach the

other. Their wildly divergent test scores tell me

nothing about their abilities and nothing about

the quality of teaching they receive.



Yet U.S. culture clings to standardized literacy

tests as a means of providing meaningful information

about students, teachers, and schools because

such tests offer the illusion of scientific

rigor (as well as those all-important quantifiable

numbers) to an endeavor that ultimately can?t be

measured in a lab and for which numbers are

meaningless. This infuriating numbers game allows

politicians and media pundits to make facile

judgments, and cynical proclamations, about education

that they turn into a relentless cycle of

testing, criticism, and punishment. From the administrators

to the teachers and students, testing

drives the curriculum, and the curriculum shapes

student identities in terms of literacy practices.



Writing as a human endeavor

Standardized testing, to be standardized,must

create questions and answers that leave no room

for interpretation. Such rigid questions and answers

remove the importance of context from literacy

practices and allow for no independent

meaning making from students. Yet it is in that

moment when an individual makes meaning in

writing and reading in a specific cultural context

that identity and literacy come together.When

literacy education becomes more about standardized

assessment, it becomes less about writing

and reading by individuals who make meaning

and have something to say. In the drive to assess

and quantify, what is forgotten is why we want

students to read and write in the first place.

Reading and writing is about communicating

with other human beings?about being part of a

society and its ongoing conversations. I think

most literacy teachers dream that students, once

their days of school are over, will be inspired and

educated to read and write about what matters to

them. That is the kind of literate identity we want

for our students.



However, the increasing pressure of standardized

testing disconnects literacy education from human

concerns. Students face writing prompts and

reading tests that have no connection to their lives,

communities, or interests. The tests are created and

then read by disconnected, uninvested, anonymous

readers?and now perhaps even computers.

Literacy practices become less about communicating

with people and more about communicating

with a faceless system or a machine.What students,

like administrators and teachers, learn from this

system is that only the numbers matter, not the

meaning or the communication.



Students and teachers have also learned

that, like any system, standardized tests can be

?gamed.?When work is written only to be assessed

rather than to communicate ideas, the activity

becomes more about ensuring that certain

qualities are present (e.g., the use of examples, the

complexity of sentences, transition devices, vocabulary)

regardless of the overall effect of the

piece of writing. In the United States, one example

of such an activity is the brief writing section

recently added to the Scholastic Aptitude Test

(SAT)?the test taken by thousands of universitybound

students that is supposed to indicate their

abilities to succeed in higher education. Using a

25-minute writing sample to determine a student?s

ability to write the kind of extended critical

prose required in university is like using a person?s

ability to back a car down a driveway to determine

whether the driver can make it through

rush-hour traffic. The writing samples on the

SAT are apparently scored on such generalizable

characteristics of writing as smooth transitions

and varied sentences rather than on content or

overall effect (Klein, 2005).



So let?s say specific examples are highly

prized, particularly those that are not about personal

experiences. Now, it doesn?t matter if the

specific examples are made up, as long as they are

specific, and so pretty soon we?ll find students

sprinkling their essays with impressively ?specific?

examples.What the students will know, and what

we as teachers will have to admit, is that writing

as a means of communicating ideas does not

matter in this situation; it?s racking up the right

number of smooth transitions and specific examples

that does. It is difficult to imagine such a situation

creating the conditions to inspire students

to think of themselves as writers and readers and

to engage in writing with any sense of ownership

or passion.



Gaming the testing machines

Even more troubling is the recent trend toward

evaluating student writing through computer

software. No longer is the student writing for any

person, even an anonymous person; instead it is

writing done to be judged by a series of algorithms

that look for quantifiable characteristics

such as transitional phrases and complex syntax.

But any of these programs can be easily ?fooled.?

Give one an essay that is gibberish but includes

the proper characteristics of syntax and vocabulary,

and you receive high marks.



In fact, computer assessment software can be

gamed by other computer software.Hesse (2005)

pointed this out in a recent presentation. First he

used the online Essay Generator (www.Essay

Generator.com) to create an essay. If you?ve never

used this website, it is great fun.You enter any

word or phrase and are immediately rewarded

with an essay on that topic with complete sentences;

sections on social, economic, and political

factors; and even important-looking citations and

a graph. But reading the essay more closely reveals

its delightful ridiculousness. For example, I just

entered the phrase ?standardized testing? and

received an essay with the following opening

sentences:



The issues involving standardized testing has been a

popular topic amongst scholars for many years. In

depth analysis of standardized testing can be an enriching

experience. Though standardized testing is a

favourite topic of discussion amongst monarchs,

presidents and dictators, there are just not enough

blues songs written about standardized testing.



The essay continues with complex and syntactically

correct sentences that make little or no

sense at all. Hesse (2005) took his randomly generated

essay, full of similar nonsense, and then

submitted it to ?intelligent? essay assessing software

and received a report that praised the high

quality of the writing, including the mature command

of language and the effective use of examples

and transitional devices. I replicated Hesse?s

experiment with similar results (and it?s worth

trying yourself if you want to be simultaneously

amused and horrified). Perhaps this is the wave of

the future:We can have computers write essays to

be read and evaluated by other computers and

leave students out of the process altogether.



Even so, the important concern in the use of

computer assessment, as well as standardized testing,

is not whether such systems can be fooled.

Those advocating the development of such software

maintain that someday computer software

may be able to read student essays for content as

well as for syntactic characteristics. They may be

right. But the sophistication of the software is not

the point. The more disturbing question is about

what writing becomes when it is produced to be

read and evaluated by a computer. What is the

point of writing for a computer? Will there be

writers who will care as deeply about what a computer

thinks about their writing in the way that

they care about pleasing a human audience? Will

anyone take writing for a computer seriously?

Students may care whether they pass a test, but

what will they learn about literacy and identity

when there is no human connection to what they

write?



I?m a die-hard humanist on this issue.

Writing and reading are about touching the mind

of another person, whether in a proposal, a poem,

or a polemic.When advocates of computer software

or standardized testing argue that their systems

will be more objective, it is an argument that

again strikes me as beside the point.Writing for

other human beings?even when they are

teachers?is by nature subjective. Everything

about writing depends on context, culture, and

the occasional unpredictability of human response.

Such is the challenge and joy of writing,

along with its intermittent frustration. Even as I

write this column, thinking I have some sense of

my audience and the context of this journal, I

know that I cannot predict all the possible responses.

I can?t think of a writer who doesn?t

know in her or his bones that writing will always

be responded to and judged subjectively.Yet standardized

literacy testing and evaluation would

have us pretend to students and their parents that

such subjective responses can be overcome with

scientific methods and better technology, and that

such methods can generate a set of numbers that

are meaningful about the quality of that writing.

And we forget that even standardized test readers

and computer software programmers are people

with their own biases and preferences.We do

these same students a significant disservice by acquiescing

to this pretense of objectivity and by not

talking to them about how real humans in real

contexts read and how best we can try to identify

and respond to such real people and situations.



In conversations with students about how

real audiences in real situations respond to writing

is where meaningful assessment can happen.

If I want to know if students are learning to write,

I first want to see those students write about subjects

that matter to them for an interested audience

and reflect on the reasons for their writing

choices. Once students get a real response from

an audience, I want them to reflect again on that

response and what it might teach them about

what they wrote and why. I always talk with students

about assessment and how their work could

be assessed. Then I discuss my choices during assessment

and how I evaluate both their writing

and their reflections on it. I?m not against assessing

writing. I tell students that every piece of

writing read by another person is assessed in

some way, even if only in the reader?s choice to

read all the way to the end. I?m just against thinking

that we can assess writing through some

pseudoscientific, technology-driven, one-size-fitsall,

a-contextual test.



Making a difference with

human response


Sommers and Saltz (2004), in their longitudinal

study of more than 400 university student writers,

found that the students who made the most

progress in terms of writing shared two characteristics.

First, these students, even if they began

their university careers as relatively weak writers,

were the ones who brought ideas and issues that

mattered to them to their choice of courses and

their responses to assignments. Equally important

for these students was the ability to see writing

assignments as something more than just fulfilling

a requirement for a grade:



When students begin to see writing as a transaction,

an exchange in which they can ?get and give,? they begin

to see a larger purpose for their writing. They have

their first glimmerings of audience; they begin to understand

that they are writing for flesh-and-blood human

beings, readers who want them to bring their

interests into a course, not simply teachers who are

poised with red pens ready to evaluate what they don?t

know. (p. 139)



In addition, Sommers (2005) found that the

responses from teachers that resulted in the most

substantial improvement in student writers took

the work and the ideas of the writers seriously and,

along with constructive criticism, provided comments

and questions that pointed the students toward

what they might improve in future pieces of

writing.Writing for a real audience about ideas that

engage individual interests and intellects and having

that audience provide thoughtful individualized

responses not only helps students become better

writers, it also helps them create their identities as

writers. Such an approach to writing is the antithesis

of standardized testing.



Of course, the sad fact is that there are people

in and out of education who are not

concerned about whether writing with passion

for real readers will inspire students and help

them develop identities as confident writers. This

camp wants to ensure that students attain a functional

literacy that makes them productive in the

workplace?in clerical and service industry jobs.

As I?ve argued before in this column, the way literacy

instruction is conceived and enacted is often

connected to issues of social class, and it?s the

same with standardized tests as well. One benefit

of standardized testing as assessment is that it is

much cheaper than the kind of individualized assessment

I advocated earlier. Affluent students

will be taught to ?game? the standardized testing

system. But these same students will also have

abundant opportunities in small classes in elite

schools to write for real readers and to read for

meaning, pleasure, and enrichment. By contrast,

it is working class and poor students, for whom

pleasure and the construction of literate identities

is deemed unimportant, who will encounter a

curriculum driven by the fear of standardized

testing. The administrators and teachers of those

students, facing large classes, few resources, and

the threat of punishments over low scores, will be

forced to create a curriculum in which students

will be taught to write only for the anonymous

readers and machines that evaluate such tests.

Certainly that will be cheaper than providing the

resources and trust in poorer schools that would

allow teachers to face the same class sizes and use

the same approaches for teaching writing that affluent

schools enjoy. But students from poorer

schools will encounter fewer and fewer opportunities

to think of themselves as writers communicating

their interests and passions for real readers.



What is to be done?

Many teachers, even when facing the pressure of

standardized testing, continue to design assignments

that fulfill the needs of assessment while

providing students with places to do writing and

reading that matters. And systems of assessment

that are not bound up in concerns about time

and technology, such as writing portfolios, can

provide responsible assessment that is valuable

for students, teachers, and institutions. However,

writing portfolios can become as rigid and impersonal

as any approach to assessment if the focus

of the writing becomes the assessment itself

rather than the communication of ideas.



The pressure toward standardized testing is

such that more must be done than just designing

good pedagogy around the margins of a testdriven

curriculum. It is time for individual teachers

at every level to take back the debate about

assessment from the people who say the only

valid evaluation of writing is timed, standardized,

and faceless. It is time to take back the debate that

maintains that only assessment that results in

quantifiable numbers is valid. Engaging in this

debate does not happen on a grand stage, it happens

as a persistent, continuing conversation with

the people in our communities. It is the responsibility

of teachers to keep explaining to students,

parents, administrators, neighbors, and newspaper

editors that writing and reading that matters

for real audiences is what creates literate citizens.

It is the responsibility of teachers to explain how

such writing can be assessed, that such instruction

can be responsible and accountable, but that

assessment and instruction happen differently in

literacy education than in other fields. Finally, it is

the responsibility of teachers not only to keep

talking about what the best practices and outcomes

for student literacy education should be

but also to publicize our successes.



Reading and writing matter, and we teachers

care deeply about them. If we want students to

think of themselves as readers and writers, then

those activities have to matter to students beyond

learning how to game the test to avoid being punished.

It is time we reclaimed the idea that having

standards does not necessarily mean accepting or

aspiring to standardization.



REFERENCES

Hesse, D. (2005, March). Who owns writing? Keynote address

at the Conference on College Composition and

Communication, San Francisco, CA.

Klein, K. (2005, April 3). How I gamed the SAT. The Los

Angeles Times, p. A20.

McNeil, L. (2000). Contradictions of reform: Educational costs

of standardized testing. London: Routledge.

Murphy, S. (1997). Literacy assessment and the politics of

identities. Reading and Writing Quarterly, 13, 261?278.

No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115

Stat. 1425 (2002).

Ohanian, S. (1999). One size fits few: The folly of educational

standards. Portsmouth, NH: Heinemann.

Orfield, G., & Kornhaber, M.L. (2001). Raising standards or

raising barriers? Inequality and high-stakes testing in public

education. New York: Century Foundation Press.

Sommers, N. (2005, March). Across the drafts: Responding to

writing, a longitudinal perspective. Paper presented at the

Conference on College Composition and

Communication, San Francisco, CA.

Sommers, N., & Saltz, L. (2004). The novice as expert:

Writing the freshman year. College Composition and

Communication, 56, 124?149.



The department editor welcomes reader comments on this column. E-mail bronwyn.williams@Louisville.edu. Mail Bronwyn T. Williams,

University of Louisville, Department of English, Humanities Building, Louisville, KY 40292, USA.

More Commentaries


FAIR USE NOTICE
This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.