Orwell Award Announcement SusanOhanian.Org Home

Bill Gates Strikes Again: Open-source automated essay-scorer will be released in June 2013

by Susan Ohanian

Essays scored by computer? Thank Bill Gates, again finding ways to standardize every teaching decision.

In 2012, the Bill and Melinda Gates Foundation gave $1 million to Massachusetts Institute of Technology (MIT) to develop and offer a new, free prototype computer science online course through EdX. This is a collaboration between MIT and Harvard. In April 2013 EdX made an announcement:

Today, we are realizing this vision as we announce that the entire edX learning platform will be released as open source on June 1, 2013. The release will include the learning platform; edX Studio, our course authoring tool; edX101, an online course on how to teach an online course on edX; and assessment tools, such as our peer grader and our AI grader based on machine learning.

An online course on how to teach an online course. The ultimate in avoiding human contact.

About edX

EdX is a not-for-profit enterprise of its founding partners Harvard University and the Massachusetts Institute of Technology focused on transforming online and on-campus learning through groundbreaking methodologies, game-like experiences and cutting-edge research. EdX provides inspirational and transformative knowledge to students of all ages, social status, and income who form worldwide communities of learners. EdX uses its open source technology to transcend physical and social borders. We're focused on people, not profit. EdX is based in Cambridge, Massachusetts in the USA.

In some announcements it's "ed.X"; in others it's Ed.X. But no matter how you capitalize it, this is definitely Gates-Speak: It is one more nail in the coffin of teacher decision-making. We're focused on people. Indeed.

NOTE: Being a non-profit doesn't mean that the people there aren't well paid for their work.

The thing to remember is that Bill Gates has so much money he can buy anything he wants. He has an aversion to teachers making individual, idiosyncratic decisions and so he funds technology to streamline, standardize, and ultimately eliminate teachers.

Below the New York Times article you'll find an essay question response written by Les Perelman (courtesy http://lists.pureparents.org/listinfo.cgi/chicagotestingresistance-pureparents.org), longtime director of writing at the Massachusetts Institute of Technology for 26 years. This essay was awarded a top grade of 6 by e-Rater, the automated grader developed by ETS. Michael Winerip summarized Perelman's findings about the 3-Rater in Facing a Robo-Grader? Just Keep Obfuscating Mellifluously , April 2012.

I looked in vain for any mention of the Bill and Melinda Gates Foundation in this New York times article, but kudos to the reporter for mentioning the petition against machine scoring: Professionals Against Machine Scoring of Student Essays in High-Stakes Assessment.
Reporters don't often give such 'voice' to the opposition.

You can sign the petition.

This whole Gates-funded EdX deal is based on three insidious propositions, alluded to in the times article below:

1) student writing iimproves by rewriting the same essay over and over and over. And over.

2) with computer grading of tests, classes can be larger;

3) folks at prestigious institutions are smart enough to give good feedback; other folk aren't.

A commenter at the NYT makes an important point: Robo-grading is a helpful handmaiden to high-stakes standardized testing. Together, they will yield even more hollow data.

Oakland, CA teacherJessica Langlois' comment at the NYT has received 289 recommendations--so far:

Grading papers takes a whale of a time, time that is often unpaid, and yet, it seems insane to not read what you have been endeavoring to teach your students all semester.

I see my students for 3 hours every week for 15 weeks. But I come to know them pretty darn well, and I want to know what they are thinking, what they are writing. Some of what I have to digest is lazy, incomprehensible or even problematic work. But some of it is really smart, and, in the best of it, you can see a student move from glib writing, peppered with arbitrary quotes, to cogent analysis, or to a clever, gripping story. This is the life blood of teachers, baby. And you get all mushy about it on Facebook, bragging to your friends about your brilliant student, and you tell that student to send out their work as an op-ed. This is the life blood of our world, darlings.

All this money going to develop software that will save universities money and supposedly democratize education. Why not just pay real, sentient teachers more for their grading time, enabling them to establish a stronger connection to and investment in their students--that would be the true democratization of education.

I like the way another commenter ended his observations for Winerip's column last year: The Webster's Unabridged Dictionary contains 470,283 words. While I only know a fraction of them, my computer can spell every one.

All hail HAL!

Essay-Grading Software Offers Professors a Break

New York Times, April 3, 2013

By John Markoff

Imagine taking a college exam, and, instead of handing in a blue book and getting a grade from a professor a few weeks later, clicking the "send" button when you are done and receiving a grade back instantly, your essay scored by a software program.

And then, instead of being done with that exam, imagine that the system would immediately let you rewrite the test to try to improve your grade.

EdX, the nonprofit enterprise founded by Harvard and the Massachusetts Institute of Technology to offer courses on the Internet, has just introduced such a system and will make its automated software available free on the Web to any institution that wants to use it. The software uses artificial intelligence to grade student essays and short written answers, freeing professors for other tasks.

The new service will bring the educational consortium into a growing conflict over the role of automation in education. Although automated grading systems for multiple-choice and true-false tests are now widespread, the use of artificial intelligence technology to grade essay answers has not yet received widespread endorsement by educators and has many critics.

Anant Agarwal, an electrical engineer who is president of EdX, predicted that the instant-grading software would be a useful pedagogical tool, enabling students to take tests and write essays over and over and improve the quality of their answers. He said the technology would offer distinct advantages over the traditional classroom system, where students often wait days or weeks for grades.

"There is a huge value in learning with instant feedback," Dr. Agarwal said. “Students are telling us they learn much better with instant feedback.”

But skeptics say the automated system is no match for live teachers. One longtime critic, Les Perelman, has drawn national attention several times for putting together nonsense essays that have fooled software grading programs into giving high marks. He has also been highly critical of studies that purport to show that the software compares well to human graders.

"My first and greatest objection to the research is that they did not have any valid statistical test comparing the software directly to human graders," said Mr. Perelman, a retired director of writing and a current researcher at M.I.T.

He is among a group of educators who last month began circulating a petition opposing automated assessment software. The group, which calls itself Professionals Against Machine Scoring of Student Essays in High-Stakes Assessment, has collected nearly 2,000 signatures, including some from luminaries like Noam Chomsky.

"Let's face the realities of automatic essay scoring," the group's statement reads in part. "Computers cannot 'read.' They cannot measure the essentials of effective written communication: accuracy, reasoning, adequacy of evidence, good sense, ethical stance, convincing argument, meaningful organization, clarity, and veracity, among others."

But EdX expects its software to be adopted widely by schools and universities. EdX offers free online classes from Harvard, M.I.T. and the University of California, Berkeley; this fall, it will add classes from Wellesley, Georgetown and the University of Texas. In all, 12 universities participate in EdX, which offers certificates for course completion and has said that it plans to continue to expand next year, including adding international schools.

The EdX assessment tool requires human teachers, or graders, to first grade 100 essays or essay questions. The system then uses a variety of machine-learning techniques to train itself to be able to grade any number of essays or answers automatically and almost instantaneously.

The software will assign a grade depending on the scoring system created by the teacher, whether it is a letter grade or numerical rank. It will also provide general feedback, like telling a student whether an answer was on topic or not.

Dr. Agarwal said he believed that the software was nearing the capability of human grading.

"This is machine learning and there is a long way to go, but it's good enough and the upside is huge," he said. "We found that the quality of the grading is similar to the variation you find from instructor to instructor."

EdX is not the first to use automated assessment technology, which dates to early mainframe computers in the 1960s. There is now a range of companies offering commercial programs to grade written test answers, and four states -- Louisiana, North Dakota, Utah and West Virginia -- are using some form of the technology in secondary schools. A fifth, Indiana, has experimented with it. In some cases the software is used as a "second reader," to check the reliability of the human graders.

But the growing influence of the EdX consortium to set standards is likely to give the technology a boost. On Tuesday, Stanford announced that it would work with EdX to develop a joint educational system that will incorporate the automated assessment technology.

Two start-ups, Coursera and Udacity, recently founded by Stanford faculty members to create "massive open online courses," or MOOCs, are also committed to automated assessment systems because of the value of instant feedback.

"It allows students to get immediate feedback on their work, so that learning turns into a game, with students naturally gravitating toward resubmitting the work until they get it right," said Daphne Koller, a computer scientist and a founder of Coursera.

Last year the Hewlett Foundation, a grant-making organization set up by one of the Hewlett-Packard founders and his wife, sponsored two $100,000 prizes aimed at improving software that grades essays and short answers. More than 150 teams entered each category. A winner of one of the Hewlett contests, Vik Paruchuri, was hired by EdX to help design its assessment software.

"One of our focuses is to help kids learn how to think critically," said Victor Vuchic, a program officer at the Hewlett Foundation. "It's probably impossible to do that with multiple-choice tests. The challenge is that this requires human graders, and so they cost a lot more and they take a lot more time."

Mark D. Shermis, a professor at the University of Akron in Ohio, supervised the Hewlett Foundation's contest on automated essay scoring and wrote a paper about the experiment. In his view, the technology -- though imperfect -- has a place in educational settings.

With increasingly large classes, it is impossible for most teachers to give students meaningful feedback on writing assignments, he said. Plus, he noted, critics of the technology have tended to come from the nation's best universities, where the level of pedagogy is much better than at most schools.

"Often they come from very prestigious institutions where, in fact, they do a much better job of providing feedback than a machine ever could," Dr. Shermis said. "There seems to be a lack of appreciation of what is actually going on in the real world."

Les Perlman's essay submitted for machine scoring and receiving top marks:

Here's Perelman top-rated essay:

Question: "The rising cost of a college education is the fault of
students who demand that colleges offer students luxuries unheard of
by earlier generations of college students -- single dorm rooms,
private bathrooms, gourmet meals, etc."

Discuss the extent to which you agree or disagree with this opinion.
Support your views with specific reasons and examples from your own
experience, observations, or reading.

In today's society, college is ambiguous. We need it to live, but we
also need it to love. Moreover, without college most of the world's
learning would be egregious. College, however, has myriad costs. One
of the most important issues facing the world is how to reduce college
costs. Some have argued that college costs are due to the luxuries
students now expect. Others have argued that the costs are a result of
athletics. In reality, high college costs are the result of excessive
pay for teaching assistants.

I live in a luxury dorm. In reality, it costs no more than rat
infested rooms at a Motel Six. The best minds of my generation were
destroyed by madness, starving hysterical naked, and publishing
obscene odes on the windows of the skull. Luxury dorms pay for
themselves because they generate thousand and thousands of dollars of
revenue. In the Middle Ages, the University of Paris grew because it
provided comfortable accommodations for each of its students, large
rooms with servants and legs of mutton. Although they are expensive,
these rooms are necessary to learning. The second reason for the
five-paragraph theme is that it makes you focus on a single topic.
Some people start writing on the usual topic, like TV commercials, and
they wind up all over the place, talking about where TV came from or
capitalism or health foods or whatever. But with only five paragraphs
and one topic you’re not tempted to get beyond your original idea,
like commercials are a good source of information about products. You
give your three examples, and zap! you’re done. This is another way
the five-paragraph theme keeps you from thinking too much.

Teaching assistants are paid an excessive amount of money. The average
teaching assistant makes six times as much money as college
presidents. In addition, they often receive a plethora of extra
benefits such as private jets, vacations in the south seas, a staring
roles in motion pictures. Moreover, in the Dickens novel Great
Expectation, Pip makes his fortune by being a teaching assistant. It
doesn’t matter what the subject is, since there are three parts to
everything you can think of. If you can’t think of more than two, you
just have to think harder or come up with something that might fit. An
example will often work, like the three causes of the Civil War or
abortion or reasons why the ridiculous twenty-one-year-old limit for
drinking alcohol should be abolished. A worse problem is when you wind
up with more than three subtopics, since sometimes you want to talk
about all of them.

There are three main reasons while Teaching Assistants receive such
high remuneration. First, they have the most powerful union in the
United States. Their union is greater than the Teamsters or
Freemasons, although it is slightly smaller than the international
secret society of the Jedi Knights. Second, most teaching assistants
have political connections, from being children of judges and
governors to being the brothers and sisters of kings and princes. In
Heart of Darkness, Mr. Kurtz is a teaching assistant because of his
connections, and he ruins all the universities that employ him.
Finally, teaching assistants are able to exercise mind control over
the rest of the university community. The last reason to write this
way is the most important. Once you have it down, you can use it for
practically anything. Does God exist? Well, you can say yes and give
three reasons, or no and give three different reasons. It doesn’t
really matter. You’re sure to get a good grade whatever you pick to
put into the formula. And that’s the real reason for education, to get
those good grades without thinking too much and using up too much

In conclusion, as Oscar Wilde said, "I can resist everything except
temptation." Luxury dorms are not the problem. The problem is greedy
teaching assistants. It gives me an organizational scheme that looks
like an essay, it limits my focus to one topic and three subtopics so
I don’t wander about thinking irrelevant thoughts, and it will be
useful for whatever writing I do in any subject.1 I don’t know why
some teachers seem to dislike it so much. They must have a different
idea about education than I do.

— Susan Ohanian





This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.