Teachers vs. TechnologyâRound One, Fight!
It was our last day at the 2012 Conference of College Composition and Communication (CCCC) in St. Louis, the biggest annual gathering of college writing teachers in the country. We were done with our own presentation, and we could eat only so many ribs in a day . . . so we were looking through the conference program to see what else was on offer . . . when a full-page listing caught our eye. It was for a âSpecial Session.â Something particularly Big and Important. This Special Session, the program told us, would be a âvigorous but civil debateâ on Automated Essay Scoring (AES), the technology that lets computers score writing assignments.
We were intrigued. When an academic conference has to specify that an event will be âcivil,â you know somethingâs up.
The ballroom was vast and long. There were about two hundred seats facing a long table and a giant PowerPoint screen. At one end of the table sat Paul Deane and Chaitanya Ramineini, researchers from Education Technology Services (ETS): the College Board, the minds behind standardized tests like the SAT, GRE, and AP exams, andâalong with their for-profit colleagues Pearson and Vantageâthe loudest promoters of AES. They were here today to present their newest innovation: the e-RaterÂŽ Scoring Engine, a computer program thatâaccording to an impressive set of studiesâcan score a student essay with the reliability of a human.
At the other end of the table sat Les Perelman, the longtime (now emeritus) Director of Undergraduate Writing at MITâand the best-publicized crusader against AES. He authored the first paragraph of this section, which he fed, along with another page-and-a-half of similar drivel, into e-RaterÂŽ. And it got a 6, the highest-possible score. The point seemed clear and obvious: machines canât read. They canât understand. And they canât, and certainly shouldnât, replace humans in educating our children.
(Between them, fittingly, sat Carl Whithaus, director of the University Writing Program at UC Davis, who works to integrate AES into human-centered teaching. Weâll get back to him later.)
Les was the perfect foil to Deane and Ramineini. They were impeccably dressed; he wore a suit that didnât fit quite right and looked a little sweaty. Their speech was memorized and newscaster-perfect; he spoke off the cuff and made no effort to hide a general-purpose ethnic accent. Their PowerPoints were sleek, branded, and serious; his was homemade and funny, complete with gifs of rabbit holes and the Twilight Zone.
None of this was by accident. They were each playing their parts to a T: the bloodlessly efficient technocrats versus the righteous, rumpled, lone defender of flesh-and-blood teaching. âVigorous but civilâ be damnedâthis was a brawl. An agon. A pundit war between good and evil, man and machine. We, the human educators of CCCC, sat and winced as the forces of evil took their best shotâand then rejoiced as our champion tore them limb from limb, down to their last unfounded assumption and logical fallacy.
He was, after all, fighting for our lives. Or at least our livelihoods. Humanistic objections to robo-teaching aside, we were precisely those humans AES stands poised to eradicate. AES, which can grade thousands of student essays in mere seconds, could convince a budget-wary public that small class sizes and individualized instruction are unnecessary luxuries. In a political moment where education funding is under constant attack, itâs not hard to imagine administrators and elected officials (even the well-meaning ones) using AES as a rationale to lay us off. So when we see Les up there proving that AES doesnât work, wonât work, canât work, of course we cheer loudly.
The problem is, outside the cozy confines of CCCC, heâs losing. Weâre losing. Education technologyâcomputer-based classroom instruction, online Kâ12 schools, MOOCs (Massive Open Online Courses), as well as AESâis all the rage. Itâs whatâs getting all the media attention, foundation funding, and government grants. Itâs the cutting edge, the thing all forward-thinking principals, superintendents, policymakers, and executives (both public and private) are talking about. Lesâs dogged defense of old-fashioned teaching might make us cheer, but itâs making everyone else yawn.
And maybe most frustrating of all, the science all seems to be on their side. Sure, Lesâs argument makes intuitive senseâ how could you trust a program that gave that pile of s#!t he wrote the highest-possible score?âbut then again, how can you argue with scores of reliable, data-driven studies?
This is the problem weâll tackle in this book. Weâll explain the technology, how it works, and what it does (and doesnât do). Weâll explain where it comes from, and how itâs come to take the form that it has, in the classroom and in our culture. Weâll also explain why the debate over the technology has taken the shape it hasâa shape that stops us from understanding the real problem or doing anything about it. Finally, weâll explain what that real problem is, and how weâteachers, students, citizensâcan work to solve it.
Because there is a problem. Itâs just quite a bit subtler, tougher, and more complicated than the standard âcivil debateâ would have us believe.
The Fight over Computer Grading (Started before Computer Grading)
Human versus machine. Good versus evil. Teachers versus technology. Kids versus computers. This is a Big Issue, and not just for writing teachers.
A few weeks after the conference, it made the New York Times. âFacing a Robo-Grader? Just Keep Obfuscating Mellifluously,â read the headline. The story, by longtime Times reporter Michael Winerip, starts with a recent study that âconcluded that computers are capable of scoring essays on standardized tests as well as human beings doââat the rate of 48,000 essays per minute.
It gives some space to ETS representatives, including Paul Deane. But most of the story belongs to Les, who demolishes the software with his usual gusto:
The ETS representatives make some important points in response, which weâll get to later. But they donâtâand canâtârefute Lesâs basic claims. Computers canât read. They canât understand truth. They can only grade by counting things: length of essays, length of sentences, and length of words. So, obviously, they shouldnât be used in place of humans. Right?
Not so fast. Take a look at this: