The title promises truth, and the book does deliver healthy doses of difficult truths. Ericsson and Haswell start out in a challenging tone by saying composition’s response to AI evaluators “has been silence.” Reinforcing this point, Haswell later asks why are we “watching, helpless, as automatons take over our skilled labor” (58)? Simple, he says: “We are where we are because for a long time now we have been asking for it” (59). As they say, the truth hurts, and this will indeed leave some people smarting.
"We are where we are because for a long time now we have been asking for it" --Richard Haswell |
“Smarting” because the book shows that AI graders/scorers/evaluators operate on such flimsy rhetorical grounds that it is shocking how prevalent they have become in writing assessment. The book provides a cocktail-party level knowledge of how these machines operate, although, as Ericsson and others point out, few people really know: most of these machines’ operations are proprietary. Intelligent Essay Assessor (IEA) claims it “understands the meaning of text” (28), but Ericsson works us through the equation-like approach IEA and other evaluators take, showing the obvious contradiction between computer meaning and what humans consider meaning based on the semiotic/rhetorical history of communication.
Many of the contributors game these systems in useful—and often entertaining—ways. Chris Anson demonstrates the flaws of Summary Street by mixing up the facts in his test submissions and showing the machine’s inane feedback, which reveals its inability to discriminate between even reversals of meaning. Timothy McGee takes “a spin” on IEA. (If you want to impress upon someone the limitations of AI evaluators, you must read McGee’s essay.) He shows specific examples of how his efforts to reverse meaning and even to submit gibberish had little effect on IEA’s feedback. Edmund Jones fools Accuplacer, finding it values length “far more than any teacher of writing would” (101), that it has a narrow view of correctness, and (as McGee also found) that sentence order appears irrelevant to its scoring. Makers of these programs claim they require “good-faith” efforts by students, but let’s be serious: How long will it take before screenagers publish all the tricks to beat these machines on their FaceBook sites?