|
Articles Conference Reviews |
2007A19HartA.19: “Reperceiving Peer Review” In “How and How Helpfully do Peers Comment on Student Writing?”, David Charney begins the panel with a comparison of comments made by Writing In the Disciplines (WID) faculty, student peers, and faculty who are writing experts. She conducted her research within a class using the University of Pittsburgh’s Scaffolded Writing and Rewriting in the Discipline (SWoRD) software. First, Charney has examined “reciprocal evaluation in content courses” or “peer review.” She explains a model of peer evaluation in which the writers not only write essays and give their peers comments on those essays but also give their reviewers comments about the helpfulness of their comments. Additionally, the reviewers comment on and grade the writers’ final drafts, attending to the ways in which the writers did or did not respond to the original peer comments. The writers and reviewers also compare the various sets of comments each essay receives and compare the helpfulness ratings of those comments with other peers commenting on the same essay. The potential benefits to student writers of such “reciprocal evaluation” in large survey courses, proclaims Charney, are that writers receive lots of different perspectives on their writing, they take their revisions seriously, they learn about writing in different forms and in different ways, they are encouraged to reflect upon their various kinds of writing, and they are foregrounding audience in their writing. Additionally, since the writing grade is based solely upon peer reviews, the teachers do not grade the essays, thereby reducing teacher effort. Some of the potential concerns of using such a system, declared Charney, include low motivation to review, the peer reviewers’ lack of expertise in the course content and in writing, the peer reviewers’ lack of experience with commenting, and the validity and reliability of the final paper grades. Charney points out, however, there are often concerns about the kinds of comments that content experts give student writers as well. For example, subject-matter experts are apt to represent the content and genre differently than novices. In addition, experts may forget what novices do not know and are likely to use specialist language when responding to student writing. In fact, speculates Charney, students may have more similar viewpoints and better expectations of what their peers should know than the content experts. Armed with these speculative concerns and benefits, Charney sets about studying an American History survey course of 111 students using the SWoRD system to write and review essays in which their writing and reviewing make up 40% of their final grades. Each student writes a six to eight (6-8) page argument-driven essay and is required to review the papers of six (6) randomly selected peers. The peer reviewers are directed to write open-ended comments on the draft based upon three criteria: flow, argument, and insight. After the writer revises and submits the essay for grading, the peer reviewers assign the essay a numerical rating according to a prescribed rubric, and the writer’s grade is determined by the average of his or her peer reviewers’ ratings. The SWoRD system also provides the peer reviewers with their “rating accuracy” and allows the writers to grade their peers on feedback helpfulness. In addition to assessing the peer comments, Charney asks a content expert to review and comment upon fifty (50) of the student drafts and a writing expert to review and comment upon fifty (50) of the student drafts. She then analyzes the range of comments for twenty-four (24) mid- to low-rated final papers. In her analysis of the three types of comments, Charney has noticed that problem/solution comments were the most common and that the content expert writes the largest number of these kinds of comments. Comments of praise are less common and are most likely to be written by students, followed by the writing expert. Summative comments are the least common type of comment. Within the problem/solution comments, Charney finds that the writing expert makes more global comments and gives more attention to providing solutions and strategies for revision along with detailed explanation within the comments, while the content expert is more focused on the substance of the essay and points out more problems while offering the fewest solutions among the three groups of respondents. The content expert and student reviewers are also more interested in paragraph and word- and sentence-level issues than the writing expert. Despite these differences among the type and amount of comments, Charney finds that the student reviewers, the content expert, and the writing expert tend to rate the essays similarly. She concludes that the kind of writing process facilitated by electronic peer review and response programs such as SWoRD does lead to better student writing. Within such a system, students do write and review and tend to like doing so, she claims. Such systems also provide a way for content and writing experts to share their expertise (through modeling) as well as a way for students to learn more about reviewing and to get reviews that are helpful to their writing. Prior to analyzing the specific scientific peer review controversy mentioned in the title, “Genre, Ideology, and Values: Analysis of Scientific Peer Review Controversy,” Necia Werner next discusses how she introduces her students to genres not just as regenerated forms or “typified rhetorical responses to recurrent situations,” but as forms that do something in the world. As a teacher, she regards genres as “the keys to understanding how to participate in the actions of one’s community” since a knowledge of genres provides an individual with a “repertoire of situationally appropriate responses.” Recognizing, with Berkenkotter and Huckin, that genres “are intimately linked to a discipline’s methodology,” Werner wonders what sorts of communication genres might encourage and what sorts they might constrain. She also wonders what values and beliefs might be contained within a generic set of practices. To begin to try to answer this question, she has taken an in-depth look at the Peters and Ceci controversy of 1982. In 1982, two cognitive scientists, Peters and Ceci, published an article titled “Peer-review practices of psychological journals: The fate of published articles, submitted again.” In this article, the authors conduct an experiment in which they fictionalized the names and institutions of the authors of twelve psychology articles, articles that have actually been written by authors from prestigious institutions and already published in top-tier American psychology journals. They resubmit these now pseudonymous articles to the same journals more than a year later with the following results: three (3) of the articles are recognized as already having been published, only one (1) of the remaining articles is accepted, with the other eight (8) being rejected on grounds of serious methodological defects and/or poor writing style. From these results, Peters and Ceci conclude that the prestige of authors and institutions significantly affects an article’s acceptance rate. After their own article is rejected by a few journals, the editor of The Behavioral and Brain Sciences decides to accept it on its “provocative basis for an open analysis of peer review,” and therefore devotes an entire volume to the Peters and Ceci article and solicited responses to it. Werner analyzes the peer responses to try to determine, among other things, how the respondents to the article (all, not coincidentally journal editors or reviewers themselves) use evidence to justify their responses, what they see as the purpose of the peer review process, how they view their roles as editors and reviewers in relationship to the authors of the articles, and what role ethics plays in the discussion. She pays particular attention to the role of opinion—to the use of the phrases “I think” or “I don’t think”—in the evaluation of the controversy and finds such evaluative opinions fall into six broad categories:
Werner finds the articulation of opinions in this last category to be the most surprising since the respondents seem to indicate reviewers and editors rarely feel compelled to provide authors with “direct remarks” or to “specify exactly the criteria for their decisions about the manuscript” and instead fall back upon replies that “point out some obvious flaws or beg off in some other way.” As classroom teachers, we should be aware of these tendencies among professional reviewers to avoid direct or specific remarks and should look out for these same tendencies among our students. We need to help our students to think about whether or not what they are saying on the papers they are reviewing is really what they want to say. In addition, by pointing out the potential limitations of the peer review process, Werner hopes to stretch her students to see such practices as rhetorically motivated and to consider how they might affect what they do as writers in the workplace and the world. In “Peer Review: Differences in Student and Professional Strategies,” Christine Neuwirth first acknowledges the assistance of Mike Palmquist and Tom Hajduk in her research on the differences between experienced and novice commentators in the peer review process. Second, Neuwirth discusses three possible ways of reflecting upon writing: 1) Externalizing the line of argument. This type of reflection, suggests Neuwirth, aids writers in constructing top-level “gists” of their own texts, in helping them to figure out what it is they want to say. Neuwirth speculates that if writers and commentators conduct such reflection in interactive oral sessions (e.g. a writer telling his or her partner about the main points of the paper) then they are able to avoid focusing on low-level details and therefore increase the chances of improving the writer’s performance on subsequent drafts. 2) Externalizing conflicting perspectives. Neuwirth describes this process as one that leads writers to create “moments of inquiry.” By playing “devil’s advocate,” reviewers aid writers to extend, modify, and revise their beliefs about topic and audience. Again, Neuwirth speculates that successful reflection in this model results in better final products. 3) Externalizing interaction. In this model of reflection, the writers reflect upon the comments they have been given by their reviewer(s). Neuwirth then describes her research process as she pairs “More Experienced Writers” (MEWs) with other MEWs and “Less Experienced Writers” (LEWs) with other LEWs. The criteria to be a MEW include membership in the English profession and a record of publication. The LEWs are freshmen writers paid to participate in this study. All of the writers are given instructions to write a persuasive essay on a topic that draws upon familiar college-related experiences (e.g. a social group or activity that many students encounter in college that may affect them in important ways). They are told who their target audience is (a statewide panel of policy-makers who had to be persuaded to adopt the writer’s position), and they are given an hour in which to write an essay. The experiment involves 12 MEWs and 12 LEWs and is conducted over the course of three weeks according to the following schedule:
For the FTF and CMC reviews, reviewers are instructed to play the “devil’s advocate” on the first drafts. For the self-reviews, writers act as their own devil’s advocates. Neuwirth measures each essay’s improvement according to a rating scale developed by Wallace et al. in the Journal of Educational Psychology, which records negative changes, no improvement, minor positive changes, and major positive changes. She also measures the number of challenges the reviewers put forth. Her results show that the MEWs have higher scores on improvement quality, but the trends are not statistically significant. The MEWs also seem to have more challenges presented to them during the FTF review sessions. She also records a positive correlation between the number of challenges and the amount of improvement for the LEWs during the CMC review sessions. In conclusion, Neuwirth notes that the LEWs are more likely to use a simple point opposition strategy in their challenges, while the MEWs focus more upon either a structure-of-argument-based strategy (which she referrs to as a topoi strategy) or a rhetorical-situation-based strategy (which she refers to as an audience/purpose strategy). In general, she finds that the MEWs employ far more rhetorical strategies than the LEWS, that as reviewers they tend to replicate more closely the process of composing, and that they keep audience and purpose in mind more readily than the LEWs. The MEWs also address a higher percentage of the challenges in their revisions, according to Neuwirth. Q&A Charney responds to a question on the development of the rubric the students in the SWoRD course use to rate other students’ essays as well as the language the rubric employs by stating that the rubric scores primary traits of the writing in three areas: flow, argument, and insight. Asked if the grades for the essays are simply averages of the six responses, Charney replies that the grades are true averages. Next, Werner is asked about changes in the genres of scholarly publications that now include other media and how these changes might influence the academic review process. Werner responds that there have been some developments in trying to create automatic ways to rate reviews (e.g. NSF proposals) but the process cannot be automated “by brute force.” Finally, Neuwirth is asked about the challenges for student peer reviewers, to which she responds that when the reviewers are trained in the role of devil’s advocate they are able to present the challenges as criticisms not personal attacks. Comments? |