Discussion: Paper review guidelines

From Joe Pater (pater@linguist.umass.edu)

Bruce Hayes has agreed to let me post the following draft review guidelines he came up with a little while ago (though see his caveat below). I thought that putting these in the public domain might inspire reviewers to better practices, and perhaps also inspire journals to provide explicit guidance of this sort to their reviewers. I’d be interested to know about other sorts of advice that could be usefully given, and if there are existing explicit guidelines that people know of.

Dear Joe,
Sure, feel free to circulate what I wrote.  Looking it over now, I realize it is far too blunt and bullying to serve as an actual “Instructions for Referees” document and I hope people will read it instead as a spontaneous cri du coeur from an experienced/beleaguered Associate Editor.

Best,
Bruce

1. Please try to have your review completed within one month. If you anticipate your review taking longer, please contact the editor and obtain her approval first.

2. Your review should include both evaluative comments and a firm editorial recommendation.  The latter may range from “accept without change” to “reject without invitation to resubmit”.  The widely-employed intermediate recommendation, “revise and resubmit”, is too vague if it is not augmented with appropriate comments. Specifically, you can say what specific changes would need to be successfully implemented in order for the paper to become publishable. You can also say whether it is acceptable to you not to review the revised version; i.e. whether you want simply to trust the author to make the appropriate changes.

Do not use “revise and resubmit” merely as a euphemism, intended to be kindly; while a recommendation for outright rejection may seem harsher, it is likely to spare many people unnecessary work. It may also be more helpful for the journal’s goals of enforcing high scholarly standards.

3. A review can achieve greater credibility if it begins with a summary of the paper’s main point; this helps establish that the reviewer understands the paper he is reviewing.

4. Review the paper in its own terms.  If it relies on a theoretical framework that has attracted publication and attention, then you should assume that it is acceptable for this framework to be used; do not use the review process as an arena for framework-warring.  There is usually plenty to say about the quality of the evidence and reasoning keeping within the general assumptions of the paper.

5. Bear in mind that very long reviews can be problematic. They certainly generate extra work for authors and editors, and in the long run they change expectations about what is required of reviewers, so that reviewing becomes a heavy burden and it is difficult for editors to find volunteer reviewers. Long reviews are also likely to result in very long revised papers, crammed with passages that lack the authors’ own conviction but are intended to mollify skeptical  reviewers. You may wish to read over your completed review before submitting it, trimming back items that on reconsideration seem unlikely to result in improvement.

6. It goes without saying that a review should maintain a high degree of professionalism and courtesy to the author. Rereading and editing your review before submission with this in mind can be helpful.

7. If you like to note typos and grammatical infelicities, feel free to put them at the end of your report.  However, it is not required that you do this, and if it delays the completion of your review you should consider omitting this task.

4 thoughts on “Discussion: Paper review guidelines

  1. Anne-Michelle Tessier

    i like these guidelines very much.

    synthesizing two of Bruce’s points: as an AE, i feel my job is made easiest but also most likely to help the author succeed if the reviewers separate their remarks into “things that need to be addressed” and “thoughts i had while reading this”. … because reviewers are experts and can have lots of great thoughts that you don’t want to waste! but that doesnt mean all of those thoughts need to end up anywhere in this particular paper. if the reviewer finds that everything they have been moved to write would need to make it into a revised version, then probably the reviewer is rewriting the paper entirely — which either means they should reject the paper, or else they should reconsider why they have so many damn allegedly important thoughts. this two tiered system also often reveals that reviewers who seem very different in their assessments are actually much more in agreement — because they all have very different thoughts, but their key points that need addressing often show considerable overlap.

    Reply
  2. Bruce Hayes

    I think Anne-Michelle’s suggestion to split “essential criticisms” from “additional thoughts” is great. I’m often struck by how deeply involved in a paper reviewers can get, and Anne-Michelle’s idea permits such involvement to be benefit authors without overburdening them.

    Reply
  3. Sonya Bird

    Great suggestion AM! I always separate “contentful” comments from editorial comments (which I can’t help myself from making). I like the idea of separating out “thoughts” too, and will definitely do this in the future!

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *