Three-Toed Sloth

All of Peer Review

Larry "All of Statistics/All of Nonparametric Statistics" Wasserman has just started a weblog, The Academic Curmudgeon. By my count, this brings the number of faculty blogs in the department to three (and the proportion to about 0.09); none of our students seem to have any. Larry's second post is about peer review.

Here is a summary of common reasons for rejecting papers:

Good Reasons For Rejecting a Paper:
1. The results are incorrect (unfixable, critical errors).
2. The results are not new.

Bad Reasons For Rejecting a Paper:
3. The referee doesn't like the paper.
4. The referee doesn't like the author's approach.

Tricky:
5. The contribution of the paper is too small.

Numbers 3 and 4 are bad because they are based on the taste of the referee which is far too subjective. Number 5 is problematic. True, we don't necessarily want top journals publishing every small idea that occurs to someone. The problem is this: almost all research, including good research, is incremental. The idea that most papers in top journals are breakthough papers is fantasy. What is too incremental to publish is highly subjective.

I think the basic problem is that most referees have the wrong view of the purpose of publishing. Ideally, publishing is about disseminating knowledge. It should not be regarded as admittance to a high and mighty priesthood.

I am going to use Larry's post as an excuse to ruminate about peer review, partly because I spent yesterday whittling down the stack of manuscripts I'd agreed, in weak moments, to review, and partly because I think Larry might be unhappy with my refereeing. Maybe he gets to see a better class of manuscript than I do (I wouldn't be surprised), but it seems to me that his list of good reasons to reject a paper is seriously incomplete.

1. The paper is so ill-written that it is not comprehensible with reasonable effort. (This is distinct from bad English, where I generally respond with a long list of corrections.)
2. The paper is comprehensible, but so vague that one cannot tell what was actually done.
3. One can tell what was done, but none of it amounts to a result — the experiments are inconclusive at best, the simulations wander aimlessly, the math is mere algebraic noodling.
4. There are results alright, but they're all old ones with a new set of labels attached to the variables. (Algorithmic information theory and the logistic map are favorite targets for this treatment.)
5. There are new results, but they belong in a different journal. (Physical Review keeps sending me papers that are, as far as I can tell, perfectly decent and straightforward ventures into population dynamics, and I keep telling the authors that there are journals like Theoretical Population Biology and the Journal of Theoretical Biology...)

I am happy to reject papers on such grounds, because they seem to follow from my understanding of the point of peer review. This is, as Larry says, about the dissemination of knowledge, not initiation into a priesthood. But the people peer-reviewers serve are not the authors but the potential readers. Passing peer review ought to endorse a manuscript, not as correct, but as possibly worthy of attention: not obviously wrong, not disconnected from the field, not out to lunch, not a waste of the reader's time. (This is more modest, and more achievable, than actively picking out the good stuff; it's a type I/type II error issue.) One important wrinkle here is that, if you're already an expert in a given field, the extra value of having somebody else filter the stream of manuscripts in that field is small — part of your expertise is being able to make such judgments reliably and cheaply yourself. But if you need to use results or ideas from another field, then you either need to become an expert there, too, or you need experts there to tell you what's worth attention. Very few scientists never need ideas from other fields, which is to say that most of us will benefit from peer review. (Similarly for hiring and tenure, but let's not dwell on such unpleasant subjects.) For complete non-experts, i.e., the lay public who ultimately support us all, peer review is about the only way they have of telling possibly-legitimate scientists from the cranks and the frauds. (More exactly, because peer review only says "not obviously wrong": anyone who can't get over the peer review barrier is so weak as to not be worth bothering with.)

I rather doubt, however, that the current journal/peer-review system is the ideal way of doing this filtering. Journals can be too conservative. Journals can not be conservative enough, when the topic is fashionable. (Not that I have anything in mind.) Journals can get locked into a vicious cycle in which they become so bad that publication there constitutes an anti-endorsement, so that only really bad scientists publish there, and they in turn become recruited as referees. (Still, an anti-endorsement is not without its own value.) There is something perverse about refereeing for commercial publishers, since publishers charge scientists larcenous rates to bring them the results of free labor on the part of authors and referees.

Ultimately, I hope that we move away from the current system towards something more like Paul Ginsparg's ideas. He envisions a system of "tiers" of publication; the lowest tiers, like the current arxiv.org, would have nearly-open submission and dissemination, and be most valuable to experts. Above them, operating more slowly, would be more selective tiers of peer review, commentary, review papers, etc., which will be more valuable to less expert readers, and won't try to filter the whole manuscript stream, which is what peer review now does. Getting there from here will probably involve lining the war-mongering parasites at Elsevier up against a wall a good deal of time and effort, but would lead to something much more efficient and intellectually valuable.

And now, back to revising our manuscript to please its referees.

Update, 29 November 2005: Dave Feldman, propelled by the burning need to procrastinate which drives so many academics to blogging, suggests that the peer review system would be much improved by free socks. He's right.

Update, 7 December 2005: For once, I wish I had comments here. (I don't feel like committing myself to the endless struggle against spam.) "Cog", of The Abstract Factory, writes in (quoted with permission):

I believe your 3 and 4 are subspecies of "the contribution is too small" (5 on Larry's list). Your 1 and 2 are both subspecies of "the paper does not communicate effectively enough for the contribution to be evaluated", which is a new reason. Number 5 is also a new reason. So you've really come up with two distinct extra reasons.

I would actually subsume Larry's reason 1, and your 1+2 under:

1'. The results are not clearly shown to be (probably) correct.

Because, of course, the burden of proof is on the authors to convince the reader that their result is not obviously wrong, not on the reviewer to show that the result is obviously wrong. To discharge this obligation requires both technical soundness and effective, precise communication.

Manual Trackback: An Ergodic Walk

The Collective Use and Evolution of Concepts; Learned Folly