Are Reviewers Too Powerful?
| Nicolai Foss|
Reviewers certainly are powerful. Are they too powerful?
When I served as Departmental Editor of the Journal of International Business Studies it occassionally happened that I issued invitations to revise and resubmit , against the advice of the reviewers. I often accepted papers for publication that at least one and sometimes two reviewers hated. Once it happened that after I had accepted such a paper, a very dissatisfied reviewer — a prominent Wharton scholar — wrote to the chief editor, complaining that I was undermining the refereeing institution. Well, I thought the reviewer was wrong and that I (and the author) was right. And I thought I had no obligation to slavishly follow his advice, which was just that, a piece of advice, and not a verdict.One more anecdote: Sometime ago I submitted a paper to an edited book volume for which the editor had the individual chapters reviewed (which, of course, is just fine). My paper was reviewed over two rounds. The final report I received from the reviewer contained a number of points that were either irrelevant or had in actuality been dealt with in the revised paper. I pointed this out to the editor, but the latter insisted that I carefully reply to the reviewer: “‘Failing to see the point’ is not, usually, an option, other than in exceptional cases.” Well, why exactly? One would have thought that an editor would be capable of making up his own mind: If he felt that my response in the revised version was inadequate, he could tell me (and choose to reject me). Rather, I was told that not responding in a “full-disclosure” manner to any comment a reviewer might make, however silly, “is not an option.” Hmmmm….
Unfortunately, in my view, reviewers are increasingly seen less as advicers and more as a jury or even a set of judges. The independently minded editors — think Robert Clower as editor of the American Economic Review — seem to be dying out. One explanation may be simple laziness: It is so much less time-consuming to essentially give over all your discretion to the reviewers than it is to have to actually read (gasp!) the paper and form your own opinion. If that is the real driver, we might as well do away entirely with editors (except, perhaps, for the purpose of selecting reviewers), and completely automatize the process. Many journals have the kind of software that could handle this. Thus, the fate of a paper in this scenario is wholly and predictably determined by reviewer voting.
Abstracting from laziness, editors may reason that the one to four selected reviewers are all experts and that he is in no position to overrule those experts. Well, we all know how often even budding PhD students review for even the major journals. Those of us who have served as editors know how difficult it is to get the right reviewers and to construct a portfolio of reviewers for a given paper. We all have examples of reviews from top journals that are just plainly incompetent. In some cases — think Academy of Management Journal – relying very much on the reviewers is a completely reasonably approach, because that journal does indeed seems capable of picking real experts to review submitted papers. However, I far from convinced that this is the general situation, even among the supposedly top journals.
UPDATE 1: Some of the issues discussed above are treated in this great paper by Bruno Frey.