Are Reviewers Too Powerful?

27 August 2006 at 10:50 am 12 comments

| Nicolai Foss|

Reviewers certainly are powerful. Are they too powerful?

When I served as Departmental Editor of the Journal of International Business Studies it occassionally happened that I issued invitations to revise and resubmit , against the advice of the reviewers. I often accepted papers for publication that at least one and sometimes two reviewers hated. Once it happened that after I had accepted such a paper, a very dissatisfied reviewer — a prominent Wharton scholar — wrote to the chief editor, complaining that I was undermining the refereeing institution. Well, I thought the reviewer was wrong and that I (and the author) was right. And I thought I had no obligation to slavishly follow his advice, which was just that, a piece of advice, and not a verdict.One more anecdote: Sometime ago I submitted a paper to an edited book volume for which the editor had the individual chapters reviewed (which, of course, is just fine). My paper was reviewed over two rounds. The final report I received from the reviewer contained a number of points that were either irrelevant or had in actuality been dealt with in the revised paper. I pointed this out to the editor, but the latter insisted that I carefully reply to the reviewer: “‘Failing to see the point’ is not, usually, an option, other than in exceptional cases.” Well, why exactly? One would have thought that an editor would be capable of making up his own mind: If he felt that my response in the revised version was inadequate, he could tell me (and choose to reject me). Rather, I was told that not responding in a “full-disclosure” manner to any comment a reviewer might make, however silly, “is not an option.” Hmmmm….

Unfortunately, in my view, reviewers are increasingly seen less as advicers and more as a jury or even a set of judges. The independently minded editors — think Robert Clower as editor of the American Economic Review — seem to be dying out. One explanation may be simple laziness: It is so much less time-consuming to essentially give over all your discretion to the reviewers than it is to have to actually read (gasp!) the paper and form your own opinion. If that is the real driver, we might as well do away entirely with editors (except, perhaps, for the purpose of selecting reviewers), and completely automatize the process. Many journals have the kind of software that could handle this. Thus, the fate of a paper in this scenario is wholly and predictably determined by reviewer voting.

Abstracting from laziness, editors may reason that the one to four selected reviewers are all experts and that he is in no position to overrule those experts. Well, we all know how often even budding PhD students review for even the major journals. Those of us who have served as editors know how difficult it is to get the right reviewers and to construct a portfolio of reviewers for a given paper. We all have examples of reviews from top journals that are just plainly incompetent. In some cases — think Academy of Management Journal – relying very much on the reviewers is a completely reasonably approach, because that journal does indeed seems capable of picking real experts to review submitted papers. However, I far from convinced that this is the general situation, even among the supposedly top journals.

UPDATE 1: Some of the issues discussed above are treated in this great paper by Bruno Frey.

Entry filed under: - Foss -, Ephemera. Tags: .

Who Killed the History of Economics? Editors, Reviewers, and Academic Judgment

12 Comments Add your own

  • 1. Bob V  |  27 August 2006 at 11:21 am

    Frankly this is the first time I’ve ever heard it suggested that reviewers aren’t supposed to decide whether a paper is to be accepted or not. (And this is saying something with my whole two years in academia and all.) My understanding was that the editor functioned only to decide what deserves to be reviewed and what occasionally break ties.

    One argument in favor of giving reviewers absolute power to reject a paper is that if they feel that their opinion can be overruled, they might not put as much effort into the reviewing activity itself.

  • 2. Nicolai Foss  |  27 August 2006 at 12:13 pm

    Bob,

    Sure, the editor decides “what occasionally break ties.” These are exactly the cases I am talking about. I agree that in most cases, if all four (or three reviewers) agree that a paper is bad, it most likely is bad. But I am worried about the increasingly prevalent practice of requiring unanimity among reviewers. That is a symptom of weak or lazy editors who will not make a real decision.

    Your argument is a good one. However, occasionally reviewers should be overruled. Reviewers make mistakes too. Here, as everywhere, is a tradeoff.

  • 3. Peter Klein  |  27 August 2006 at 2:02 pm

    Other strong and idiosyncratic editors, besides Clower, include Gordon Tullock at Public Choice and Murray Rothbard at the Journal of Libertarian Studies and the early Review of Austrian Economics. Both used referees sparingly, if at all, confident in their own judgments of quality. Tullock was also legendary for his quick turanounds — he usually responded to each author, personally, within a day or two of receiving a manuscript.

  • 4. Jung-Chin Shen  |  27 August 2006 at 2:02 pm

    I guess another possibility is that many editors are risk-averse: They tend not to overrule reviewers in order to attribute the rejection responsibilities to anonymous reviewers. As many journals are actually controlled by a small number of invisible academic community members (To cite an non-management journal, though it is not the real example in my mind because of its single-blind reviewing process, most Quarterly Journal of Economics authors have connections with either Harvard or MIT), such practice may enhance publishing reciprocity behaviors among the members of the community. For example, I know one journal that Nicolai likes very much has ever exhibited such tendency: some of their department editors accept each other’s papers. If quality is solely socially constructed, such practice will not hurt the reputation of the journal and may persist over time.

  • 5. Peter Klein  |  27 August 2006 at 2:06 pm

    Further to Jung-Chin’s point, it’s also a matter of Type I versus Type II error. Editors would rather not publish a paper that is later judged to be good than publish a paper that is later judged to be bad.

  • 6. Bob V  |  27 August 2006 at 3:14 pm

    I have been intrigued by Jung-Chin’s point. The hands-down most respected journal for folks like me to publish is Management Science. My perception is that Management Science is a clique. However, it isn’t as if everyone else gets annoyed and submits their best work to another journal. The best work still ends up in MS despite its apparent prejudice. How damaging is this, and how long can it be sustained?

  • 7. are reviewers too powerful? « orgtheory.net  |  27 August 2006 at 6:42 pm

    [...] Yes, says Nicolai Foss at Organizations & Markets – see his post. [...]

  • 8. Bo  |  29 August 2006 at 4:57 am

    Just a quick note to this extremely interesting and relevant discussion.

    While I may agree in principle that some editors are delegating too much power and decision-making to their reviewers, my understanding of this process is that part of the reason for having reviewers in the first place is to ensure a fair and essentially BLIND review process. The fact that the editor knows the identity of the author(s) presents somewhat of a problem to me because we all know that (intended or not) human beings tend to be biased – perhaps toward known names?

    If we want truly blind and fair reviews then we need to give most of the power to the reviewers. It is then the job of the editor to ensure that a given paper is reviewed by the best qualified reviewers. In my view – this is where the problem lies: how do we ensure quality reviews – even from the best qualified people (who are busy and may be negatively biased toward anything out of the ordinary: read their own literature tradition)?

    In my view, the role of the editor is not only to break up ties but to carefully weigh the comments of the reviewers in order to make the final decision. Reviewers make mistakes. Reviewers may be biased etc. Some arguments may be (and should be) stronger than others. For instance, in many to tier journals you will typically (at least in theory) get three reviewers, where one is a methodologist (may or may not be within your field), one is a theory person (perhaps in your field) and one is a more general type reviewer. How do you distinguish between the comments and recommendations from these three? It may be that the paper is rejected based on a poor methodology, yet the paper makes an extremely interesting theoretical contribution etc. The methods person may not see (or be willing to entertain) the possibility of revising this paper along these lines – even if it does mean a complete methodological makeover – yet if the editor believes in the novelty of the argumentation and that the methodology problems could be solved – perhaps this paper should be given at least a second chance?

  • 9. Peter Klein  |  29 August 2006 at 9:56 am

    Bo, two remarks: First, blind reviewing has been rendered technologically obsolete, courtesy of Google. Second, they key point on identities is not that the editor knows the author’s identity, but that the author knows the editor’s identity. Bias or shirking on the editor’s part affects the editor’s reputation. Bias or shirking on the anonymous reviewer’s part has a far smaller effect. (It may affect the reviewer’s reputation with the editor, but not with the author.)

  • 10. Bob V  |  29 August 2006 at 12:06 pm

    my understanding of this process is that part of the reason for having reviewers in the first place is to ensure a fair and essentially BLIND review process.

    Is this true? Most of the journals in my field are only single-blind (reviewers know who the author is) rather than double-blind (both reviewers and authors are in the dark). It would seem to me that the only reason to *not* have a double-blind process is to take the name of the author into account when reviewing the paper.

  • 11. Bo  |  30 August 2006 at 7:30 am

    In my field we have double blind review (I think and hope) – and I prefer this because, in theory at least, it ensures some kind of fair or due process. If you as a reviewer have time (and find it appropriate) to look through Google for who the author(s) are, then that seems to me to suggest that you are NOT intending to be fair – in essence you are biasing yourself. I have NEVER even considered looking for who the author might be on the web – why would I? Unless I somehow make this a criteria for my decision as to whether or not this paper is quality (sounds stupid to me)..

    In fact, I have several times returned the paper to the editor because I suspect I know who the authors are by their sample etc (and I happen to know these people personally). In these cases I do not feel that I should review the paper because I might be biased and would form opinions about my decision based on personal relationships rather than the quality and content of the paper. I know – this is because I am weak and cannot separate my personal life from my professional life etc – but who really can? I think it makes sense to send it back to the editor unless you really think that you are the only one in the world who can review this paper or who is THE expert in the field (sadly I am not – nor do I ever expect to be – one such expert)…

  • 12. Jung-Chin Shen  |  31 August 2006 at 10:00 am

    In the case of economics journals, two papers seem relevant to this thread.

    (1) The Editors and Authors of Economics Journals: A Case of Institutional Oligopoly? Geoffrey M. Hodgson and Harry Rothman, Economic Journal, 1999.

    Abstract: This paper examines data on the institutional backgrounds of editors and authors of the top 30 economics journals, identified by their 1995 citation impact. It is revealed, for example, that 70.8% of the journal editors were located in the United States, and twelve U.S. universities accounted for the location of more than 38.9%. Concerning journal article authors, 65.7% were located in U.S. institutions and twelve U.S. universities accounted for 21.8%. Arguably, the degree of institutional and geographical concentration of editors and authors may be unhealthy for innovative research in economics.

    (2) Diversity in Economics: An Analysis of Journal Quality Perceptions, Kostas Axarloglou and Vasilis Theoharakis, Journal of European Economic Association, December 2003, 1(6): 1402-1423.

    Abstract: It is still debatable whether scientific diversity is a virtue or a disadvantage for the develop-ment of a discipline. Nonetheless, diversity among scientists with respect to their journal quality perceptions plays an important role in hiring and promotion decisions. In this article we examine the degree of diversity within economics based on the journal quality perceptions of 2,103 AEA economists worldwide. Specifically, we empirically test for factors that might explain differences in an economist’s journal quality perceptions. These factors include an economist’s geographic origin, school of thought, journal affiliation, field of specialization and research orientation. Indeed, we find that a significant degree of diversity in journal quality perceptions exists between economists that belong in different subgroups. These results might explain the frequent debates in tenure and promotion committees where journal standings are used for the evaluation of a researcher’s output.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


Authors

Nicolai J. Foss | home | posts
Peter G. Klein | home | posts
Richard Langlois | home | posts
Lasse B. Lien | home | posts

Guests

Former Guests | posts

Networking

Recent Posts

Categories

Feeds

Our Recent Books

Nicolai J. Foss and Peter G. Klein, Organizing Entrepreneurial Judgment: A New Approach to the Firm (Cambridge University Press, 2012).
Peter G. Klein and Micheal E. Sykuta, eds., The Elgar Companion to Transaction Cost Economics (Edward Elgar, 2010).
Peter G. Klein, The Capitalist and the Entrepreneur: Essays on Organizations and Markets (Mises Institute, 2010).
Richard N. Langlois, The Dynamics of Industrial Capitalism: Schumpeter, Chandler, and the New Economy (Routledge, 2007).
Nicolai J. Foss, Strategy, Economic Organization, and the Knowledge Economy: The Coordination of Firms and Resources (Oxford University Press, 2005).
Raghu Garud, Arun Kumaraswamy, and Richard N. Langlois, eds., Managing in the Modular Age: Architectures, Networks and Organizations (Blackwell, 2003).
Nicolai J. Foss and Peter G. Klein, eds., Entrepreneurship and the Firm: Austrian Perspectives on Economic Organization (Elgar, 2002).
Nicolai J. Foss and Volker Mahnke, eds., Competence, Governance, and Entrepreneurship: Advances in Economic Strategy Research (Oxford, 2000).
Nicolai J. Foss and Paul L. Robertson, eds., Resources, Technology, and Strategy: Explorations in the Resource-based Perspective (Routledge, 2000).

Follow

Get every new post delivered to your Inbox.

Join 263 other followers