Protesting Against and Sanctioning Bad Reviewers

5 October 2008 at 8:50 am 3 comments

| Nicolai Foss |

Keynes famously complained (whined) that Hayek, in his review of A Treatise of Money, hadn’t treated the book with the measure of relative goodwill that an author is entitled to expect from a reviewer. He also equally famously informed Hayek that he (Keynes) had changed his mind, so that Hayek’s two-part, article-length review was irrelevant anyway. Hayek later explained that this was the main reason why he had chosen not to review Keynes’ General Theory (the story is a bit more complex, see this paper).

While Keynes was no doubt whining — those who care can check Hayek’s very careful and balanced review — he does have a point: Aren’t authors entitled to a certain measure of goodwill in the sense that they can reasonably expect that the reviewer has tried to understand what the author is talking about, doesn’t misinterpret and misrepresent him too badly, doesn’t kill a paper because of very minor problems, etc. etc.? While I trust that most people would agree with this, we also know that we may occasionally get reviewers (whether anonymous or, in the case, of published reviews, usually non-anonymous) who are not at all inclined to show such goodwill.

In fact, the now increasingly talked about problem with anonymous bullying on internet fora etc. is really a staple of the journal review process and has probably been so for as long as that institution has existed. Some people just cannot resist the combination of the anonymity of the review process and editors who are not likely to impose sanctions on nasty and malicious behavior. Other reviewers demonstrably fail to understand the basic point of a paper and smash it based on a reading that is simply wrong.

A case in point: I recently got back three reviews of a paper invited for a special issue of a management journal (not a terribly high-ranking one) on Austrian econ. One reviewer loved the paper, another liked the basic idea but thought that much more development was needed, and the third reviewer went out of his way to kill the paper. According to this reviewer, “this submitted manuscript knocks down a straw man” and it deals with “a transparent flaw hardly worthy of great commentary.” The review ends by saying, “Concluding that we should look at Hayek (1937) rather than Hayek (1945) is a rather cryptic — and in this case, fitting — end to this manuscript.” Perhaps meant to be funny, but just nasty. In actuality, the paper dealt with positions that have been defended by leading management scholars (so much for the “straw man” and “transparent flaw”). And for somebody with a knowledge of Hayek’s work, the Hayek (1945) and Hayek (1937) references shouldn’t be “cryptic” at all. The reviewer simply hadn’t cared to try to understand what the paper was fundamentally about.

What can one do in such situations? The gut reaction usually is to blame authors: “They just didn’t wrote clearly enough,” “They failed to motivate and position the paper, define the RQ, explain the sampling frame, the methods, the results and the implications for PhD students in the first sentence” etc. etc. The instinct of most people is that a rejected authors simply has a bad case, per definition.

The gut reaction is, however, often wrong: While numerous terribly written papers are indeed submitted to the journals, so are lots of terrible review reports. I have heard (US) colleagues reviewing for the major journals boast that they never spend more than an hour (everything included!) on a paper. Of course, this can only lead to reviews that are superficial. Nastiness may sometimes be a cover for this.  

However, while journals, of course, sanction bad research (and even bad writing, well, sometimes), authors very seldom do anything to try to sanction bad reviews.  An author will, unless he is really big guy, have little bargaining power.  Therefore, most of us are inclined to do nothing, even when we receive demonstrably nonsensical reviews. In performing various editorial roles, I have sometimes received back reports from reviewers that were clearly of very low quality (in which case I have usually tried to solicit an additional opinion). I have never experienced an author protesting over a report that was sub-par in the quality dimension. On the contrary, I have experienced authors — hypocritically, I think — welcoming reviews that in actuality weren’t any good quality-wise. Why this behavior? Authors may think that they will only come across merely as whining.

However, if one really has a case, I think one is entitled to protest (by carefully explaining to the editor the folly of the reviewer). I have once successfully protested a rejection which was based on lousy reports (the paper was reviewed anew and finally accepted). I have helped colleagues successfully do the same thing (even with top journals). In other words, it is doable. For the sake of the review institution, we may even be obligated to protest bad reviews, in order to get those lemon reviewers sacked. And we all need to take our review responsibilities very, very seriously.

Entry filed under: - Foss -.

Call for Papers: Org Economics and Org Capabilities Amethyst and Public Choice

3 Comments Add your own

  • 1. James  |  5 October 2008 at 10:16 am

    This makes me think about error rates and the balance of false positives and false negatives. The author’s perspective is that a false positive still yields a publication (hooray!), but false negatives are costly and wounding, as described above.

    The editor’s take is the opposite: false positives are costly to the reputation of the journal, and false negatives, perhaps b/c they don’t have to live with the consequences of the error, have less obvious sorts of impacts.

    Add in reviewers seeking to be *efficient* and assumptions about the distribution of submissions (lots of good papers vying for a home) and you get the current mess. Also, conditioning people to complain might be burdensome, as at least some complaints will have no merit.

    In short, perhaps sanctioning reviewers only gets at part of the (systemic) current problem.

  • 2. Michael E. Marotta  |  6 October 2008 at 8:31 am

    The assumption is that science is dispassionate. We address ideas, facts and theories, not the people who assert them. Yet, therein lies a fallacy: the stolen concept. The word “assert” is strong. It is in and of our nature to be emotional. The request for goodwill is a request for a ppsitive emotional response — whether it is deserved or not.

    Surfing for sociology blogs, I came across one that derided the packaging of food products for reinforcing the white male chauvinist underpinnings of our exploitative commercial society. There is nothing surpirsing in that in a forum for sociologists. But does it exhibit goodwill? The writers did not offer their own, better, classless, sexless, genderless, ethnicitiless packaging. We are supposed to accept this as “critical sociology.” They only gave off bad will — and expected to get away with it.

    In accounting, “goodwill” is the difference between the market value of the outstanding common stock and the book value of a company’s assets. Goodwill must be earned.

  • 3. Elaka granskare « Nonicoclolasos  |  26 February 2009 at 5:47 am

    […] hans första bedömning, vilket kan missgynna en vid framtida kontakter. En som trots det rekommenderar att man klagar ibland är professor Nicolai […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


Authors

Nicolai J. Foss | home | posts
Peter G. Klein | home | posts
Richard Langlois | home | posts
Lasse B. Lien | home | posts

Guests

Former Guests | posts

Networking

Recent Posts

Categories

Feeds

Our Recent Books

Nicolai J. Foss and Peter G. Klein, Organizing Entrepreneurial Judgment: A New Approach to the Firm (Cambridge University Press, 2012).
Peter G. Klein and Micheal E. Sykuta, eds., The Elgar Companion to Transaction Cost Economics (Edward Elgar, 2010).
Peter G. Klein, The Capitalist and the Entrepreneur: Essays on Organizations and Markets (Mises Institute, 2010).
Richard N. Langlois, The Dynamics of Industrial Capitalism: Schumpeter, Chandler, and the New Economy (Routledge, 2007).
Nicolai J. Foss, Strategy, Economic Organization, and the Knowledge Economy: The Coordination of Firms and Resources (Oxford University Press, 2005).
Raghu Garud, Arun Kumaraswamy, and Richard N. Langlois, eds., Managing in the Modular Age: Architectures, Networks and Organizations (Blackwell, 2003).
Nicolai J. Foss and Peter G. Klein, eds., Entrepreneurship and the Firm: Austrian Perspectives on Economic Organization (Elgar, 2002).
Nicolai J. Foss and Volker Mahnke, eds., Competence, Governance, and Entrepreneurship: Advances in Economic Strategy Research (Oxford, 2000).
Nicolai J. Foss and Paul L. Robertson, eds., Resources, Technology, and Strategy: Explorations in the Resource-based Perspective (Routledge, 2000).