Reviewing Your Friends

13 March 2008 at 3:51 pm Nicolai Foss 11 comments

| Nicolai Foss |

Sometime ago I received a request from the Academy of Management Review to review a paper that I immediately recognized, having read it in an earlier version. I informed the editor that I couldn’t do the review because I knew the identity of the authors. About a month later I met one of the authors. I told her that I had been asked to review one of her papers, but had declined, to which she replied: “You should have done the review! We would’ve liked to have that paper in the AMR.” I confess to being somewhat baffled by all the implicit assumptions in this reply (i.e., the paper was actually of good quality, I would have recommended it, etc.), but also by its flagrant disrespect for the principle or ideal of anonymity in reviewing.

This happened a couple of years ago, and made me alert to similar violations of the principle of anonymity. I have talked informally to many people about it, and I have had to update my priors concerning such violations considerably. My admittedly casual empiricism suggests that in a large number of cases, reviewers at the top journals have a pretty good idea about the identity of the authors. The papers are likely to have been presented at a conference where the reviewer was also present. The reviewer may accept to review the paper because he liked what he heard at the conference and wants the ideas to be promoted. Afterwards, I have learned that I myself have been a beneficiary of such behavior.

Such collusive behavior may be predicted to increase as its expected value increases. Increasing publication pressure has raised this value. Even in the large Academy divisions (e.g., the BPS), it is likely that the same people will repeatedly be asked to review each others’ papers. Given that they can easily infer each others’ identities, reputation effects emerge to sustain collusion. I have heard rumours (not about management scholars, though) that a closed network of top dogs will routinely circulate their new working papers, essentially saying, “Here is a new paper of mine. You are likely to become a reviewer. Treat it well.” Some questions to the O&M readership:

Do you have any juicy anecdotes to illustrate the above?
Other kinds of casual empiricism?
Has it become worse?
Do you routinely Google the paper title when you are asked to review?
What can we do to hinder violations of the principle of anonymity?

Entry filed under: - Foss -.

Biblical Wisdom for Academics Big Think in Management Research

11 Comments Add your own

1. brayden | 13 March 2008 at 10:46 pm

The better fit you are as a reviewer for a paper is positively correlated with the likelihood that you are already aware of the paper’s existence. It makes sense. We all work in tight academic niches. You know almost everyone else working in your niche. You attend the same conferences, share drafts of papers, and have similar social networks with the people who are writing papers you would likely review. With all of that niche overlap, there is a high probability that you’d be familiar with a lot of the papers that you are requested to review.
2. Nicolai Foss | 14 March 2008 at 7:21 am

Brayden, That is of course entirely correct and pretty much the starting point of what I wrote — but what are, in your view, the implications of this vis-a-vis the ethical norms concerning anonymity?
3. brayden | 14 March 2008 at 9:21 am

I think the reviewer has to ask himself or herself whether the review could be biased because of friendship, loyalty, etc. There’s a big difference in the way you approach papers that you have read or heard about before and papers that were written by close associates or friends. The latter would introduce real ethical problems as you might be favorably biased towards the paper. In those cases you might want to decline to review.
4. anon | 14 March 2008 at 12:15 pm

I think true objective, blind review is an extreme rarity.

1) As discussed above, given specialties most folks know full well who the author of a paper is, and there’s a big friendship/niche-effect. Now, I actually don’t think that this necessarily is as huge a problem if the clique truly is producing good work — in many cases these cliques are doing exactly that. So, you have strong scholars reviewing the work of other strong scholars and there’s a level of competence there where I think the end outcome is not hurt (it would be hurt if the paper went to ignorant reviewers outside the clique), but rather improved because of these cliques. Hmm, controversial perhaps.

2) A bigger problem I think is the assignment of reviewers. That is, editors have full control over whether a paper gets accepted or not by who they assign as the reviewers. Thus, reviewer assignment is scarcely a random draw — so, if you send in a social construction-type (or, econ for that matter) of paper and the editor is perhaps antagonistic to that type of work, then the “draw” of reviewers will reflect that and the paper gets rejected. Now, thats perhaps a bit of a jaundiced view of things but I think nonetheless a reality.

3) Now, the above said — perhaps blind review is over-rated. It depends on who reviews, and the quality of reviews, right? Ideally I’d like to trust the judgment and capacity and wisdom of the editor to make tough calls even if, say, all the blind reviewers hated a paper. If the paper’s good, then the editor ought to take the perogative of publishing it anyway and perhaps via their own comments molding the paper through the R&R process.

Rather scattered thoughts…apologies.
5. JC Spender | 15 March 2008 at 5:13 am

Of course it all depends on what one thinks is the main function of the reviewer. To boil that down to merely recommending or rejecting may miss the point. I would argue the reviewer is the person who is asked to work hard to find the paper’s nuggets of ‘contribution’ and then help the author/s make them available to the reader. If there are none, then rejection seems appropriate. Seldom is acceptance recommended without some guidance about improving the paper.

The issue of friendship or acquaintance with the material need not affect this overmuch. Indeed one can be more critical of one’s friends, especially those whose work one respects, than one should be of a novice author more dependent on guidance and encouragement. To be familiar with the work merely eases this process along.

Perhaps the best reasons for refusing to review a paper would be (a) one cannot understand it enough to search for nuggets, (b) one understands it but cannot contribute to its improvement, or (c) one dislikes it to the point one cannot contribute to its improvement. Again I don’t see friendship or familiarity really entering into this.

Anon’s point about the selection of the reviewer may be more pertinent. An editor who looks favorably on the draft can obviously help translate that valuation into a publication decision by passing it to a reviewer who will be supportive, or vice versa. One might say this interferes with the process – but what else is the editor’s role and process as she/he tries to establish and implement the journal’s editorial policy? The ‘danger’ is that the editor’s choice is based not on the paper’s merits, as measured against that policy, but on some other grounds.

The implication is that editors have significantly more influence on the review and publication process than reviewers, but often, through the press of work, make their decisions without as careful an examination as they might. But I am sure all of us can cite many contrary examples wherein editors have done a tremendous amount of work, drawing on the reviewers’ contributions and adding substantially to them in their efforts to shape a promising paper, or even to explore every possibility before finally rejecting it.

All of which goes to show the review and publication process is as complicated and important a part of our disciplinary activity as is the researching and writing part.
6. srp | 15 March 2008 at 6:33 pm

I also think that the cult of anonymity is pretty pointless. I’ve seen no evidence that econ journals have lower quality than AMR and AMJ, even though the former (traditionally) were not anonymous.

In fact, knowing the authors’ other work is often very helpful in making sense out of the paper you’re trying to review. Anonymity is especially ridiculous when one paper is a follow-on to another as part of a research program.
7. kjh | 15 March 2008 at 8:08 pm

Amen to that!
8. stevphel | 15 March 2008 at 8:32 pm

Do you have any juicy anecdotes to illustrate the above?

The Dean at my former school saw “shopping” the paper around to likely reviewers as 50% of the work in getting a paper publsihed in a top tier journal.

I know of at least one sub-field in management that takes an invited 15 senior scholars on a summer retreat to read and improve each others’ papers. Of course, they are also likely to review those papers at top journals.

Other kinds of casual empiricism?

If top scholars gravitate to top schools and are then routinely invited to review the work of others at top schools is anything lost – doesn’t the cream float to the top? I think the answer is something is lost. It is very difficult for heterdox ideas to penetrate this system. New ideas have to be legitimated by the top schools first then flow down. This delays or hinders the dissemination of good ideas.

Has it become worse?

I would say yes – as the pressure to publish XXX papers in top tier journals for tenure increases. The signal to noise ratio is also very high meaning only top tier articles tend to get read/cited.

Do you routinely Google the paper title when you are asked to review?

No. But I have done so occasionally.

What can we do to hinder violations of the principle of anonymity?

I think good journals tend to assign papers to good reviewers (those with a track record of thoughtful comments) rather than simply the “expert” in a particular field.
9. Donald A. Coffin | 16 March 2008 at 2:12 pm

No anecdotes, no. But there is research showing that acceptance rates of articles written by junior/female/minority authors rise when refereeing is blind…just as we have substantial evidence that auditions for symphony orchestras are rated differently when the auditions are blind. Whatever else is going on, blind reviewing of research and performance have widened the pool of contributors.
10. Ali Shams | 18 March 2008 at 6:05 pm

I think we need to accept that scientific method has some flaws and that’s not a bad thing as in the physical world that we live in nothing is perfect.

The cornerstone of scientific method, as you pointed out eloquently, is just peer-review that is based on the quality of ideas presented in article/book.

The over-communicated world is obviously a threat to this as we have not yet completely used to it. not every one does what you did.

I tend to stay optimistic and think that as we move forward scientists will learn that just reviews will benefit them in the long run. (If a journal only publishes papers based on relationships, The accuracy of the papers will eventually fall. This may not be evident immediately but sooner or later it will lose it’s place to a newer player in the market)

If not, we need to think of a new scientific method.
11. David Hoopes | 18 March 2008 at 11:07 pm

I think most of the sub fields of strategy where tons of work is cranked out is generally a set of friends reviewing each other’s work. I won’t give examples.