Testing for Bias in Peer Review

16 September 2008 at 2:31 pm Peter G. Klein Leave a comment

| Peter Klein |

In a working paper entitled “Can We Test for Bias in Scientific Peer Review?”, Andrew Oswald proposes a method of detecting whether journal editors (and the peer review process generally, I suppose) discriminate against certain kinds of authors. His approach, in a nutshell, is to look for discrepancies between the editor’s comparison of two papers and how those papers were ultimately compared by the scholarly community (based on citations). In tests he runs on two high-ranking American economics journals, he doesn’t find a bias by QJE editors against authors from England or Europe (or in favor of Harvard authors), but he does find that JPE editors appear to discriminate against their Chicago colleagues.

That’s Andy Eggers, writing in the Social Science Statistics Blog. As Andy points out, it’s not completely clear what (raw) citation counts, and hence the experiment itself, are measuring. Also, Oswald uses within-journal paper order as a signal of the editor’s assessment of quality. Still, the technique is interesting, particularly if being the “lead paper” of a top journal generates additional citations, independent of paper quality.

(From the You Can’t Win department: I once had a colleague who had published two or three papers in the JPE, but these papers weren’t highly cited, which the department counted as a strike against him, on the assumption that every JPE paper should get at least a few cites merely be appearing in the JPE.)

NB: An older, unpublished paper by Smart and Waldfogel uses the same technique.