Testing for Bias in Peer Review

16 September 2008 at 2:31 pm Leave a comment

| Peter Klein |

In a working paper entitled “Can We Test for Bias in Scientific Peer Review?”, Andrew Oswald proposes a method of detecting whether journal editors (and the peer review process generally, I suppose) discriminate against certain kinds of authors. His approach, in a nutshell, is to look for discrepancies between the editor’s comparison of two papers and how those papers were ultimately compared by the scholarly community (based on citations). In tests he runs on two high-ranking American economics journals, he doesn’t find a bias by QJE editors against authors from England or Europe (or in favor of Harvard authors), but he does find that JPE editors appear to discriminate against their Chicago colleagues.

That’s Andy Eggers, writing in the Social Science Statistics Blog. As Andy points out, it’s not completely clear what (raw) citation counts, and hence the experiment itself, are measuring. Also, Oswald uses within-journal paper order as a signal of the editor’s assessment of quality. Still, the technique is interesting, particularly if being the “lead paper” of a top journal generates additional citations, independent of paper quality.

(From the You Can’t Win department: I once had a colleague who had published two or three papers in the JPE, but these papers weren’t highly cited, which the department counted as a strike against him, on the assumption that every JPE paper should get at least a few cites merely be appearing in the JPE.)

NB: An older, unpublished paper by Smart and Waldfogel uses the same technique.

Entry filed under: - Klein -, Institutions, Methods/Methodology/Theory of Science.

Klein Bottle The Financial Crisis

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


Nicolai J. Foss | home | posts
Peter G. Klein | home | posts
Richard Langlois | home | posts
Lasse B. Lien | home | posts


Former Guests | posts


Recent Posts

Recent Comments



Our Recent Books

Nicolai J. Foss and Peter G. Klein, Organizing Entrepreneurial Judgment: A New Approach to the Firm (Cambridge University Press, 2012).
Peter G. Klein and Micheal E. Sykuta, eds., The Elgar Companion to Transaction Cost Economics (Edward Elgar, 2010).
Peter G. Klein, The Capitalist and the Entrepreneur: Essays on Organizations and Markets (Mises Institute, 2010).
Richard N. Langlois, The Dynamics of Industrial Capitalism: Schumpeter, Chandler, and the New Economy (Routledge, 2007).
Nicolai J. Foss, Strategy, Economic Organization, and the Knowledge Economy: The Coordination of Firms and Resources (Oxford University Press, 2005).
Raghu Garud, Arun Kumaraswamy, and Richard N. Langlois, eds., Managing in the Modular Age: Architectures, Networks and Organizations (Blackwell, 2003).
Nicolai J. Foss and Peter G. Klein, eds., Entrepreneurship and the Firm: Austrian Perspectives on Economic Organization (Elgar, 2002).
Nicolai J. Foss and Volker Mahnke, eds., Competence, Governance, and Entrepreneurship: Advances in Economic Strategy Research (Oxford, 2000).
Nicolai J. Foss and Paul L. Robertson, eds., Resources, Technology, and Strategy: Explorations in the Resource-based Perspective (Routledge, 2000).

%d bloggers like this: