The Peer-Review Fetish
29 September 2010 at 11:24 am Peter G. Klein 5 comments
| Peter Klein |
I respect peer review as much as the next person and have done my share of publishing in peer-reviewed outlets. But I question the belief, expressed often in academic, media, and policy circles, that “not peer reviewed” means “worthless” and “peer reviewed” means “should be accepted without question.” (A corollary belief is that “funded by a private foundation or company” means “biased” while “funded by a government grant” means “neutral.”) In practice, the distinctions are not nearly so clean.
My thoughts on this were triggered by a revealing statement from Ronald Coase, quoted by Josh Gans and George Shepherd in their study of famous economics papers that were initially rejected, about his limited experience with peer review: “I have never found any difficulty in getting my articles published. I have either published in house journals (e.g. Economica) or the article was written as a result of a request (e.g. for a conference) and publication was assured.” Certainly no one would discount the importance Coase’s 1937 and 1960 papers because they weren’t rigorously peer reviewed. (Can you imagine the inane referee remarks that “The Problem of Social Cost” would have generated?) More generally, consider the Journal of Law and Economics during Coase’s editorship in the 1960s and 1970s — the high-water mark of the JLE‘s influence. Or, for that matter, Public Choice under Gordon Tullock, the JPE under George Stigler, or the Journal of Libertarian Studies under Murray Rothbard. These were edited somewhat unevenly, led by charismatic and strong-willed editors with idiosyncratic tastes, yet have been vastly influential in their respective fields.
Peer review serves a useful function and probably improves the quality of published output, on average. But let’s not make a fetish of it.
Entry filed under: - Klein -, Institutions, Methods/Methodology/Theory of Science, Myths and Realities.
Organizations and Markets 
1.
Abhi S | 29 September 2010 at 11:51 am
Someone may find this study interesting: http://www.springerlink.com/content/g1l56241734kq743/. Moral of the story: peer reviewers are vulnerable to biases. Go figure.
2.
srp | 29 September 2010 at 1:28 pm
There was a time when the readership of a journal was important. Peer review as a way of keeping things out that few would want to read made sense from the editor’s point of view as long as he could trust/monitor the reviewers. But a strong editorial hand could be just as effective. The peer reviewer was there to help make the journal interesting rather than as a gatekeeper to legitimacy.
Today, most potential citers of articles in economics have read them before they were officially published and journals serve a kind of certifying and archiving purpose. Few sit down with an issue of a journal and go through it as they once might have. So peer review has shifted to a role for which it is less suited–certifying articles as worthy of being in the official public record. (I ignore here the various abuses that we all know about–I’m focusing on how peer review works now when it is operating in what is considered its legitimate mode.)
3.
Thomas | 29 September 2010 at 1:45 pm
I don’t think very many people actually think ‘that “not peer reviewed” means “worthless” and “peer reviewed” means “should be accepted without question”.’ But I do think there is an argument for a view like the following:
If someone has published no peer-reviewed articles then that someone has not shown what he or she is worth to the academy, and if someone publishes 1 or 2 peer-reviewed articles every year then their worth, as academics, is beyond question. That doesn’t settle all the issues of merit, nor even of whether they should be employed at a university (and certainly not what university or in what sort of position). But it does help us to decide what someone is “worth” and what questions can be left on the side.
My view in a nutshell: Peer review is full of flaws. That one paper has passed peer-review and another has been rejected proves very little about their relative quality. But if a certain topic is never written about in peer-reviewed articles, or a certain researcher never publishes such articles, then we have a good (if still imperfect) indication that the topic or researcher are on the fringes of research.
4.
Bo | 3 October 2010 at 3:12 am
hmm…I wonder. Most “high-quality” peer-reviewed journals of today seem to be rather incremental and low-risk taking in their nature; quality seems to be equated with methodological rigor and then the unfortunate constant quest for “theoretical contribution”, which in essence is a way for reviewers and editors to reject any paper they do not like – to expect the 1000+ academic articles published every year to all make a theoretical contribution is simply silly.
Peer-reviewing serves an important role in that it somewhat secures a minimum of “academic standard” (whatever that is) and acts as a way to legitimize the contribution. In some cases it may also serve its role as a way to reduce the individual power, influence and thoughts of the editor, however, given that editors choose the reviewers and choose which reviewer(s) to pay most attention to….its role here is questionable….
5. Oltre la peer review? | 4 October 2010 at 12:32 pm
[…] peer review è importante ma non facciamone un feticcio: the-peer-review-fetish come si riconosce in questo blog tenuto da studiosi internazionali di economia […]