AEA Drops Double-Blind Reviewing

26 May 2011 at 12:52 pm Peter G. Klein 16 comments

| Peter Klein |

An announcement from the American Economic Association:

On April 15, 2011, the Executive Committee voted to drop “double-blind” refereeing for the Association’s journals. The change to “single-blind” refereeing (the referees’ identity remains undisclosed) is effective July 1, 2011. Easy access to search engines increasingly limits the effectiveness of the double-blind process in maintaining author anonymity. Double-blind refereeing also increases administrative costs of the journals and makes it harder for referees to identify an author’s potential conflicts of interest arising, for example, from consulting.

Entry filed under: - Klein -, Methods/Methodology/Theory of Science.

Competitive Advantage, Network Advantage, and Vienna Fundamentalism

16 Comments Add your own

1. david | 26 May 2011 at 1:29 pm

But what about the ideological (or other) conflicts or biases of the referee? Won’t they be more likely to be a factor now that they will know the author’s name (and yet the author still won’t know theirs)? It seems that if one seeks a disciplined process, it ought to be double-blind or “no-blind” but not single-blind. Single-blind might be best if you are seeking orthodoxy though.
2. FC | 26 May 2011 at 1:58 pm

Why should a reviewer be expected “to identify an author’s potential conflicts of interest”?

The editor should ask and the author should disclose. Full stop.
3. srp | 26 May 2011 at 6:37 pm

The best part of single-blind is avoiding ridiculous contortions in the paper to “hide” your identity. If you’re building on your own previous work, for example, you shouldn’t have to pretend that there isn’t a coherent research program involved.
4. Per Bylund | 27 May 2011 at 6:13 am

Two things struck me when reading this. First, that now there is no limit to self-citations in order to hide who you are – so we’ll probably see many more articles where the author builds his/her case off his/her previous publications. And second, that we will probably see fewer articles critical of accepted theories and doctrine – especially from un- or lesser known researchers – since they can be more easily dismissed (with or without reason). Well-known researchers might not be affected, however, since the reviewers will be able to base their judgment on reputation and won’t necessarily have to see to the content.

A completely open (“seeing”?) process would be better. Or why not have “expert reviewing” where experts in the field get to help the author toward the finish product and get their final comments published alongside the article…? That should allow for more differing views and better debate, and would be an incentive to put time and effort into reviewing.
5. Warren Miller | 27 May 2011 at 7:59 pm

I agree w/Per – single-blind reviewing will result in fewer papers criticizing the party line. And economics has one of those, as everyone reading this with a Ph.D. in the field knows.

In his 1993 AMR paper, “Barriers to the Advance of Organizational Science: Paradigm Development as a Dependent Variable,” Pfeffer made the case for, in essence, doing to organizational science what the economists had already done to economics: make it a field closed to dissent by asserting that the paradigm was well-developed. From this dubious notion, I’m told, came the word ‘Pfefferdigm.’

I guess the lamestream economists–which, if Austrians are heterodox, must surely be homodox–can’t stand the possibility that someone, somewhere, may refuse to quaff the PCE Kool-Aid, or to genuflect to the mirage of the firm-as-production-function, or to froth over the mountains of Samuelsonian calculus that is connected not one whit to the way people and organizations function and make choices. It reminds me of H.L. Mencken’s definition of ‘Puritanism’: “the haunting fear that someone, somewhere, may be happy.” I guess ‘haunting’ haunted the AEA, and it got mad as hell and decided not to take it any more.

I’m disappointed that AEA has caved to the apparatchiks, but I’m not surprised. Any field with basic textbooks that purposefully exclude a Nobel laureate (look for Hayek’s name in one) because his ideas not only were different from theirs, .but also have proved theirs to be an intellectual hoax, doesn’t have much to recommend it.

Lack of intellectual diversity begets lack of intellectual diversity. Eventually there is only a Stalinesque mindset. Maybe that’s the lesson of the true Pfefferdigm.
6. Michael Marotta | 28 May 2011 at 8:54 am

The double blind submission was adopted for journals because it is the method of laboratory science. It is the best way we have to avoid fooling yourself with results you want to believe. More deeply, objective inquiry does not test a proposition with the data from which the hypothesis was developed. The ethos of science holds that the truth of a proposition is independent of the person who suggests it.

It is a sad admission that the double blind process is “too much work” for the AEA. The risk of confirmation bias is greatly increased.

The review process has known failures. Erroneous theories were accepted for publication and famous works were rejected because of the name of the author, or lack of it. For a nice write-up, see: “Rejecting Nobel class articles and resisting Nobel class discoveries.” by Juan Miguel Campanario, Departamento de Física, Universidad de Alcalá, Madrid, Spain. http://www2.uah.es/jmc/nobel/nobel.html (“Rejected by six journals.”) The failures do not invalidate the system. You can strike out two out of three trips to the plate, and still lead the league in batting.

But the failures do warn us against being too ready to agree with our expectations. Now, the AEA has embraced what Richard P. Feynman caled “cargo cult science:” going through the motions while missing the point.
7. Warren Miller | 28 May 2011 at 12:20 pm

@Michael: Thanks for bringing up Richard Feynman. He was a show-stopper, and he’d be yowling about this one. Anyone in the teaching business–professor or consultant–who’s not read his very funny Surely You’re Joking, Mr. Feynman! is really missing something.

I remember the experiment he did–live and on camera–in front of his colleagues on the panel investigating the Challenger disaster in 1986. He proved conclusively that Morton Thiokol and NASA had guaranteed the astronauts’ doom by insisting that the mission launch when the air temperature was too cold to keep the Thiokol “O” rings from contracting, with the attendant leakage of fuel. Too bad that capital punishment is off the table for executives in the private sector and in government agencies who kill people.
8. Joel West | 28 May 2011 at 5:22 pm

Although I don’t have hard data, it seems as though journals that have abandoned double-blind reviewing have also abandoned the ethos that it shouldn’t matter who the author is. Famous authors get a better chance of publication, and controversial (or unknown) authors get a worse chance. This encourages “coasting.”
9. bork | 29 May 2011 at 7:12 pm

Is there any data on the degree to the search-engine problem actually exists?

What proportion of submitted papers do you estimate can be found online (as a working paper, a conference paper, on REPEC, or listed on a CV)?

Even if you assume that all reviewers will search, that doesn’t mean there’s a problem.

Unless the problem is total (i.e., all reviewing is already single-blind), I can’t see the rationale for eliminating double-blinded review. Some experimental evidence for how 2x vs 1x blinded review impacts reviewing can be found in the AER:

http://www.jstor.org/pss/2006906
10. srp | 30 May 2011 at 8:00 pm

The venting here is way in excess of the offense, if any. AER isn’t publishing Austrian stuff very much under double-blind conditions–it is precisely off-paradigm papers where blinding is less important because the referee immediately knows (by reading the content) that he or she is dealing with an outsider. The impact of single-blind vs. double-blind, in the pre-Internet experiment by Blank cited by bork above, was almost entirely to improve everyone’s chances of acceptance and to help out members of higher-status departments a bit more.

If I’m running a journal and want to serve my readers–not those who want to publish, but the consumers of the product–then the identity of the author is definitely a factor in deciding whether to publish. It’s not the only factor, but you can damn well bet that readers are more interested in a claim by someone whose previous work shows great competence or originality than someone less renowned. That’s why people pay Tom Wolfe or Michael Lewis or Malcolm Gladwell a lot of money for articles–readers are more likely to read the magazine in which they appear.
11. Warren Miller | 30 May 2011 at 11:35 pm

I respectfully disagree, Steve. If you want to serve your readers, why don’t you give them interesting and insightful papers to read? Or is it a population game? Or is an editor intent on ingratiating her- or him-self to the High and the Mighty?

More important, how would Wolfe, Lewis, and Gladwell ever get a start, given that they’re all “off paradigm,” with a close-minded, our-way-or-the-highway outfit such as AEA?
12. Bo | 31 May 2011 at 3:14 am

I also take issue with the reference to “popular” magazines above. In my book, this is precisely the difference between magazines and scientific journals; the former is there to serve (entertain) the readers wheras the latter is there to contribute to the ungoing scientific discussion (hence the need for reviewers etc).

To state that just because a person has a history of writing good articles this ensures his/her future work is of same quality and relevance is absolutely crazy. I am sure you are all familiar with papers published by “big names” that makes you wonder how they ever got into the journals given their redundance, self-citations and complete lack of anything new – well now we know…

Related, I actually think that if we want to go to a single-blind review process – then perhaps the reviewers should be the ones revealed! This would ensure several things:

1) We would not see crazy reviewers assaulting the authors
2) We would not see reviewers attempting to promote their own ideas through the review process by pointing ensanely to own prior work
3) We would avoid reviewers that are not expect in the area having influence on what is published
4) It would ensure accountability on behalf of both reviewers and editors in the process
13. Jim Rose | 31 May 2011 at 4:56 am

Roger Garrison told a story about one referee recommending against the acceptance of his 1996 paper because the 1993 paper he was commenting on should not have been published either, in that referee’s view.

The 1993 paper that should not have been published, apparently, was one of Milton Friedman’s last major journal articles setting out his little known and very interesting plucking model of business cycle fluctuations.

The 1993 Friedman paper was part of a 80th birthday symposium in Friedman’s honour!
14. Warren Miller | 31 May 2011 at 9:23 am

Thank you, Jim. I never should have married my ex-wife, either. :-)
15. Peter Klein | 31 May 2011 at 11:16 am

I appreciate the concerns raised in the thread about conformity and bias. But, keep in mind that there is also competition among journals, and between journals and alternative outlets. See, for example, the discussion here: https://organizationsandmarkets.com/2010/09/29/the-peer-review-fetish/
16. srp | 1 June 2011 at 10:22 pm

1. Everyone ignored the point that blinding is of no avail for off-paradigm work. You can’t hide your lyin’ eyes.

2. How do new people get published without blind review? Amazingly, they impress editors and reviewers with the quality of their work. As Peter points out, journals compete for attention, and having interesting articles is a good way to do that. This works in the popular press, too–Gladwell, et al were not blind-reviewed at any point, I assure you.

3. Of course, eminent people produce crap on a regular basis. But with the exact same content, a more credible author is more interesting to a wider readership. This is no different than branding for any experience or credence good. It makes a difference on the margin.