Controversy Over JPE Paper on File Sharing
23 June 2008 at 12:26 pm Peter G. Klein 7 comments
| Peter Klein |
Stan Liebowitz, no stranger to controversy (1, 2), maintains that the Oberholzer-Gee and Strumpf paper on file sharing, published last year in the Journal of Political Economy, is fundamentally flawed. Stan submitted a comment (longer version here) to the JPE which was rejected by editor Steve Levitt. Stan believes that Oberholzer-Gee and Strumpf are guilty not merely of sloppiness, but academic dishonesty, and is upset that they refuse to share their data. The German newspaper Handelsblatt has written an article about the controversy. Handelsblatt focuses on Levitt’s decision to ask Strumpf to write a reply and then to use the reply as an anonymous referee report in rejecting Liebowitz’s comment. That doesn’t trouble me as much as the authors’ unwillingness to share the data (and the JPE’s refusal to insist on it). More generally, notes the newspaper:
The impression that procedural standards of economics journals are not particularly strict is widely shared in the profession. Zurich-based economist Ernst Fehr, an associate editor of the top-five journal “Quarterly Journal of Economics” and of “Science” points to a lack of clear rules as to when an editor should recuse himself because of potential prejudice. Science journals also seem to deal more openly with the competition among scientists. “Authors who submit an article to a science journal can say who they do not want to review their article”, praises Fehr, a choice which is typically not given to economists.
One internationally renowned economist, who did not want to be named, expresses the complaint more bluntly: “Little scandals and big scandals are commonplace: editors who publish articles in their own journals, referees or editors who decide about articles submitted by their own doctoral students.”
The pointer is from Craig Newmark, who writes: “Until important empirical results in economics are, as a matter of routine, carefully scrutinized and until they are provably replicable, economics will never get the respect that physics and biology and chemistry get. And that’s a shame.”
Update: Additional comments from John Lott and John Palmer.
Entry filed under: - Klein -, Institutions, Methods/Methodology/Theory of Science.
Organizations and Markets 
1.
David Hoopes | 23 June 2008 at 10:05 pm
So how controversial is the querty paper? Or, is there any reason to suspect that Leibowitz’ story in that paper is untrue? It seems the controversy is why people still rely on the evidence published in the original querty work that is thin at best. Are there known problems with Leibowitz’ querty observations?
2.
Brian Pitt | 24 June 2008 at 7:30 am
This merely scratches the surface of the infamous practice of editors accepting results on faith. (of course if the editors reran the models, the peer review process would take three times as long.)
3.
Peter Klein | 24 June 2008 at 8:30 am
David, just to be clear, it isn’t Stan’s analysis that’s in question here, but the analysis in the original paper by Oberholzer-Gee and Strumpf. As to the QWERTY story, my sense is that the QWERTY-as-market-failure crowd, after initially ignoring the Liebowitz-Margolis critique altogether, shifted to the position that Stan’s analysis is basically correct but irrelevant, because we have lots of additional evidence for market failure caused by network effects and path dependence. Personally, I find the empirical evidence on the David-Arthur-Katz-Shapiro-Farrell-Saloner side of the argument extremely weak. Some have even questioned the professional ethics of that group.
4.
David Hoopes | 24 June 2008 at 1:52 pm
I had trouble getting authors to share data way back when on a paper I wrote examining measures of geographic diversification.
5.
Patrick R. Sullivan | 24 June 2008 at 4:53 pm
It would be a better world–academic and elsewhere–if there were more scholars like Stan Liebowitz, and fewer people refusing to listen to him.
6.
Tom | 24 June 2008 at 9:16 pm
So what is the big deal? Levitt and his co-authors don’t give out the data many times for their papers and Levitt doesn’t require others to give out their data either. So I don’t see any inconsistency here. It is not like Levitt is applying different standards to other people’s work. It just seems that this is standard practice in economics. Is there anyone who gives out their data regularly to others? If not, what is the big deal?
7.
Peter Klein | 24 June 2008 at 10:17 pm
Actually several journals, such as the American Economic Review, now require authors of empirical papers to make datasets and programs available as a condition of publication. Of course, in the vast majority of cases nobody cares anyway. However, if the subject is particularly controversial, and the analysis and results subject to reasonable doubt, it seems bad form to ignore the criticism altogether. (As I mentioned above, that’s exactly what Paul David and company did when their QWERTY story was challenged.)