Nature Gives Up on Open-Source Peer Review

11 January 2007 at 12:30 pm 4 comments

| Peter Klein |

The open-source, wiki model does not, apparently, work well for scientific publishing. Nature had placed a selection of submitted manuscripts online and invited feedback from researchers around the world, promising to take the feedback into consideration as part of the formal review process. But the scientific community showed little interest. Few authors were willing to participate in the experiement, and the online papers didn’t get much feedback.

During Nature’s trial, only 5 percent of 1,369 papers ranging from astronomy to neuroscience that were selected for traditional peer review were also posted on the Internet for open commentary. Of those, 33 papers received no comments. The rest received a total of 92 technical comments.

The journal concluded that many researchers were either too busy or had no real incentive in evaluating their colleagues’ work publicly. In addition, none of the editors found the posted comments influenced their decision whether a paper gets published.

I’m a little surprised by this. According to Lerner and Tirole, the open-source model should work in settings with strong reputation effects. One would think that in small, close-knit, specialized scientific communities the incentives to provide useful feedback — assuming it’s not anonymous — would be fairly high. On the other hand, there are opportunities to do so at conferences, seminars, workshops, the faculty lounge — and even blogs! — and the opportunity costs of doing it via Nature’s setup may have been too high.

Entry filed under: - Klein -, Institutions.

Call for Papers: Putting Social Capital to Work How Does Management Affect Capabilities?

4 Comments Add your own

  • 1. Russell Duhon  |  11 January 2007 at 8:03 pm

    I think there are a few explanations:

    One, while there are strong reputation effects in academia, these comments have little impact on reputation.

    Two, they have even less impact on beneficial signals.The results of signalling are perhaps an even greater incentive for many contributors.

    Three, Lerner and Tirole missed a number of things — for instance, many open source projects receive most or substantial contributions from people paid to contribute (sometimes by third parties, sometimes by principals). Also, open source (like any software) contributions have a multiplier effect — your contributions make later contributions more valuable, an increase that can be counted as an incentive to contribute if the software is of value to you in some way (which it almost always is for contributors). These two effects not mentioned by Lerner and Tirole are also not present with the Nature commentary.

    Another, almost tangential thing is the delay in impact. Open source contributions on active projects typically ‘matter’ instantly or very quickly, depending on if one has commit access. The comments to nature were only used at some remove in evaluating the papers, and even then their effect was obscured (well, would have been if they had any effect).

    A further difference with open source projects is that open source projects typically have central contributors who act as catalysts and drivers for the community; people who have bought in and stimulate further contribution by example and enticement. In successful open source projects, these people are present from the beginning, typically among those who start the project. Nature seems to have just thrown the doors open without having socialites to draw out the shy folk.

    And a last observation is that lots of open source projects fail. Despite Nature’s stature, they might not (likely didn’t, even) have managed to find a combination of structure, publicity, people, and papers that would result in a successful ‘open source’ review process.

    So yeah, those’re why I’m not surprised, and think there’s a lot of potential left unexplored. I’m sad that others might be discouraged by Nature’s failure to stimulate interest.

  • 2. Mark  |  21 February 2007 at 1:56 pm

    I agree with Russell. The experimental design looks flawed.

    Motivating incentives for wiki participants are poorly understood, but one shouldn’t ignore the obvious. Academic reviews require hours of work. One doesn’t pick up a paper and review it for the shear joy of reading. No one would say a review is like a cross-word puzzle. To justify the energy, there must be a reward, the more immediate the better.

    The comment that no editor let the public influence their decision is telling. Why waste 3 or 4 hours on a review which is going to be ignored?

    A better experiment would be to wiki-fy the existing process.

  • 3. TMLutas  |  8 November 2007 at 9:46 pm

    There are lots of different ways to do open source. Some work, others do not. This plays out in the fight over licenses that cover various types of open source pojects. GPL v BSD v Apache v APSL v Microsoft Shared Source and on it goes. The fights over licenses are long and vicious.

    Nature picked a process that didn’t work right away. Boo hoo, so did the Mozilla foundation. Mozilla persevered, refining the process, and now we have Firefox. Nature, on the other hand, packed it in relatively early with a pout and some parting shots.

    Not impressive.

  • […] have been moves towards and away from open source peer review, some academics have been given tenure with their blogs considered […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


Nicolai J. Foss | home | posts
Peter G. Klein | home | posts
Richard Langlois | home | posts
Lasse B. Lien | home | posts


Former Guests | posts


Recent Posts



Our Recent Books

Nicolai J. Foss and Peter G. Klein, Organizing Entrepreneurial Judgment: A New Approach to the Firm (Cambridge University Press, 2012).
Peter G. Klein and Micheal E. Sykuta, eds., The Elgar Companion to Transaction Cost Economics (Edward Elgar, 2010).
Peter G. Klein, The Capitalist and the Entrepreneur: Essays on Organizations and Markets (Mises Institute, 2010).
Richard N. Langlois, The Dynamics of Industrial Capitalism: Schumpeter, Chandler, and the New Economy (Routledge, 2007).
Nicolai J. Foss, Strategy, Economic Organization, and the Knowledge Economy: The Coordination of Firms and Resources (Oxford University Press, 2005).
Raghu Garud, Arun Kumaraswamy, and Richard N. Langlois, eds., Managing in the Modular Age: Architectures, Networks and Organizations (Blackwell, 2003).
Nicolai J. Foss and Peter G. Klein, eds., Entrepreneurship and the Firm: Austrian Perspectives on Economic Organization (Elgar, 2002).
Nicolai J. Foss and Volker Mahnke, eds., Competence, Governance, and Entrepreneurship: Advances in Economic Strategy Research (Oxford, 2000).
Nicolai J. Foss and Paul L. Robertson, eds., Resources, Technology, and Strategy: Explorations in the Resource-based Perspective (Routledge, 2000).

%d bloggers like this: