Nature picked a process that didn’t work right away. Boo hoo, so did the Mozilla foundation. Mozilla persevered, refining the process, and now we have Firefox. Nature, on the other hand, packed it in relatively early with a pout and some parting shots.

Not impressive.

Motivating incentives for wiki participants are poorly understood, but one shouldn’t ignore the obvious. Academic reviews require hours of work. One doesn’t pick up a paper and review it for the shear joy of reading. No one would say a review is like a cross-word puzzle. To justify the energy, there must be a reward, the more immediate the better.

The comment that no editor let the public influence their decision is telling. Why waste 3 or 4 hours on a review which is going to be ignored?

A better experiment would be to wiki-fy the existing process.

One, while there are strong reputation effects in academia, these comments have little impact on reputation.

Two, they have even less impact on beneficial signals.The results of signalling are perhaps an even greater incentive for many contributors.

Three, Lerner and Tirole missed a number of things — for instance, many open source projects receive most or substantial contributions from people paid to contribute (sometimes by third parties, sometimes by principals). Also, open source (like any software) contributions have a multiplier effect — your contributions make later contributions more valuable, an increase that can be counted as an incentive to contribute if the software is of value to you in some way (which it almost always is for contributors). These two effects not mentioned by Lerner and Tirole are also not present with the Nature commentary.

Another, almost tangential thing is the delay in impact. Open source contributions on active projects typically ‘matter’ instantly or very quickly, depending on if one has commit access. The comments to nature were only used at some remove in evaluating the papers, and even then their effect was obscured (well, would have been if they had any effect).

A further difference with open source projects is that open source projects typically have central contributors who act as catalysts and drivers for the community; people who have bought in and stimulate further contribution by example and enticement. In successful open source projects, these people are present from the beginning, typically among those who start the project. Nature seems to have just thrown the doors open without having socialites to draw out the shy folk.

And a last observation is that lots of open source projects fail. Despite Nature’s stature, they might not (likely didn’t, even) have managed to find a combination of structure, publicity, people, and papers that would result in a successful ‘open source’ review process.

So yeah, those’re why I’m not surprised, and think there’s a lot of potential left unexplored. I’m sad that others might be discouraged by Nature’s failure to stimulate interest.