The cornerstone of scientific method, as you pointed out eloquently, is just peer-review that is based on the quality of ideas presented in article/book.

The over-communicated world is obviously a threat to this as we have not yet completely used to it. not every one does what you did.

I tend to stay optimistic and think that as we move forward scientists will learn that just reviews will benefit them in the long run. (If a journal only publishes papers based on relationships, The accuracy of the papers will eventually fall. This may not be evident immediately but sooner or later it will lose it’s place to a newer player in the market)

If not, we need to think of a new scientific method.

The Dean at my former school saw “shopping” the paper around to likely reviewers as 50% of the work in getting a paper publsihed in a top tier journal.

I know of at least one sub-field in management that takes an invited 15 senior scholars on a summer retreat to read and improve each others’ papers. Of course, they are also likely to review those papers at top journals.

Other kinds of casual empiricism?

If top scholars gravitate to top schools and are then routinely invited to review the work of others at top schools is anything lost - doesn’t the cream float to the top? I think the answer is something is lost. It is very difficult for heterdox ideas to penetrate this system. New ideas have to be legitimated by the top schools first then flow down. This delays or hinders the dissemination of good ideas.

Has it become worse?

I would say yes - as the pressure to publish XXX papers in top tier journals for tenure increases. The signal to noise ratio is also very high meaning only top tier articles tend to get read/cited.

Do you routinely Google the paper title when you are asked to review?

No. But I have done so occasionally.

What can we do to hinder violations of the principle of anonymity?

I think good journals tend to assign papers to good reviewers (those with a track record of thoughtful comments) rather than simply the “expert” in a particular field.

In fact, knowing the authors’ other work is often very helpful in making sense out of the paper you’re trying to review. Anonymity is especially ridiculous when one paper is a follow-on to another as part of a research program.

The issue of friendship or acquaintance with the material need not affect this overmuch. Indeed one can be more critical of one’s friends, especially those whose work one respects, than one should be of a novice author more dependent on guidance and encouragement. To be familiar with the work merely eases this process along.

Perhaps the best reasons for refusing to review a paper would be (a) one cannot understand it enough to search for nuggets, (b) one understands it but cannot contribute to its improvement, or (c) one dislikes it to the point one cannot contribute to its improvement. Again I don’t see friendship or familiarity really entering into this.

Anon’s point about the selection of the reviewer may be more pertinent. An editor who looks favorably on the draft can obviously help translate that valuation into a publication decision by passing it to a reviewer who will be supportive, or vice versa. One might say this interferes with the process - but what else is the editor’s role and process as she/he tries to establish and implement the journal’s editorial policy? The ‘danger’ is that the editor’s choice is based not on the paper’s merits, as measured against that policy, but on some other grounds.

The implication is that editors have significantly more influence on the review and publication process than reviewers, but often, through the press of work, make their decisions without as careful an examination as they might. But I am sure all of us can cite many contrary examples wherein editors have done a tremendous amount of work, drawing on the reviewers’ contributions and adding substantially to them in their efforts to shape a promising paper, or even to explore every possibility before finally rejecting it.

All of which goes to show the review and publication process is as complicated and important a part of our disciplinary activity as is the researching and writing part.

1) As discussed above, given specialties most folks know full well who the author of a paper is, and there’s a big friendship/niche-effect. Now, I actually don’t think that this necessarily is as huge a problem if the clique truly is producing good work — in many cases these cliques are doing exactly that. So, you have strong scholars reviewing the work of other strong scholars and there’s a level of competence there where I think the end outcome is not hurt (it would be hurt if the paper went to ignorant reviewers outside the clique), but rather improved because of these cliques. Hmm, controversial perhaps.

2) A bigger problem I think is the assignment of reviewers. That is, editors have full control over whether a paper gets accepted or not by who they assign as the reviewers. Thus, reviewer assignment is scarcely a random draw — so, if you send in a social construction-type (or, econ for that matter) of paper and the editor is perhaps antagonistic to that type of work, then the “draw” of reviewers will reflect that and the paper gets rejected. Now, thats perhaps a bit of a jaundiced view of things but I think nonetheless a reality.

3) Now, the above said — perhaps blind review is over-rated. It depends on who reviews, and the quality of reviews, right? Ideally I’d like to trust the judgment and capacity and wisdom of the editor to make tough calls even if, say, all the blind reviewers hated a paper. If the paper’s good, then the editor ought to take the perogative of publishing it anyway and perhaps via their own comments molding the paper through the R&R process.

Rather scattered thoughts…apologies.