Evaluating New Ideas: Looking Across and Looking Beyond
17 August 2014 at 2:16 am Nicolai Foss 4 comments
| Nicolai Foss |
Those of us who have experience with research councils and other funding bodies with expert evaluations of the submitted research are familiar wilth folklore, such as “When evaluating economists routinely smash non-economics projects,” “sociologists are a total incrowd and will not tolerate any application of rational choice method, serious econometrics or common sense,” etc. Of course, this is part of the various conspiracy theories about how, notably, economists seek to establish intellectual hegemony.
However, the folklore may be wrong. In a new paper, “Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance and Resource Allocation in Science,” Kevin Boudreau, Eva Guinan, Karim Lakhani and Christoph Riedl look at the grant proposal process at a major research university and show that evaluators tend to treat proposals more harshly the closer they were to their own areas of expertise. However, evaluators also treat highly novel proposals negatively. Taking issues of ecological and external validity into account, there are obvious implications for the understanding of the nature of the exploitation/exploration tradeoff: There may indeed be a bias against exploring in the domains of highly novel ideas (as predicted by the literature), but the harsh evaluation of new, but well understood ideas may mean that there is a domain of relatively novel and less well understood ideas within which firms will explore.
Entry filed under: - Foss -, Theory of the Firm.
Organizations and Markets 
1.
LR | 17 August 2014 at 2:44 am
“… evaluators tend to treat proposals more harshly the closer they were to their own areas of expertise.”
Isn’t this the stereotype for nearly every academic conference in economics? Not a surprising finding at all.
2.
Nicolai Foss | 17 August 2014 at 8:26 am
You are wrong: it may very well be “surprising” if e.g. you are a soc or polsci. I have listened to lots of complaining colleagues who had proposals rejected, and I can assure you that they would surely be very surprised to learn that, e.g., econs are even harsher towards reps of their own discipline. (I think I had indicated in the blog that this was the context).
3.
Randy | 18 August 2014 at 12:01 pm
Interesting paper; thanks, Nicolai. I interpret their data and analysis to suggest that within a discipline, closeness would be even more compromising. So if one proposed a project on entrepreneurship, the e-wonks would shoot it down with more zeal than other types of economists or management scholars. There must be a nonlinearity, though. We see the results in journal publications that in-group reviewers support their group members in tightly defined “schools” (I think back to the heyday of pop ecology) when closely aligned out-group members might reject.
4.
Kevin Boudreau | 17 August 2016 at 8:51 pm
Randy – the patterns are consistent with a cognitive (bounded rationality) effect – distinct from a strategic or socialized in group effect (these later effects remain popular go-to conspiracy theories in this literature, but I’ve yet to see proof positive evidence). The effect in this paper is consistent with close evaluators “seeing” more informational cues and therefore also seeing more of what is imperfect with a proposal. This gives more informed, but lower mean evaluation. This was one of my favorite papers to theorize and test in empirics, as it forced a look beyond usual economic and sociological explanations to explain the patterns in the data.
Nicolai – Thanks for posting and commenting. I much appreciate this.
K