Evaluating New Ideas: Looking Across and Looking Beyond

17 August 2014 at 2:16 am 4 comments

| Nicolai Foss |

Those of us who have experience with research councils and other funding bodies with expert evaluations of the submitted research are familiar wilth folklore, such as “When evaluating economists routinely smash non-economics projects,” “sociologists are a total incrowd and will not tolerate any application of rational choice method, serious econometrics or common sense,” etc. Of course, this is part of the various conspiracy theories about how, notably, economists seek to establish intellectual hegemony.

However, the folklore may be wrong. In a new paper, “Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance and Resource Allocation in Science,” Kevin Boudreau, Eva Guinan, Karim Lakhani and Christoph Riedl look at the grant proposal process at a major research university and show that evaluators tend to treat proposals more harshly the closer they were to their own areas of expertise. However, evaluators also treat highly novel proposals negatively. Taking issues of ecological and external validity into account, there are obvious implications for the understanding of the nature of the exploitation/exploration tradeoff: There may indeed be a bias against exploring in the domains of highly novel ideas (as predicted by the literature), but the harsh evaluation of new, but well understood ideas may mean that there is a domain of relatively novel and less well understood ideas within which firms will explore. 

Entry filed under: - Foss -, Theory of the Firm.

Making Money from Behavioral Social Science? History Lesson: 200 Years of Management, uhm, Thought

4 Comments Add your own

  • 1. LR  |  17 August 2014 at 2:44 am

    “… evaluators tend to treat proposals more harshly the closer they were to their own areas of expertise.”

    Isn’t this the stereotype for nearly every academic conference in economics? Not a surprising finding at all.

  • 2. Nicolai Foss  |  17 August 2014 at 8:26 am

    You are wrong: it may very well be “surprising” if e.g. you are a soc or polsci. I have listened to lots of complaining colleagues who had proposals rejected, and I can assure you that they would surely be very surprised to learn that, e.g., econs are even harsher towards reps of their own discipline. (I think I had indicated in the blog that this was the context).

  • 3. Randy  |  18 August 2014 at 12:01 pm

    Interesting paper; thanks, Nicolai. I interpret their data and analysis to suggest that within a discipline, closeness would be even more compromising. So if one proposed a project on entrepreneurship, the e-wonks would shoot it down with more zeal than other types of economists or management scholars. There must be a nonlinearity, though. We see the results in journal publications that in-group reviewers support their group members in tightly defined “schools” (I think back to the heyday of pop ecology) when closely aligned out-group members might reject.

  • 4. Kevin Boudreau  |  17 August 2016 at 8:51 pm

    Randy – the patterns are consistent with a cognitive (bounded rationality) effect – distinct from a strategic or socialized in group effect (these later effects remain popular go-to conspiracy theories in this literature, but I’ve yet to see proof positive evidence). The effect in this paper is consistent with close evaluators “seeing” more informational cues and therefore also seeing more of what is imperfect with a proposal. This gives more informed, but lower mean evaluation. This was one of my favorite papers to theorize and test in empirics, as it forced a look beyond usual economic and sociological explanations to explain the patterns in the data.

    Nicolai – Thanks for posting and commenting. I much appreciate this.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


Nicolai J. Foss | home | posts
Peter G. Klein | home | posts
Richard Langlois | home | posts
Lasse B. Lien | home | posts


Former Guests | posts


Recent Posts



Our Recent Books

Nicolai J. Foss and Peter G. Klein, Organizing Entrepreneurial Judgment: A New Approach to the Firm (Cambridge University Press, 2012).
Peter G. Klein and Micheal E. Sykuta, eds., The Elgar Companion to Transaction Cost Economics (Edward Elgar, 2010).
Peter G. Klein, The Capitalist and the Entrepreneur: Essays on Organizations and Markets (Mises Institute, 2010).
Richard N. Langlois, The Dynamics of Industrial Capitalism: Schumpeter, Chandler, and the New Economy (Routledge, 2007).
Nicolai J. Foss, Strategy, Economic Organization, and the Knowledge Economy: The Coordination of Firms and Resources (Oxford University Press, 2005).
Raghu Garud, Arun Kumaraswamy, and Richard N. Langlois, eds., Managing in the Modular Age: Architectures, Networks and Organizations (Blackwell, 2003).
Nicolai J. Foss and Peter G. Klein, eds., Entrepreneurship and the Firm: Austrian Perspectives on Economic Organization (Elgar, 2002).
Nicolai J. Foss and Volker Mahnke, eds., Competence, Governance, and Entrepreneurship: Advances in Economic Strategy Research (Oxford, 2000).
Nicolai J. Foss and Paul L. Robertson, eds., Resources, Technology, and Strategy: Explorations in the Resource-based Perspective (Routledge, 2000).

%d bloggers like this: