Do Social Scientists Misuse the Term “Natural Experiment”?

5 February 2010 at 9:18 am Leave a comment

| Peter Klein |

Richard Nielsen thinks so:

I’m on board with using the language of experiments, but I’ve also seen more than a few recent papers framed as “natural experiments” that are really just observational studies with no particular claim to special status. The spread of experimental language into observational studies may have downsides as well as benefits.

Until recently, I basically assumed that when people said they had a natural experiment, what they really meant was that they had a credible instrument: a variable that breaks the link between treatment assignment and the potential outcomes for some or all of the units. However, the lead [Political Analysis] article places difference-in-differences, regression discontinuity, and matching methods under the tent of natural experiments. While I like (and use) these techniques and find them compelling, only some of them explicitly rely on an IV-type argument. Maybe I have more to learn.

The problem with any randomization that isn’t controlled by the researcher is that extreme skeptics like me can then try to spin complicated stories about how confounding could occur.

Nielson is talking about political-science research, but economists and management scholars also use  the term “natural experiment” more loosely (e.g., to include difference-in-differences models). But Nielson (if I understand him correctly) seems to be mixing the specific method of analyzing the natural experiment with the presence or absence of a credible instrument. In other words, he’s concerned that people are using the term “natural experiment” to mean “any situation in which variation is introduced by nature,” rather than “a situation in which I can tell a convincing story about identification.” I don’t think economists are guilty of using it in the former way. As Angrist and Krueger put it, “A common criticism of the natural experiments approach to instrumental variables is that it does not spell out fully the underlying theoretical relationships. . . . [But] there is usually a well-developed story or model motivating the choice of instruments.” And if this story is persuasive, then discontinuity analysis or differences-in-difference modeling should be fine. Right?

Entry filed under: - Klein -, Methods/Methodology/Theory of Science.

“I, Pencil,” Updated Missouri Economics Conference

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


Authors

Nicolai J. Foss | home | posts
Peter G. Klein | home | posts
Richard Langlois | home | posts
Lasse B. Lien | home | posts

Guests

Former Guests | posts

Networking

Recent Posts

Categories

Feeds

Our Recent Books

Nicolai J. Foss and Peter G. Klein, Organizing Entrepreneurial Judgment: A New Approach to the Firm (Cambridge University Press, 2012).
Peter G. Klein and Micheal E. Sykuta, eds., The Elgar Companion to Transaction Cost Economics (Edward Elgar, 2010).
Peter G. Klein, The Capitalist and the Entrepreneur: Essays on Organizations and Markets (Mises Institute, 2010).
Richard N. Langlois, The Dynamics of Industrial Capitalism: Schumpeter, Chandler, and the New Economy (Routledge, 2007).
Nicolai J. Foss, Strategy, Economic Organization, and the Knowledge Economy: The Coordination of Firms and Resources (Oxford University Press, 2005).
Raghu Garud, Arun Kumaraswamy, and Richard N. Langlois, eds., Managing in the Modular Age: Architectures, Networks and Organizations (Blackwell, 2003).
Nicolai J. Foss and Peter G. Klein, eds., Entrepreneurship and the Firm: Austrian Perspectives on Economic Organization (Elgar, 2002).
Nicolai J. Foss and Volker Mahnke, eds., Competence, Governance, and Entrepreneurship: Advances in Economic Strategy Research (Oxford, 2000).
Nicolai J. Foss and Paul L. Robertson, eds., Resources, Technology, and Strategy: Explorations in the Resource-based Perspective (Routledge, 2000).