Do Social Scientists Misuse the Term “Natural Experiment”?

5 February 2010 at 9:18 am Peter G. Klein Leave a comment

| Peter Klein |

I’m on board with using the language of experiments, but I’ve also seen more than a few recent papers framed as “natural experiments” that are really just observational studies with no particular claim to special status. The spread of experimental language into observational studies may have downsides as well as benefits.

Until recently, I basically assumed that when people said they had a natural experiment, what they really meant was that they had a credible instrument: a variable that breaks the link between treatment assignment and the potential outcomes for some or all of the units. However, the lead [Political Analysis] article places difference-in-differences, regression discontinuity, and matching methods under the tent of natural experiments. While I like (and use) these techniques and find them compelling, only some of them explicitly rely on an IV-type argument. Maybe I have more to learn.

The problem with any randomization that isn’t controlled by the researcher is that extreme skeptics like me can then try to spin complicated stories about how confounding could occur.

Nielson is talking about political-science research, but economists and management scholars also use the term “natural experiment” more loosely (e.g., to include difference-in-differences models). But Nielson (if I understand him correctly) seems to be mixing the specific method of analyzing the natural experiment with the presence or absence of a credible instrument. In other words, he’s concerned that people are using the term “natural experiment” to mean “any situation in which variation is introduced by nature,” rather than “a situation in which I can tell a convincing story about identification.” I don’t think economists are guilty of using it in the former way. As Angrist and Krueger put it, “A common criticism of the natural experiments approach to instrumental variables is that it does not spell out fully the underlying theoretical relationships. . . . [But] there is usually a well-developed story or model motivating the choice of instruments.” And if this story is persuasive, then discontinuity analysis or differences-in-difference modeling should be fine. Right?