| Nicolai Foss |
It seems to be rather generally accepted that the Gold Standard of empirically-based science is the randomized experiment. Mosts economists and management scholars subscribe to this view, although its critics include notables like James Heckman (here). Arguably, the greatest badge of honor that one can aspire to nowadays as an economist (let’s forget about management scholars here ;-)) is to publish an experimentally-based paper in Nature or Science. However, one thing is the method of randomized experiments per se; quite another thing is the actual design of such experiments in social science and psychology.
In a recent paper, “The Weirdest People in the World,” Joseph Henrich, Steven Heine, and Ara Norenzayan point out that most designs involve samples drawn entirely from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies, in practice often first-year students.
Of course, any serious experimental paper should be forthcoming about potential problems of external and ecological validity. The problem is certainly not neglected; in fact, some journals ban papers based on experiments involving students. However, the point of the Henrich et al. paper is to document how massive the problem really is in terms of the extremely widespread use of samples drawn from a total outlier population, namely WEIRD people and the sweeping conclusions drawn from experiments using WEIRD subjects. To establish this they compare to non-WEIRD samples. They end their paper by discussing what may be done in terms of practical research heuristics and research policy with respect to dealing with generalizability.