Are We Quacks?
| Lasse Lien |
Rich Bettis makes an important point in a forthcoming issue of SMJ. Bettis points out how two unfortunate practices interact with each other to create a very serious and fundamental problem for knowledge accumulation in (strategic) management.
One is the widespread practice of running numerous regressions on a given dataset and subsequently adapting (or in milder cases “tuning”) hypotheses or theory to fit the data. By itself this practice is quite unfortunate, since data patterns can and will occur by chance, and the more regression models one tries the more likely that one will “find” something. We obviously do not want such random patterns to influence either theory building or our catalog of empirical findings. However, this problem would be a great deal less serious if replication studies were common and we gladly published non-findings. Random correlations in the data would not survive replication tests, and would be eliminated fairly quickly.
As we all know, in management, replication studies cannot get published and are basically just not done. To make matters worse, we don’t publish non-findings either. This is the second unfortunate practice. Taken together these two practices may in the worst case indicate that much of what we think we know in management are just random data patterns, discovered through data mining, and protected by our lack of replication studies and refusal to publish non-findings. This is a sobering thought. As Bettis points out, we should all be very thankful that replication studies are more common in medical research than in management.
What is the solution? Well, a first step might be to launch the Journal of Managerial Replication Studies and give it the prestige it deserves. Either SMS or AOM should see the launch of such a journal as a crucial responsibility. I mean, we really don’t want to be quacks, do we?
HT: Helge Thorbjørnsen