Economists, (Hard) Data, and (Soft) Data

17 March 2012 at 10:51 am Nicolai Foss 6 comments

| Nicolai Foss |

Economists have typically been suspicious of data generated by (mail, telephone) surveys and interviews, and have idolized register data. The former are soft and mushy data, the latter are hard and serious ones. I have always been a bit sceptical regarding whether the traditional economist’s suspicion of soft data is really that well-founded; after all, the statistical agencies of the world and other government institutions that are in the business of data collection are populated by fallible individuals and respondents are the same ones that respond to, say, a mail survey conducted by Prof. N. J. Foss, PhD. (Having recently conducted a major data collection effort with a public statistical agency, my skepticism has dramatically increased!)

The argument is sometimes made that there may be a legal duty to respond to the queries of a government agency and this means a high response rate and accurate reporting. However, it appears that we know rather little about the accuracy of data generated in this way, and it is quite conceivable that measurement error is high, exactly because the provision of data is “forced” (those anarcho-capitalist types out there may delight in providing errorneous data!). The serious content of the traditional economist’s prejudice is rather, I think, that surveys often have respondents reacting to subjective scales rather than providing absolute numbers. This is a warranted concern, but not a critique of surveys and interviews per se, because these methods do not imply commitment to subjective scales per se.

As a rule register data are not available that can be used to address numerous interesting issues in organizational economics, labor economics, productivity research and so on. Scholars working on these issues have to resort to those softy surveys and interviews that have been the workhorses of business school faculty for decades. This is a new recognition in economics. Case in point: A recent paper by Nicholas Bloom and John Van Reenen, “New approaches to surveying organizations.” There is absolutely nothing, I submit, in this short, well-written paper that would surprise virtually any empirically oriented business school professor (i.e., virtually all bschool professors) to whom this would not be anything “new” at all, but rather old hat.

This is not a critique of Profs. Bloom and Van Reenen at all (on the contrary, it is excellent that they educate their economist colleagues in this way). It is just striking and a little bit amusing, however, that we have had to wait until 2010 until empirical approaches that have been mainstream in management research for decades reach the pages of the American Economic Review.

Entry filed under: - Foss -, Methods/Methodology/Theory of Science, Myths and Realities.

Lazear and Spletzer on Creative Destruction Capabilities and Organizational Economics Once More

6 Comments Add your own

1. Nick Bloom | 17 March 2012 at 10:27 pm

Very stimulating and interesting post – thanks for writing this.

The broad principles and ideas of this I totally agree with, and yes economists are strangely two-sided on using survey data in Census form but not directly. I have often had the same thought but never articulated it or thought so clearly.

The one point I’m not sure I agree is that there is nothing new in our methodology. In particular I could not find any prior papers in any field that: (a) used double-blind surveys with open questions (b) that are randomly drawn from the population (rather than a non-random sample of convenience), and (c) matches this data up to externally collected performance indicators (rather than using surveys to measure both management and performance).

Such a paper may exist, but in all our years of working on this and presenting to economics and strategy seminars, and publishing work in places like the SMJ and AMP we have never come across this, and neither has any referee or editor suggested such a paper. So individually these three steps are not new, but somehow no prior paper seems to have combined these together.

Also I should point out the Census is hopefully about 6 months away from releasing a 50,000 plant survey on management practices across the US (this will of course be anonymous, but still great data).

Thanks,

Nick
2. Nicolai Foss | 18 March 2012 at 2:55 am

Nick: I think you are correct in saying that “these three steps are not new, but somehow no prior paper seems to have combined these together.” Therefore, you are right that I underestimated the novelty of what you say in the AER piece. For the record, I am a huge fan of your work, so, again, the blog is a critique of econ practice, not of your paper.
3. economists and survey skepticism « orgtheory.net | 19 March 2012 at 7:03 pm

[…] at Evil Twin, Nicolai Foss gently chides Bloom and Van Reenen for publishing a paper in the AER proceedings called “New Approaches to […]
4. Bo | 2 April 2012 at 12:46 am

hmmm, I submit that many papers (at least claim) to be doing b) and c) as mentioned above (though (a) I am not so sure about). To be sure, many (most?) management papers may be somewhat lazy in applying these rigorous data collection methodologies, however, reviewers of top (management) journals seem to be getting better at pointing out these limitations and weaknesses.

Related, however, is the question of using different (and correct) data analysis methodologies given a particular type of data (and/or research question). For instance, while survey data by NJ Foss may be more prone to measurement error, such mesurement error can be treated as such and dealth with (at least to a certain extend) via various analytical methods, including SEM etc. Yet, to assume that Census (hard) data is without such measurement error is, of course, errornous..It is not only the respondents that introduce bias; the person(s) who codes data often inadvertibly also introduces bias and measurement error of various kinds..

I am currently involved in a rather large government sponsored data collection effort in Australia where we are dealing with both these issues to various degrees…I also remember working with secondary (“hard”) data gathered by OECD and the World Bank a few years ago and wondering about comparability across countries (and years). I called up an expert from OECD who basically told me “do not assume you can compare any number across a large number of countries as definitions, coding etc differs somewhat; do not ever attempt to compare numbers across years as definitions, coding etc differs substatially”….
5. Peter Klein | 2 April 2012 at 9:22 am

Ironically, using secondary data collected by a government agency or other well-known provider often, in practice, shields the researcher from these kinds of criticisms. To give only one example, seasoned researchers on M&As, IPOs, etc. know that SDC Platinum (now part of Thompson) is riddled with errors, but everybody uses it because, well, everybody uses it. The better studies cross-check dates and other key variables against other sources, but many don’t. — and still get published.
6. Bo | 3 April 2012 at 5:18 pm

I suppose the question is: is poor data better than no data? In social science where there are no absolutes and we approximate reality at best via socially constructed “latent” constructs it may be ok as it may show, at least to some extent, central tendencies…

The great thing about SDC etc is that, while fraught with mistakes and errors, it at least creates a possibility of re-testing and verification of results (should anyone be so inclined); something that is otherwise sorely missed in social science research with all our idiosyncratic samples…