Comments on: Sid Winter on Methodology

By: innovation system policy idea « Innovation Leadership Network

innovation system policy idea « Innovation Leadership Network — Sun, 28 Jun 2009 20:04:40 +0000

[...] Klein just blogged about his favourite quote at the DRUID Summer Conference, from Sid Winter. Oddly, one of my favourite quotes at the conference came from Peter, when he was [...]

By: Rafe

Rafe — Thu, 25 Jun 2009 04:16:39 +0000

Daniel Klein and Pedro Romero published a report on the contents of all the articles in a year of the J of Ec Theory.
http://www.econjournalwatch.org/pdf/KleinRomeroAbstractMay2007.pdf
They looked at 66 articles and asked three hard but fair quetions, starting with “Theory of what?” Almost half the papers did not relate to any identifiable theory.
Two thirds fell over at the second question “Why should we care?” (could the results make a difference?).
The third queation asked whether the results did make a difference.
Only 8 of the 66 got over the three hurdles with any credit.

By: Rafe

Rafe — Thu, 25 Jun 2009 00:27:34 +0000

The point is to be clear from the beginning as to whether a statistically significant result will make a difference to a debate that is worth travelling to Copenhagen, or the library, or your desk, to get involved in.

By: Peter Klein

Peter Klein — Wed, 24 Jun 2009 22:08:43 +0000

I don’t think Sid would disagree. I took him to mean that in practice, most people stop after step 2. They devote all their energy to finding statistically significant relationships (torturing the data as necessary), without thinking much about what it might mean. I think he was particularly critical of some of the conference presentations in this regard.

By: spostrel

spostrel — Wed, 24 Jun 2009 21:56:45 +0000

That statement could use a little context. I always thought statistical significance was an aid in tracking down cause and effect: 1) identify pattern or association. 2) Is it likely association is by pure chance? (stat sig). 3) If not likely to be pure chance, is there a set of identifiable causal linkages we can postulate or establish?

It’s not obvious to me how skipping step 2) improves the discovery process.