Posts filed under ‘Methods/Methodology/Theory of Science’
| Peter Klein |
I sometimes worry that the blog format is being displaced by Facebook, Twitter, and similar platforms, but Patrick Dunleavy from the LSE Impact of Social Science Blog remains a fan of academic blogs, particularly focused group blogs like, ahem, O&M. Patrick argues that blogging (supported by academic tweeting) is “quick to do in real time”; “communicates bottom-line results and ‘take aways’ in clear language, yet with due regard to methods issues and quality of evidence”; helps “create multi-disciplinary understanding and the joining-up of previously siloed knowledge”; “creates a vastly enlarged foundation for the development of ‘bridging’ academics, with real inter-disciplinary competences”; and “can also support in a novel and stimulating way the traditional role of a university as an agent of ‘local integration’ across multiple disciplines.”
Patrick also usefully distinguishes between solo blogs, collaborative or group blogs (like O&M), and multi-author blogs (professionally edited and produced, purely academic). O&M is partly academic, partly personal, but we have largely the same objectives as those outlined in Patrick’s post.
See also our recent discussion of academics and social media.
| Peter Klein |
Should academic work be classified primarily by discipline, or by problem? Within disciplines, do we start with theory versus application, micro versus macro, historical versus contemporary, or something else? Of course, there may be no single “optimal” classification scheme, but how we think about organizing research in our field says something about how we view the nature, contributions, and problems in the field.
There’s a very interesting discussion of this subject in the History of Economics Playground blog, focusing on the evolution of the Journal of Economic Literature codes used by economists (parts 1, 2, and 3). I particularly liked Beatrice Cherrier’s analysis of the AEA’s decision to drop “theory” as a separate category. The Machlup-Hutchison-Rothbard exchange helps establish the context.
[T]he seemingly administrative task of devising new categories threw AEA officials, in particular AER editor Bernard Haley and former AER interim editor Fritz Machlup, into heated debates over the nature and relationships of theoretical and empirical work.
Machlup campaigned for a separate “Abstract Economic Theory” top category. At the time of the revision, he was engaged in methodological work, striving to find a third way between Terence Hutchison’s “ultraempiricism,” and the “extreme a priorism” of his former mentor, Ludwig Von Mises (see Blaug, ch.4). He believed it was possible to differentiate between “fundamental (heuristic) hypotheses, which are not independently testable,” and “specific (factual) assumptions, which are supposed to correspond to observed facts or conditions.” The former was found in Keynes’s General Theory, and the latter in his Treatise on Money, Machlup explained. He thus proposed that empirical analysis be classified independently, under two categories: “Quantitative Research Techniques” and “Social Accounting, Measurements, and Numerical Hypotheses” (e.g., census data, expenditure surveys, input-output matrices, etc.). On the contrary, Haley wanted every category to cover the theoretical and empirical work related to a given subject matter. In his view, separating them was impossible, even meaningless: “Is there any theory that is not abstract? And, for that matter, is there any economic theory worth its salt that is not applied,” he teased Machlup. Also, he wanted to avoid the idea that “class 1 is theory, the rest are applied … How about monetary theory, international trade theory, business cycle theory?” He accordingly designed the top category to encompass price theory, but also statistical demand analysis, as well as “both theoretical and empirical studies of, e.g., the consumption function [and] economic growth models of the Harrod-Domar variety,” among other subjects. He eschewed any “theory” heading, which he replaced with titles such as “Price system; National Income Analysis.” His scheme eventually prevailed, but “theory” was reinstated in the title of the contentious category.
| Peter Klein |
Joe Gillis: You’re Norma Desmond. You used to be in silent pictures. You used to be big.
Norma Desmond: I *am* big. It’s the *pictures* that got small.
– Sunset Boulevard (1950)
John List gave the keynote address at this weekend’s Southern Economic Association annual meeting. List is a pioneer in the use by economists of field experiments or randomized controlled trials, and his talk summarized some of his recent work and offered some general reflections on the field. It was a good talk, lively and engaging, and the crowd gave him a very enthusiastic response.
List opened and closed his talk with a well-known quote from Paul Samuelson’s textbook (e.g., this version from the 1985 edition, coauthored with William Nordhaus): “Economists . . . cannot perform the controlled experiments of chemists and biologists because they cannot easily control other important factors.” While professing appropriate respect for the achievements of Samuelson and Nordhaus, List shared the quote mainly to ridicule it. The rise of behavioral and experimental economics over the last few decades — in particular, the recent literature on field experiments or RCTs — shows that economists can and do perform experiments. Moreover, List argues, field experiments are even better than using laboratories, or conventional econometric methods with instrumental variables, propensity score matching, differences-in-differences, etc., because random assignment can do the identification. With a large enough sample, and careful experimental design, the researcher can identify causal relationships by comparing the effects of various interventions on treatment and control groups in the field, in a natural setting, not an artificial or simulated one.
While I enjoyed List’s talk, I became increasingly frustrated as it progressed, and found myself — I can’t believe I’m writing these words — defending Samuelson and Nordhaus. Of course, not only neoclassical economists, but nearly all economists, especially the Austrians, have denied explicitly that economics is an experimental science. “History can neither prove nor disprove any general statement in the manner in which the natural sciences accept or reject a hypothesis on the ground of laboratory experiments,” writes Mises (Human Action, p. 31). “Neither experimental verification nor experimental falsification of a general proposition are possible in this field.” The reason, Mises argues, is that history consists of non-repeatable events. “There are in [the social sciences] no such things as experimentally established facts. All experience in this field is, as must be repeated again and again, historical experience, that is, experience of complex phenomena” (Epistemological Problems of Economics, p. 69). To trace out relationships among such complex phenomena requires deductive theory.
Does experimental economics disprove this contention? Not really. List summarized two strands of his own work. The first deals with school achievement. List and his colleagues have partnered with a suburban Chicago school district to perform a series of randomized controlled trials on teacher and student performance. In one set of experiments, teachers were given various monetary incentives if their students improved their scores on standardized tests. The experiments revealed strong evidence for loss aversion: offering teachers year-end cash bonuses if their student improved had little effect on test scores, but giving teacher cash up front, and making them return it at the end of the year if their students did not improve, had a huge effect. Likewise, giving students $20 before a test, with the understanding that they have to give the money back if they don’t do well, leads to large improvements in test scores. Another set of randomized trials showed that responses to charitable fundraising letters are strongly impacted by the structure of the “ask.”
To be sure, this is interesting stuff, and school achievement and fundraising effectiveness are important social problems. But I found myself asking, again and again, where’s the economics? The proposed mechanisms involve a little psychology, and some basic economic intuition along the lines of “people respond to incentives.” But that’s about it. I couldn’t see anything in these design and execution of these experiments that would require a PhD in economics, or sociology, or psychology, or even a basic college economics course. From the perspective of economic theory, the problems seem pretty trivial. I suspect that Samuelson and Nordhaus had in mind the “big questions” of economics and social science: Is capitalism more efficient than socialism? What causes business cycles? Is there a tradeoff between inflation and unemployment? What is the case for free trade? Should we go back to the gold standard? Why do nations to go war? It’s not clear to me how field experiments can shed light on these kinds of problems. Sure, we can use randomized controlled trials to find out why some people prefer red to blue, or what affects their self-reported happiness, or why we eat junk food instead of vegetables. But do you really need to invest 5-7 years getting a PhD in economics to do this sort of work? Is this the most valuable use of the best and brightest in the field?
My guess is that Samuelson and Nordhaus would reply to List: “We are big. It’s the economics that got small.”
See also: Identification versus Importance
Peter invited me to reply to [Warren Miller’s] comment, so I’ll try to offer a defense of formal economic modeling.
In answering Peter’s invitation, I’m at a bit of a disadvantage because I am definitely NOT an IO economist (perhaps because I actually CAN relax). Rather, I’m a strategy guy — far more interested in studying the private welfare of firms than the public welfare of economies (plus, it pays better and is more fun). So, I am in a much better position to comment on the benefits that the game-theoretic toolbox is currently starting to bring to the strategy field than on the benefits that it has brought to the economics discipline over the last four decades (i.e., since Akerlof’s 1970 Lemons paper really jump-started the trend).
Peter writes, “game theory was supposed to add transparency and ‘rigor’ to the analysis.” I have heard this argument many times (e.g., Adner et al, 2009 AMR), and I think it is a red herring, or at least a side show. Yes, formal modeling does add transparency and rigor, but that’s not its main benefit. If the only benefit of formal modeling were simply about improving transparency and rigor then I suspect that it would never have achieved much influence at all. Formal modeling, like any research tool or method, is best judged according to the degree of insight — not the degree of precision — that it brings to the field.
I can’t think of any empirical researcher who has gained fame merely by finding techniques to reduce the amount of noise in the estimate of a regression parameter that has already been the subject of other previous studies. Only if that improved estimation technique generates results that are dramatically different from previous results (or from expected results) would the improved precision of the estimate matter much — i.e., only if the improved precision led to a valuable new insight. In that case, it would really be the insight that mattered, not the precision. The impact of empirical work is proportionate to its degree of new insight, not to its degree of precision. The excruciatingly unsophisticated empirical methods in Ned Bowman’s highly influential “Risk-Return Paradox” and “Risk-Seeking by Troubled Firms” papers provide a great example of this point.
The same general principle is true of theoretical work as well. I can’t think of any formal modeler who has gained fame merely by sharpening the precision of an existing verbal theory. Such minor contributions, if they get published at all, are barely noticed and quickly forgotten. A formal model only has real impact when it generates some valuable new insight. As with empirics, the insight is what really matters, not the precision. (more…)
| Peter Klein |
As a second-year economics PhD student I took the field sequence in industrial organization. The primary text in the fall course was Jean Tirole’s Theory of Industrial Organization, then just a year old. I found it a difficult book — a detailed overview of the “new,” game-theoretic IO, featuring straightforward explanations and numerous insights and useful observations but shot through with brash, unsubstantiated assumptions and written in an extremely terse, almost smug style that rubbed me the wrong way. After all, game theory was supposed to add transparency and “rigor” to the analysis, bringing to light the hidden assumptions of the old-fashioned, verbal models, but Tirole combined math and ad hoc verbal asides in equal measure. (Sample statement: “The Coase theorem (1960) asserts that an optimal allocation of resources can always be achieved through market forces, irrespective of the legal liability assignment, if information is perfect and transactions are costless.” And then: “We conclude that the Coase theorem is unlikely to apply here and that selective government intervention may be desirable.”) Well, that’s the way formal theorists write and, if you know the code and read wisely, you can gain insight into how these economists think about things. Is it the best way to learn about real markets and real competition? Tirole takes it as self-evident that MIT-style theory is a huge advance over the earlier IO literature, which he characterizes as “the old oral tradition of behavioral stories.” He does not, to my knowledge, deal with the “new learning” of the 1960s and 1970s, associated mainly with Chicago economists (but also Austrian and public choice economists) that emphasized informational and incentive problems of regulators as well as firms.
Tirole is one of the most important economists in modern theoretical IO, public economics, regulation, and corporate finance, and it’s no surprise that the Nobel committee honored him with today’s prize. The Nobel PR team struggled to summarize his contributions for the nonspecialist reader (settling on the silly phrase that his work shows how to “tame” big firms) but you can find decent summaries in the usual places (e.g., WSJ, NYT, Economist) and sympathetic, even hagiographic treatments in the blogosphere (Cowen, Gans). By all accounts Tirole is a nice guy and an excellent teacher, as well as the first French economics laureate since Maurice Allais, so bully for him.
I do think Tirole-style IO is an improvement over the old structure-conduct-performance paradigm, which focused on simple correlations, rather than causal explanations and eschewed comparative institutional analysis, modeling regulators as omniscient, benevolent dictators. The newer approach starts with agency theory and information theory — e.g., modeling regulators as imperfectly informed principals and regulated firms as agents whose actions might differ from those preferred by their principals — and thus draws attention to underlying mechanisms, differences in incentives and information, dynamic interaction, and so on. However, the newer approach ultimately rests on the old market structure / market power analysis in which monopoly is defined as the short-term ability to set price above marginal cost, consumer welfare is measured as the area under the static demand curve, and so on. It’s neoclassical monopoly and competition theory on steroids, and hence side-steps the interesting objections raised by the Austrians and UCLA price theorists. In other words, the new IO focuses on more complex interactions while still eschewing comparative institutional analysis and modeling regulators as benevolent, albeit imperfectly informed, “social planners.”
As a student I found Tirole’s analysis extremely abstract, with little attention to how these theories might work in practice. Even Tirole’s later book with Jean-Jacques Laffont, A Theory of Incentives in Procurement and Regulation, is not very applied. But evidently Tirole has played a large personal and professional role in training and advising European regulatory bodies, so his work seems to have had a substantial impact on policy. (See, however, Sam Peltzman’s unflattering review of the 1989 Handbook of Industrial Organization, which complains that game-theoretic IO seems more about solving clever puzzles than understanding real markets.)
| Peter Klein |
Here at O&M we have been somewhat skeptical of the behavioral social science literature. Sure, in laboratory experiments, people often behave in ways inconsistent with “rational” behavior (as defined by neoclassical economics). Yes, people seem to use various rules-of-thumb in making complex decisions. And yet, it’s not clear that the huge literature on such biases and heuristics tells us much we don’t already know.
An interesting essay by Steven Poole argues the behavioralists’ claims are overstated, mainly by relying on a narrow, superficial notion of rationality as the benchmark case. Contemporary psychology suggests that people interpret the questions posed in laboratory experiments in a nuanced, contextual manner in which their seemingly “irrational” answers are actually reasonable.
There are many other good reasons to give ‘wrong’ answers to questions that are designed to reveal cognitive biases. The cognitive psychologist Jonathan St B T Evans was one of the first to propose a ‘dual-process’ picture of reasoning in the 1980s, but he resists talk of ‘System 1’ and ‘System 2’ as though they are entirely discrete, and argues against the automatic inference from bias to irrationality. . . . In general, Evans concludes that a ‘strictly logical’ answer will be less ‘adaptive to everyday needs’ for most people in many such cases of deductive reasoning. ‘A related finding,’ he continues, ‘is that, even though people may be told to assume the premises of arguments are true, they are reluctant to draw conclusions if they personally do not believe the premises. In real life, of course, it makes perfect sense to base your reasoning only on information that you believe to be true.’ In any contest between what ‘makes perfect sense’ in normal life and what is defined as ‘rational’ by economists or logicians, you might think it rational, according to a more generous meaning of that term, to prefer the former. Evans concludes: ‘It is far from clear that such biases should be regarded as evidence of irrationality.’
Poole also argues strongly against the liberal-paternalist “nudges” advocated by Cass Sunstein and Richard Thaler, noting that “there is something troubling about the way in which [nudging] is able to marginalise political discussion.” Moreover, “nudge politics is at odds with public reason itself: its viability depends precisely on the public not overcoming their biases.” Poole concludes:
[T]here is less reason than many think to doubt humans’ ability to be reasonable. The dissenting critiques of the cognitive-bias literature argue that people are not, in fact, as individually irrational as the present cultural climate assumes. And proponents of debiasing argue that we can each become more rational with practice. But even if we each acted as irrationally as often as the most pessimistic picture implies, that would be no cause to flatten democratic deliberation into the weighted engineering of consumer choices, as nudge politics seeks to do. On the contrary, public reason is our best hope for survival. Even a reasoned argument to the effect that human rationality is fatally compromised is itself an exercise in rationality. Albeit rather a perverse, and – we might suppose – ultimately self-defeating one.
Worth a read. Even climate-change skepticism gets a nod, in a form consistent with some reflections here.
| Nicolai Foss |
As readers of this blog will know, probably to a nauseating extent, microfoundations have been central in much (macro) management theory over the last decade. Several articles, special issues, and conferences have been dedicated to microfoundations, most recently a Strategic Management Society Special Conference at the Copenhagen Business School. Some, uhm, highlyspirited exchanges have taken place (e.g., AoM 2013), with proponents of those foundations being accused of economics imperalism and whatnot, and critics of microfoundations receiving push-back for endorsing defunct Durkheimian collectivism (an obviously justified criticism). Here is recent civilized exchange on the subject between Professor Rodolphe Durand, HEC Paris, and myself. Complete with heavy Euro accents of different origins.