Posts filed under ‘Methods/Methodology/Theory of Science’
| Peter Klein |
Joe Gillis: You’re Norma Desmond. You used to be in silent pictures. You used to be big.
Norma Desmond: I *am* big. It’s the *pictures* that got small.
– Sunset Boulevard (1950)
John List gave the keynote address at this weekend’s Southern Economic Association annual meeting. List is a pioneer in the use by economists of field experiments or randomized controlled trials, and his talk summarized some of his recent work and offered some general reflections on the field. It was a good talk, lively and engaging, and the crowd gave him a very enthusiastic response.
List opened and closed his talk with a well-known quote from Paul Samuelson’s textbook (e.g., this version from the 1985 edition, coauthored with William Nordhaus): “Economists . . . cannot perform the controlled experiments of chemists and biologists because they cannot easily control other important factors.” While professing appropriate respect for the achievements of Samuelson and Nordhaus, List shared the quote mainly to ridicule it. The rise of behavioral and experimental economics over the last few decades — in particular, the recent literature on field experiments or RCTs — shows that economists can and do perform experiments. Moreover, List argues, field experiments are even better than using laboratories, or conventional econometric methods with instrumental variables, propensity score matching, differences-in-differences, etc., because random assignment can do the identification. With a large enough sample, and careful experimental design, the researcher can identify causal relationships by comparing the effects of various interventions on treatment and control groups in the field, in a natural setting, not an artificial or simulated one.
While I enjoyed List’s talk, I became increasingly frustrated as it progressed. I found myself — I can’t believe I’m about to write these words — defending Samuelson and Nordhaus. Of course, not only neoclassical economists, but nearly all economists, especially the Austrians, have denied explicitly that economics is an experimental science. “History can neither prove nor disprove any general statement in the manner in which the natural sciences accept or reject a hypothesis on the ground of laboratory experiments,” writes Mises (Human Action, p. 31). “Neither experimental verification nor experimental falsification of a general proposition are possible in this field.” The reason, Mises argues, is that history consists of non-repeatable events. “There are in [the social sciences] no such things as experimentally established facts. All experience in this field is, as must be repeated again and again, historical experience, that is, experience of complex phenomena” (Epistemological Problems of Economics, p. 69). To trace out relationships among such complex phenomena requires deductive theory.
Does experimental economics disprove this contention? Not really. List summarized two strands of his own work. The first deals with school achievement. List and his colleagues have partnered with a suburban Chicago school district to perform a series of randomized controlled trials on teacher and student performance. In one set of experiments, teachers were given various monetary incentives if their students improved their scores on standardized tests. The experiments revealed strong evidence for loss aversion: offering teachers year-end cash bonuses if their student improved had little effect on test scores, but giving teacher cash up front, and making them return it at the end of the year if their students did not improve, had a huge effect. Likewise, giving students $20 before a test, with the understanding that they have to give the money back if they don’t do well, leads to large improvements in test scores. Another set of randomized trials showed that responses to charitable fundraising letters are strongly impacted by the structure of the “ask.”
To be sure, this is interesting stuff, and school achievement and fundraising effectiveness are important social problems. But I found myself asking, again and again, where’s the economics? The proposed mechanisms involve a little psychology, and some basic economic intuition along the lines of “people respond to incentives.” But that’s about it. I couldn’t see anything in these design and execution of these experiments that would require a PhD in economics, or sociology, or psychology, or even a basic college economics course. From the perspective of economic theory, the problems seem pretty trivial. I suspect that Samuelson and Nordhaus had in mind the “big questions” of economics and social science: Is capitalism more efficient than socialism? What causes business cycles? Is there a tradeoff between inflation and unemployment? What is the case for free trade? Should we go back to the gold standard? Why do nations to go war? It’s not clear to me how field experiments can shed light on these kinds of problems. Sure, we can use randomized controlled trials to find out why some people prefer red to blue, or what affects their self-reported happiness, or why we eat junk food instead of vegetables. But do you really need to invest 5-7 years getting a PhD in economics to do this sort of work? Is this the most valuable use of the best and brightest in the field?
My guess is that Samuelson and Nordhaus would reply to List: “We’re are big. It’s the economics that got small.”
See also: Identification versus Importance
Peter invited me to reply to [Warren Miller’s] comment, so I’ll try to offer a defense of formal economic modeling.
In answering Peter’s invitation, I’m at a bit of a disadvantage because I am definitely NOT an IO economist (perhaps because I actually CAN relax). Rather, I’m a strategy guy — far more interested in studying the private welfare of firms than the public welfare of economies (plus, it pays better and is more fun). So, I am in a much better position to comment on the benefits that the game-theoretic toolbox is currently starting to bring to the strategy field than on the benefits that it has brought to the economics discipline over the last four decades (i.e., since Akerlof’s 1970 Lemons paper really jump-started the trend).
Peter writes, “game theory was supposed to add transparency and ‘rigor’ to the analysis.” I have heard this argument many times (e.g., Adner et al, 2009 AMR), and I think it is a red herring, or at least a side show. Yes, formal modeling does add transparency and rigor, but that’s not its main benefit. If the only benefit of formal modeling were simply about improving transparency and rigor then I suspect that it would never have achieved much influence at all. Formal modeling, like any research tool or method, is best judged according to the degree of insight — not the degree of precision — that it brings to the field.
I can’t think of any empirical researcher who has gained fame merely by finding techniques to reduce the amount of noise in the estimate of a regression parameter that has already been the subject of other previous studies. Only if that improved estimation technique generates results that are dramatically different from previous results (or from expected results) would the improved precision of the estimate matter much — i.e., only if the improved precision led to a valuable new insight. In that case, it would really be the insight that mattered, not the precision. The impact of empirical work is proportionate to its degree of new insight, not to its degree of precision. The excruciatingly unsophisticated empirical methods in Ned Bowman’s highly influential “Risk-Return Paradox” and “Risk-Seeking by Troubled Firms” papers provide a great example of this point.
The same general principle is true of theoretical work as well. I can’t think of any formal modeler who has gained fame merely by sharpening the precision of an existing verbal theory. Such minor contributions, if they get published at all, are barely noticed and quickly forgotten. A formal model only has real impact when it generates some valuable new insight. As with empirics, the insight is what really matters, not the precision. (more…)
| Peter Klein |
As a second-year economics PhD student I took the field sequence in industrial organization. The primary text in the fall course was Jean Tirole’s Theory of Industrial Organization, then just a year old. I found it a difficult book — a detailed overview of the “new,” game-theoretic IO, featuring straightforward explanations and numerous insights and useful observations but shot through with brash, unsubstantiated assumptions and written in an extremely terse, almost smug style that rubbed me the wrong way. After all, game theory was supposed to add transparency and “rigor” to the analysis, bringing to light the hidden assumptions of the old-fashioned, verbal models, but Tirole combined math and ad hoc verbal asides in equal measure. (Sample statement: “The Coase theorem (1960) asserts that an optimal allocation of resources can always be achieved through market forces, irrespective of the legal liability assignment, if information is perfect and transactions are costless.” And then: “We conclude that the Coase theorem is unlikely to apply here and that selective government intervention may be desirable.”) Well, that’s the way formal theorists write and, if you know the code and read wisely, you can gain insight into how these economists think about things. Is it the best way to learn about real markets and real competition? Tirole takes it as self-evident that MIT-style theory is a huge advance over the earlier IO literature, which he characterizes as “the old oral tradition of behavioral stories.” He does not, to my knowledge, deal with the “new learning” of the 1960s and 1970s, associated mainly with Chicago economists (but also Austrian and public choice economists) that emphasized informational and incentive problems of regulators as well as firms.
Tirole is one of the most important economists in modern theoretical IO, public economics, regulation, and corporate finance, and it’s no surprise that the Nobel committee honored him with today’s prize. The Nobel PR team struggled to summarize his contributions for the nonspecialist reader (settling on the silly phrase that his work shows how to “tame” big firms) but you can find decent summaries in the usual places (e.g., WSJ, NYT, Economist) and sympathetic, even hagiographic treatments in the blogosphere (Cowen, Gans). By all accounts Tirole is a nice guy and an excellent teacher, as well as the first French economics laureate since Maurice Allais, so bully for him.
I do think Tirole-style IO is an improvement over the old structure-conduct-performance paradigm, which focused on simple correlations, rather than causal explanations and eschewed comparative institutional analysis, modeling regulators as omniscient, benevolent dictators. The newer approach starts with agency theory and information theory — e.g., modeling regulators as imperfectly informed principals and regulated firms as agents whose actions might differ from those preferred by their principals — and thus draws attention to underlying mechanisms, differences in incentives and information, dynamic interaction, and so on. However, the newer approach ultimately rests on the old market structure / market power analysis in which monopoly is defined as the short-term ability to set price above marginal cost, consumer welfare is measured as the area under the static demand curve, and so on. It’s neoclassical monopoly and competition theory on steroids, and hence side-steps the interesting objections raised by the Austrians and UCLA price theorists. In other words, the new IO focuses on more complex interactions while still eschewing comparative institutional analysis and modeling regulators as benevolent, albeit imperfectly informed, “social planners.”
As a student I found Tirole’s analysis extremely abstract, with little attention to how these theories might work in practice. Even Tirole’s later book with Jean-Jacques Laffont, A Theory of Incentives in Procurement and Regulation, is not very applied. But evidently Tirole has played a large personal and professional role in training and advising European regulatory bodies, so his work seems to have had a substantial impact on policy. (See, however, Sam Peltzman’s unflattering review of the 1989 Handbook of Industrial Organization, which complains that game-theoretic IO seems more about solving clever puzzles than understanding real markets.)
| Peter Klein |
Here at O&M we have been somewhat skeptical of the behavioral social science literature. Sure, in laboratory experiments, people often behave in ways inconsistent with “rational” behavior (as defined by neoclassical economics). Yes, people seem to use various rules-of-thumb in making complex decisions. And yet, it’s not clear that the huge literature on such biases and heuristics tells us much we don’t already know.
An interesting essay by Steven Poole argues the behavioralists’ claims are overstated, mainly by relying on a narrow, superficial notion of rationality as the benchmark case. Contemporary psychology suggests that people interpret the questions posed in laboratory experiments in a nuanced, contextual manner in which their seemingly “irrational” answers are actually reasonable.
There are many other good reasons to give ‘wrong’ answers to questions that are designed to reveal cognitive biases. The cognitive psychologist Jonathan St B T Evans was one of the first to propose a ‘dual-process’ picture of reasoning in the 1980s, but he resists talk of ‘System 1’ and ‘System 2’ as though they are entirely discrete, and argues against the automatic inference from bias to irrationality. . . . In general, Evans concludes that a ‘strictly logical’ answer will be less ‘adaptive to everyday needs’ for most people in many such cases of deductive reasoning. ‘A related finding,’ he continues, ‘is that, even though people may be told to assume the premises of arguments are true, they are reluctant to draw conclusions if they personally do not believe the premises. In real life, of course, it makes perfect sense to base your reasoning only on information that you believe to be true.’ In any contest between what ‘makes perfect sense’ in normal life and what is defined as ‘rational’ by economists or logicians, you might think it rational, according to a more generous meaning of that term, to prefer the former. Evans concludes: ‘It is far from clear that such biases should be regarded as evidence of irrationality.’
Poole also argues strongly against the liberal-paternalist “nudges” advocated by Cass Sunstein and Richard Thaler, noting that “there is something troubling about the way in which [nudging] is able to marginalise political discussion.” Moreover, “nudge politics is at odds with public reason itself: its viability depends precisely on the public not overcoming their biases.” Poole concludes:
[T]here is less reason than many think to doubt humans’ ability to be reasonable. The dissenting critiques of the cognitive-bias literature argue that people are not, in fact, as individually irrational as the present cultural climate assumes. And proponents of debiasing argue that we can each become more rational with practice. But even if we each acted as irrationally as often as the most pessimistic picture implies, that would be no cause to flatten democratic deliberation into the weighted engineering of consumer choices, as nudge politics seeks to do. On the contrary, public reason is our best hope for survival. Even a reasoned argument to the effect that human rationality is fatally compromised is itself an exercise in rationality. Albeit rather a perverse, and – we might suppose – ultimately self-defeating one.
Worth a read. Even climate-change skepticism gets a nod, in a form consistent with some reflections here.
| Nicolai Foss |
As readers of this blog will know, probably to a nauseating extent, microfoundations have been central in much (macro) management theory over the last decade. Several articles, special issues, and conferences have been dedicated to microfoundations, most recently a Strategic Management Society Special Conference at the Copenhagen Business School. Some, uhm, highlyspirited exchanges have taken place (e.g., AoM 2013), with proponents of those foundations being accused of economics imperalism and whatnot, and critics of microfoundations receiving push-back for endorsing defunct Durkheimian collectivism (an obviously justified criticism). Here is recent civilized exchange on the subject between Professor Rodolphe Durand, HEC Paris, and myself. Complete with heavy Euro accents of different origins.
| Peter Klein |
Carl Menger’s methodology has been described as essentialist. Rather than building artificial models that mimic some attributes or outcomes of an economic process, Menger sought to understand the essential characteristics of phenomena like value, price, and exchange. As Menger explained to his contemporary Léon Walras, Menger and his colleagues “do not simply study quantitative relationships but also the nature [or essence] of economic phenomena.” Abstract models that miss these essential features — even if useful for prediction — do not give the insight needed to understand how economies work, what entrepreneurs do, how government intervention affects outcomes, and so on.
I was reminded of the contrast between Menger and Walras when reading about Henri Matisse and Pablo Picasso, the great twentieth-century pioneers of abstract art. Both painters sought to go beyond traditional, representational forms of visual art, but they tackled the problem in different ways. As Jack D. Flam writes in his 2003 book Matisse and Picasso: The Story of Their Rivalry and Friendship:
Picasso characterized the arbitrariness of representation in his Cubist paintings as resulting from his desire for “a greater plasticity.” Rendering an object as a square or a cube, he said, was not a negation, for “reality was no longer in the object. Reality was in the painting. When the Cubist painter said to himself, ‘I will paint a bowl,’ he set out to do it with the full realization that a bowl in a painting has nothing to do with a bowl in real life.” Matisse, too, was making a distinction between real things and painted things, and fully understood that the two could not be confused. But for Matisse, a painting should evoke the essence of the things it was representing, rather than substitute a completely new and different reality for them. In contract to Picasso’s monochromatic, geometric, and difficult-to-read pictures, Matisse’s paintings were brightly colored, based on organic rhythms, and clearly legible. For all their expressive distortions, they did not have to be “read” in terms of some special language or code.
Menger’s essentialism is concisely described in Larry White’s monograph The Methodology of the Austrian School Economists and treated more fully in Menger’s 1883 book Investigations Into the Method of the Social Sciences. For more on economics and art, see Paul Cantor’s insightful lecture series, “Commerce and Culture” (here and here).
[An earlier version of this post appeared at Circle Bastiat.]
| Nicolai Foss |
Here is a new paper by major Stanford finance scholar, Paul Pfleiderer on what he calls “chameleon models” and their misuse in finance and economics. Lots of catchy concepts, e.g., “theoretical cherry picking” and “bookshelf models,” and an fine critical discussion of Friedmanite instrumentalism. The essence of the paper is this:
Chameleons arise and are often nurtured by the following dynamic. First a bookshelf model is constructed that involves terms and elements that seem to have some relation to the real world and assumptions that are not so unrealistic that they would be dismissed out of hand. The intention of the author, let’s call him or her “Q,” in developing the model may to say something about the real world or the goal may simply be to explore the implications of making a certain set of assumptions. Once Q’s model and results
become known, references are made to it, with statements such as “Q shows that X.” This should be taken as short-hand way of saying “Q shows that under a certain set of assumptions it follows (deductively) that X,” but some people start taking X as a plausible statement about the real world. If someone skeptical about X challenges the assumptions made by Q, some will say that a model shouldn’t be judged by the realism of its assumptions, since all models have assumptions that are unrealistic. Another rejoinder made by those supporting X as something plausibly applying to the real world might be that the truth or falsity of X is an empirical matter and until the appropriate empirical tests or analyses have been conducted and have rejected X, X must be taken seriously. In other words, X is innocent until proven guilty. Now these statements may not be made in quite the stark manner that I have made them here, but the underlying notion still prevails that because there is a model for X, because questioning the assumptions behind X is not appropriate, and because the testable implications of the model supporting X have not been empirically rejected, we must take X seriously. Q’s model (with X as a result) becomes a chameleon that avoids the real world filters.