Rival Hypotheses in the Same Paper?

3 June 2007 at 6:20 am Nicolai Foss 13 comments

| Nicolai Foss |

Apparently, having rival hypotheses in the same paper is becoming a don’t do! — at least in management! My colleagues who are more empirically minded than I am tell me of rejections that are motivated solely by having rival hypotheses in the same paper. Big guys in the relevant field (e.g., strategic management, organization, international business) also argue against rival hypotheses-in-the-same-paper in doctoral consortia, professional development workshops, etc.

Instead, what is recommended is this:

1) concentrate on developing one set of non-rival hypotheses and deliberately neglect contraditory hypotheses

2) allow for hypotheses that may seem rival, but really aren’t, because they are special cases of a more over-arching framework. In the latter case, the theory development exercises become a matter of identifying the conditions under which H1 is true (and H2 is false) and the conditions under which H1 is false (but H2 is true) etc.

I have great difficulties with this recommendation.

Recommendation 1) seems to me to be dubious for fundamental ethics reasons (if not fraudulent). Surely, this procedure may lead to a neglect of counter-arguments (and relevant evidence) that runs directly counter to the scientific ideal.

Option 2) seems less controversial: Isn’t an overall integrated framework clearly superior to simply stating contradictory hypotheses? Well, suppose that the hypotheses are actually contradictory, because the underlying theories are completely opposed. There may well be cases where there simply is no overarching framework that can subsume the rival theories as special cases. For example, how should theories from Durkheimian sociology be integrated in this fashion with, say, standard micro-economic? Or, the theories may be so distant that the overarching framework becomes so general that the result is trivial and uninformative.

Furthermore, this proposal, though aesthetically pleasing, may simply be too ambitious: Building integrative theories that can subsume established, seemingly rival theories as special cases is a task for the very best theorists, in short, something that can only be done by a small minority in any field. To require this of any paper with rival hypotheses is ambitious to the degree where it threathens scientific progress because much less empirical work will be forthcoming under this methodological stipulation. In this case, as in the case of option 1) above, valuable information may not be produced and published.

Entry filed under: - Foss -, Methods/Methodology/Theory of Science.

Color Me Beautiful More Robbins

13 Comments Add your own

1. Vladimir Dzhuvinov | 3 June 2007 at 9:12 am

Science should not know fear.
2. Carl Henning Reschke | 3 June 2007 at 2:48 pm

a) 1)Could you name those big guys and 2) some evidence for these statements particularly with 3) some info on context that allows to judge their aims?

b) to me option 1 – in your presentation – seems like another example for how our culture is loosing its foundation in classical “education”. Seems to match into observations of a loss of “good” traditional cultural practices.

In school I still learned that it is good argumentative practice (and a better strategy) to deal with potential counter arguments and criticisms by taking them apart in the course of your argument.

Science is about making decisions about rival hypotheses, interpretations of evidence based on a set of observations, tests, experiments (depending on in which field you work and what is feasible).

Thus, it is good scientific practice to compare rival hypotheses / theories (i.e. systems of interpretation of facts, data, or other evidence), by testing them with a set of data.

Thus so far and with respect to such a context your statement seems right.

Rival hypotheses might (or rather often are indeed) special cases of an underlying reality. There is some truth in all observations, some are better than others, some are more suitable tham others given a specific context. Think of the parts of an elephant that is examined by several blind men. Everyone comes up with different observations and theories about “reality”.

Thus systems of scientifc thought have the ability to press observations into procrustes beds that seem to lead to different “proven” true interpretations of reality – which are thus artefacts of more or less subtle differences in scientific “cognition”.

c) option 2) works if scientists (and “ordinary” people) are able to switch their perspective and manage to develop dialectically the synthesis from thesis and antithesis – or if you already know the overarching framework and your thesis and antithesis map into it – or are constructed so as to map into it. BTW: Is this ethically acceptable?

Literary examples that nicely shows how the integrative methods work are Arthur Conan Doyle’s stories about Sherlock Holmes. Arthur Conan Doyle was incidentally influenced by Charles S. Peirce’ pragmatist philosophy, which stresses abduction (something akin to intuition) as source of knowledge. I cannot claim to be an expert on Peirce, but my understanding of Peirce’ abduction is that it is this process of generating new knowledge by integrating controversial elements into a larger picture.

And now to show an example of how integrative thinking can work:

However, there is also a reality where option 1) is justified and sensible even for classically educated and creative scientists: if testing rival hypotheses is becoming a mechanical exercise (to satisfy reviewers) that neglects potential connections between the rival hypotheses. Thus the problem would lie with the scientists handing in papers mindlessly comparing (straw
-men) rival hypotheses.

That is what we actually often can observe in scientific practice: ideologically blinded representatives of schools of thoughts mindlessly hurling arguments about “reality” at each other – based on selective interpretations of data – without looking for an integrative theory. Discussions, arguments, controversies means business, mean papers for scientists i.e. reputation, tenure and resources – cui bono? Hony soit qui mal y pense. ;)

One example (that should be of interest for some readers) for how scientific (dis-)integration can work is evolutionary versus neoclassical economics.

Both are different interpretations of the same underlying economic behavior / reality. Both theoretical systems can be transformed into each other by making different assumptions about the “value” of human and social systems parameters. So they are controversial elements (if you take a narrow lens) of an overarching framework (if you take a wider perspective).

Milton Friedman has said something to that effect already in his essay on positive methodology in economics in 1953. He argues, evolutionary appraoches are generally feasible, however these would just be too costly relative to methodological individualism (which is wedded to neoclassical economics) in terms of the required theoretical Überbau, so you should apply Occam’s razor and forget about evolutionary economics. In which lies a truth if you can infer psychological states of actors (i.e. know the preferences) and if you can consequently model these .

Can you individually? Sometimes! Can economists infer these for agregates of social actors? Seldomly correctly.
3. bee | 3 June 2007 at 4:21 pm

Theory testing is about coming to a single hypothesis that is substantiated by the data. If one concludes a paper with rival or multiple hypotheses one has not met this criteria.

A paper should offer a theory that parsimoniously explains a phenomena. A single account is the logical end of a paper. Multiple possible explanations means that the researcher has not yet completed his/her task. This logic rests upon Popper and the application of Occam’s razor.

I find the suggestion that a paper (other than a literature review) ending by offering multiple explanations to a phenomena incomplete. It stands more as a piece of journalism than scholarship.

With that said, I think that many papers finish with a set of hypotheses tested and yet offer no clear explanation of the larger phenomena in question.
4. Carl Henning Reschke | 3 June 2007 at 4:43 pm

Bringing Durkheim and (neoclassical) standard microconomics together … nice challenge. Let’ s try this route:

The argument goes along the following lines: Durkheim – Simmel – Parsons – Luhmann to explain differentiation of social / cognitive / mental world. Veblen to connect this to economics. Evolutionary economics to link social and economic change to neoclassical microeconomics via the argument that both are special cases of a more general, evolutionary framework.

Individuals perception / state of mind / tradition / norms / values / preference curves / utility leads to actions and / or communication and influences other individuals who observe / communicate with each other.

Result: aggregation in something called culture made up of mental world of individuals influencing each other across time, geography and levels of aggregation. Looks like a mycelle / rhizome if you graph it (includes thus french postmodern philosophers – n’est-ce pas? ;) ). May develop over time (think history!) from simple origins to systems possibly as diverse as today’s societies with their differentiated niches and countercultures. Each with different world views, traditions, norms, values and of course utility curves for individual members of the (overlapping) social niches (or circles according to Simmel, or sociodemographic milieus according to marketers – economics shines through at the end of the tunnel).

To connect developmental psychologists (think Piaget, Erickson) to sociology: Observation of behavior / education by parents, wider family, peer group and differentiated society forms world view of individuals, who influence their environment again (e.g. by actions or by communication (remember Luhmann’s communication is to be understood as signal, includes thus actions that are observed by “receiver”)):

action -> (leads to) observation -> state of mind -> new action -> new observation -> (possibly) new state of mind a.s.o.

For our sceptical economists:

state of mind, culture, world view => (forms) preferences => utility -> economic decisions

((Can be interpreted as action theoretic sociological framework, however, it is also system theoretic and furthermore a process-theoretic perspective. You can also start with observation (of others) to accomodate those sociological theories that rely on these.))

Now we have to decide: standard microeconomics or not?

Let’s assume a neoclassical world: everyone can compute like a rocket scientist and knows everything instantly, furthermore adaptation speed wrt changes in social structure, institutions and artefacts is (nearly) zero. There is only one best option, and options are unanimously ranked and follow each other smoothly on a plane describing / mapping their characteristics. Nice steady hill. Everybody goes for the best, and gets as far as he / she can. Selection prevails. That is where neoclasssical economics is implicitly evolutionary and tends to fall into social darwinist traps – because it does not understand the limitations of a purely selectionist view (relative to the creative side of evolutionary mutations / socioeconomic innovations), and consequently has to claim that it is not evolutionary at all.

Well, some characteristics of the real world are actually quite nicely captured. Internet, long-distance travel has flattened some smaller hills, or opned some barriers between niches: barriers have been torn down, competition increases (e.g. individuals / organzations in European states like Germany have to compete with those from other European and far Eastern states.)

However, there is no single equilibrium that is reached on a more or less but generally rather smooth ride along a path of adaptation.

Thus if we change the speed of adaptation and information and the flow of information is restricted to some niches / cirlces / in-groups, we move towards a more evolutionary world, with competing niches and several optima. Many best options, many decisions, options cannot be ordered unanimously. People need time to digest information and adapt their mind set, world view, social structures, institutions, artefacts. Some are slower (think bureaucrats) some are faster (think private equity, strategy consultants). Many hills, some higher than others, some connectd to others others not.

Also captures reality to a certain degree.

Innovation is missing so far: depends on world view and accpetance of new ideas / concepts / implementations thereof into peer groups / social circles / sociodemographic milieus / organizations.

Invention depends on world-view, change thereof by encountering new cultural environments / groups / views artefacts.

Model for embodied invention:

artefact -> (leads to) observation -> state of mind -> new artefact -> new observation -> (possibly) new state of mind a.s.o.

BTW: its a heuristic circle so we can include evolutionary epistemology and – at least – some philosophers of science and knowledge as well.

Problem of emergence: related to “crossing” between real and mental world, and levels of aggregation, ontology of things / concepts. If emergence is what we do not know, cannot explain its goes away, or rather moves, over time. Issues for another post ;)

Just a sketch.
5. Bob V | 3 June 2007 at 7:43 pm

I can think of a few reasons to avoid rival hypotheses:

– It usually is an indication of an insufficient literature review. It need not be, but it usually is.
– In testing rival hypotheses, you are guaranteed to break one of them. Someone’s paradigm will lose, and breaking paradigms is a no-no. Extend and modify all you want, but never break. Research that breaks paradigms doesn’t fly because it doesn’t really add understanding. Understanding comes from expressing when in particular the paradigm doesn’t work–not just that it didn’t work.
– The presentation is too confusing for a journal paper. Saying things you end up disproving is a no-no.
6. Nicolai Foss | 4 June 2007 at 12:35 am

Bee,
“Theory testing is about coming to a single hypothesis that is substantiated by the data.” Are you sure that is TESTING? (or rather theory-simulation (Bob Lucas style)).

“Multiple possible explanations means that the researcher has not yet completed his/her task. This logic rests upon Popper and the application of Occam’s razor.” No, it does not. it has nothing to do with either.

“I find the suggestion that a paper (other than a literature review) ending by offering multiple explanations to a phenomena incomplete. It stands more as a piece of journalism than scholarship.” Nobody suggestedanything about “ending by offering…etc.” The point was whether it was legitimate to test rival hypotheses in the same paper. You have offered no _arguments_ why this shouldn’t be legitimate, just your opinions.
7. Nicolai Foss | 4 June 2007 at 12:41 am

Bob V,

“It usually is an indication of an insufficient literature review. It need not be, but it usually is.” How can this be? I can understand why it would be a matter of a very expansive literature review. Is this what you mean?

“- In testing rival hypotheses, you are guaranteed to break one of them. Someone’s paradigm will lose, and breaking paradigms is a no-no. Extend and modify all you want, but never break. Research that breaks paradigms doesn’t fly because it doesn’t really add understanding. wUnderstanding comes from expressing when in particular the paradigm doesn’t work–not just that it didn’t work.” — What an amazing position. Do you really believe this yourself? So if we find a “paradigm” (Whatever that exactly is) falsified, we haven’t added to “understanding”?
– “The presentation is too confusing for a journal paper. Saying things you end up disproving is a no-no.” — But why? Too “confusing”? Why? And what is necessarily wrong with “saying this you disprove”? What is wrong with stating rival theories/hypotheses as well as possible in order to find out which one is right? Like Bee, you don’t give reasons, just your opinions, however grounded.
8. Bob V | 4 June 2007 at 6:08 am

Please don’t accuse my opinions of being at all grounded. They are just my thoughts from being in the industry for a couple of years. I probably should have added that disclaimer.

– From the consortia I’ve attended and my own (ill-fated) attempts, I’ve noticed that when I’ve attempted to write a paper that includes rival hypotheses, I have said “here is an interesting question, let’s see which way the data points.” Neither set of hypotheses are really grounded or developed well. I don’t have any proof of this being the usual case, but it seems to be what goes on.

– Understanding comes from expressing when in particular the paradigm doesn’t work–not just that it didn’t work.”
— What an amazing position. Do you really believe this yourself?
No, I don’t believe it. Actually, I am a bit bitter about it. However, I do know people who have been told by others–perhaps the same big guys in the relevant field–that they can’t just say an idea does not work. It won’t get published.

– As for why stating hypotheses that are rival is confusing to readers, I agree that this should not be the case. My thought is based on “Untangling the Relationship Between Displayed Emotions and Organizational Sales: the case of convenience stores” in AMJ 1988, Vol 31, No. 3. The paper was one of the papers selected for the book “Doing Exemplary Research” (1992). The paper is great because they start off with a set of hypotheses which they disprove and then come up with another explanation for. They say that the reviewers thought it was confusing, and that it was standard practice to write hypotheses post-hoc to support their results even though that’s supposed to be a bad thing. The reason they couldn’t do that was that they had a prior AMR paper that committed them to testing those hypotheses. So, it seems to me that there is significant resistance to writing about hypotheses one does not believe in.

My apologies for not providing any reasons. I don’t think I really have any. Rather, these are my impressions of how things seem to work here.
9. David Hoopes | 4 June 2007 at 9:41 am

I thought the whole point of testing rival hypotheses was to see which was better supported by the data.

The advice as given at doctoral symposia may simply be suggestions for easier publication. Similarly, if you want to get tenure, you are better off doing a series of incremental papers than going for some grand theory. I’ve heard scholars who do not believe that this approahc is interesting but nonetheless recommend it to junior scholars. Obviously, it did not sink in very well with me (at least the part about publishing a lot of [incremental] papers).
10. charlie williams | 5 June 2007 at 1:03 pm

I’ve also heard practical advice to grad students that the perfect dissertation has three theories that lead to alternative outcomes for a parameter: positive, negative, and no effect. Isn’t the crux of the issue whether there is a meaningful null? If a non-effect is a reasonable alternative, then there is some value in testing a single hypothesis against that.

Clearly we move our understanding forward most when we find the place where different theories lead to rival hypotheses. And it would improve our theory to improve our discussion of where these situations would arise.

The problem with the horse race model, is that too often the researcher has a favored outcome that is not acknowledged, and the rival prediction is much less well operationalized than the favored alternative.
11. spostrel | 5 June 2007 at 11:15 pm

This discussion is rather depressing. I agree that there are people out there who want the pretense of science rather than the real thing. Exploratory research, findings that cut against one’s original theory, shootouts between rival hypotheses–all these things are too untidy for some souls. Unfortunately, progress in understanding often comes from these sources.

So we are pressured to present the fiction that we always knew that we would find what we did in fact find. Paint those bullseyes around the bullet holes. Very odd, when you think about the whole purpose of research–to find out things you didn’t know before.
12. Nicolai Foss | 6 June 2007 at 1:15 am

Steve, I entirely agree!! I have never been a big fan of Kuhn but this discussion has illustrated the “normal science” at its worst, I am afraid.
13. links for 2007-06-19 at Jacob Christensen | 19 June 2007 at 7:27 am

[…] Rival Hypotheses in the Same Paper? « Organizations and Markets My colleagues who are more empirically minded than I am tell me of rejections that are motivated solely by having rival hypotheses in the same paper. (tags: academic research sociology) […]