Good to Great: Neither Good nor Great

19 December 2008 at 1:01 am Peter G. Klein 18 comments

| Peter Klein |

I’m not a fan of “guru” books like In Search of Excellence, Built to Last, and Good to Great, for reasons well documented by Phil Rosenzweig in his excellent Halo Effect. These books suffer from ad hoc generalization, sampling on the dependent variable, and a host of related methodological and expository flaws. If Rosenszweig’s critique is startling, then two articles from the November 2008 Academy of Management Perspectives on Jim Collins’s Good to Great — perhaps the leading guru book of our time — are devastating. Here is Bruce Resnick and Timothy Smunt:

With sales of more than 4.5 million copies, Good to Great by Jim Collins provides an inspiring message about how a few major companies became great. His simple but powerful framework for creating a strategy any organization can use to go from goodness to greatness is certainly compelling. However, was Collins truly able to identify 11 great companies? Or was the list of great companies he generated merely the result of applying an arbitrary screening filter to the list of Fortune 500 companies? To test the durability of his greatness filter, we conducted a financial analysis on each of the 11 companies over subsequent periods. We found that only one of the 11 companies continues to exhibit superior stock market performance according to Collins’ measure, and that none do so when measured according to a metric based on modern portfolio theory. We conclude that Collins did not find 11 great companies as defined by the set of parameters he claimed are associated with greatness, or, at least, that greatness is not sustainable.

Later they add:

Our analysis of Collins’ Good to Great study methodology suggests that it suffered from three major problems: 1) data mining with respect to the selection of the starting month of the company transformation period, 2) the failure to test for the sustainability of greatness over subsequent time periods, and 3) the failure to use modern portfolio theory that accounts for the costs of risk and then whether the performance differences are statistically significant.

“Good to Great, or Just Good?” by Bruce Niendorf and Kristine Beck is equally damning:

Good to Great has been on BusinessWeek’s best-seller list since its October 2001 release. In Good to Great, author Jim Collins identified a set of 11 firms as great, then used them to derive five management principles he believed led to “sustained great results.” We contend that due to two fatal errors, Good to Great provides no evidence that applying the five principles to other firms or time periods will lead to anything other than average results. We explain the two errors and empirically test our contention. When ranked with the 2006 Fortune 500, the 11 Good to Great firms have an average ranking of 202nd. In addition, in terms of long-term stock return performance, the Good to Great firms do not differ significantly from the average company on the S&P 500. Our evidence is consistent with the conclusion that although the Good to Great firms may be good, they aren’t great.

Please, don’t let your imagination run wild during this revolutionary time. Throw such books into the blue ocean!

Entry filed under: - Klein -, Management Theory, Methods/Methodology/Theory of Science.

EJPE Now Available Indigenous Entrepreneurship in Rural China

18 Comments Add your own

1. Hakan | 19 December 2008 at 2:21 am

I think the second abstract provided here is almost the same as the first one – may be a paste error.
2. Peter Klein | 19 December 2008 at 9:49 am

Thanks, you’re right, I’ve corrected the error.
3. Rob Millard | 19 December 2008 at 4:28 pm

But …. are the corporations in the sample still practicing the same management practices that Collins identified as being the differentiating factors? Leaders come and go and practices change. To nail this one properly, you need to find the answer to this question as well as analyzing their later financial performance.
4. Peter Klein | 19 December 2008 at 4:36 pm

Rob, remember that the firms in Collins’s sample were chosen not on the basis of their leaders or practices, but their financial performance, from which Collins concluded that the leaders and practices in place at the time were the cause of their above-average performance. In other words, Collins didn’t control for changes in leaders or practices. These two studies simpy replicate Collins’s approach using a different time period, showing that his results don’t hold.

Rosenszweig covers all this pretty thoroughly in The Halo Efffect.
5. Rafe Champion | 19 December 2008 at 7:33 pm

Have people checked for lagged effects, in case bad/good leaders moved on before the bad/good effects emerged at the bottom line?
6. michael webster | 19 December 2008 at 10:47 pm

Peter, do you have any theory or speculation why Rosenszweig isn’t more widely lauded?

It is a terrific read, and dead on.

Does it just remove too many icon?
7. simone | 21 December 2008 at 7:17 pm

These books sell because many executives refuse to really think about their business. They want a formula that can be easily applied and communicated. The formula employed by many is a “Collins story” plus “best practices.” This minimizes thought and ensures that Corporate Relations and HR can develop the communications.
8. David Hoopes | 25 December 2008 at 12:24 am

It’s pretty amazing how many books pick companies with recent success and then create explanations. It reminds me of students who assume that because a company did well they made all of the right decisions. Or, because things did not turn out well they made bad decisions. For students (especially undergrads) I think this is forgivable. For faculty at or other reasonably well educated and experienced people it is not so forgivable (this I cannot forgive).
9. Todd | 1 January 2009 at 5:29 pm

Peter,

I just started reading your blog. I came upon this post of yours. The company I work in has asked us to read Good to Great for a team discussion on good management. I’m very interesting in getting hold of these two articles critiquing the book from the Academy of Management Perspectives. Any idea how I can read into that? Also, why do you think Collin’s criteria for sustained greatness in companies is flawed. He claims in the introductory chapter in the book that his research team made conclusions of the “timeless” concepts of company success through empirical deductions directly from data they had, as opposed to beginning with a biased theory. Your views are welcomed, along with any accredited sources of critique against the “principles” mentioned in this book.
10. Peter Klein | 1 January 2009 at 11:53 pm

Todd, the two articles address this question directly. I don’t know how you can get them without being a subscriber (or Academy of Management member) but I’ll keep my eye out for ungated versions.
11. Todd | 2 January 2009 at 2:47 pm

Thanks Peter. You know, I don’t know whom exactly to ask but what do you think the role of just mere ‘luck’ is in positive business performance? Because if the sample that Mr. Collins chose for the 11 good to greats just had some amount of luck not only in being chosen by him and his team, but also in the 15 year growth, then you cannot discount an element in the experiment called regression to the mean. Maybe I shouldn’t say luck but we all know that oil prices, the stock market, business decisions etc all involves ups and downs. If Mr. Collins had analyzed the financial performance of these good to greats further into the future, he may have gotten different results, most likely worse than what he started with. I hope you understand where I’m going here. I’m not a statistician but I do remember some things learned in school.
12. Peter Klein | 2 January 2009 at 11:57 pm

Todd, I’m sure regression to the mean plays a role. Pick any time period, and the firms that perform the best (by whatever criteria) during that period are unlikely to be at the top in subsequent periods. So I think you’re quite right!
13. Todd | 3 January 2009 at 6:03 pm

Thanks Peter. I don’t know if luck is the word. Maybe randomness in stock performance is better. Collins claims that he chose 15 years as a guage for performance bucase such luck (or maybe randomness) cannot go on in a company for 15 years. I wonder if this claim is really true? That if you increase your time step, you eliminate error from chance?
14. Christopher | 30 January 2009 at 12:54 pm

I agree that there is no such thing as a true guru book, but as a management consultant i have seen lots of organisations that do not practice any of the methodologies described in good to great and beleive that they suffer because of this. Surely the disciplined people, action plus the leadership thoughts on Level 5 leadership could take most organisations a long way towards creating more shareholder value? Look at GE since Jack Welch left? Or Apple since Steve Jobs illness was communicated to Wall Street? What about choosing one thing to be great at and executing with rigour and discipline? I think good to great may not be the silver bullet but is certainly a step in the right direction for most organisations. …..thoughts?
15. Is Your Business A Social Media Fox Or Hedgehog? « SM@RT Social Media For Business | 18 April 2010 at 9:17 pm

[…] but there is just one BIG problem. Because of flaws in the way his research was designed, Collins never proved anything he claimed, including the Hedgehog Concept. In fact, while the 11 companies Collins focused on were […]
16. This Blog Is Dead: I’m Starting Over | 13 August 2011 at 2:19 pm

[…] as a bit of a data geek (and someone who prefers fact to myth), I must note that Good to Great has been called out for having a flawed statistical methodology. So I won’t speak to whether or not the Hedgehog Concept produces the stated results for large […]
17. Nicolay Worren | 11 March 2013 at 3:33 am

Peter,

interesting post that I have shared with my Linkedin contacts. A couple of comments:

The methodology used by Collins is obviously dependent upon the questions being asked. If you ask about A and B, and forget to check C, C will never be identified as a “principle”, even though it may in fact be more important than A and B. I note, for example, that Collins seems disinterested in organization design variables, and doesn’t bother investigating whether org. design played a role (in any case, it’s difficult to find out using this retrospective approach).

Another more philosophical issue: As an organization – or indeed a society – is it more important to promote the “great”, or avoid the “bad”? I am leaning toward the latter. The quality of our society would be significantly enhanced, I believe, if we could just get rid of the really bad institutions, practices, and people. The quality of our organizations would also be improved if we could get rid of the bad.
Why don’t we use Collins’ methods and write the book “From bad to good”?
18. Klein, Peter G. | 11 March 2013 at 10:34 am

Nicolay, you’re right that Collins doesn’t may enough attention to organizational design. Indeed, I’d argue he doesn’t rely on any general theoretical perspective — transaction cost economics, agency theory, the resource-based view — on the determinants of performance. In any case, as we’re already discussing, it’s a moot point because his research design doesn’t allow us to say anything about determinants.