Skewness in Journal Rankings

10 September 2009 at 4:40 am Lasse 3 comments

| Lasse Lien |

Here is an interesting piece about journal rankings in economics (abstract below). See also Steve Phelan’s comment and link about similar issues in management (#3).

Nearly all journal rankings in economics use some weighted average of citations to calculate a journal’s impact. These rankings are often used, formally or informally, to help assess the publication success of individual economists or institutions. Although ranking methods and opinions are legion, scant attention has been paid to the usefulness of any ranking as representative of the many articles published in a journal. First, because the distributions of citations across articles within a journal are seriously skewed, and the skewness differs across journals, the appropriate measure of central tendency is the median rather than the mean. Second, large shares of articles in the highest-ranked journals are cited less frequently than typical articles in much-lower-ranked journals.

Source: Wall, Howard J. (2009), “Don’t Get Skewed Over by Journal Rankings,” The B.E. Journal of Economic Analysis & Policy: Vol. 9 (1).

Entry filed under: - Lien -, Papers.

Corporate Diversification Humor Professorial Role Models

3 Comments Add your own

1. Steve Phelan | 10 September 2009 at 11:21 am

Precisely! This is where those with a large number of tier one hits become curiously silent :-)
2. gabrielrossman | 10 September 2009 at 11:58 am

I posted some code a few months ago on how to build confidence intervals when using this kind of data (aggregates of counts). in this case the article is the “title,” the journal is the “trait,” and the cites are “sales.”

Bootstrapping superstars
3. ajb | 11 September 2009 at 8:51 am

Actually, this points up the need for departments to trust their own judgment. Publication in quality journals is only one signal. A good department should be able to ignore that signal if they feel they have better information. Mechanical reliance on any index is what leads to conformity and adverse selection.