The Worthless University Rankings

The Times Higher World University Rankings, which were released this weekk. The main table can be found here and the methodology used to concoct them here.

Here I wish to reiterate the objection I made last year to the way these tables are manipulated year on year to create an artificial “churn” that renders them unreliable and impossible to interpret in an objective way. In other words, they’re worthless. This year, editor Phil Baty has written an article entitled Standing still is not an option in which he makes a statement that “the overall rankings methodology is the same as last year”. Actually it isn’t. In the page on methodology you will find this:

In 2015-16, we excluded papers with more than 1,000 authors because they were having a disproportionate impact on the citation scores of a small number of universities. This year, we have designed a method for reincorporating these papers. Working with Elsevier, we have developed a new fractional counting approach that ensures that all universities where academics are authors of these papers will receive at least 5 per cent of the value of the paper, and where those that provide the most contributors to the paper receive a proportionately larger contribution.

So the methodology just isn’t “the same as last year”. In fact every year that I’ve seen these rankings there’s been some change in methodology. The change above at least attempts to improve on the absurd decision taken last year to eliminate from the citation count any papers arising from large collaborations. In my view, membership of large world-wide collaborations is in itself an indicator of international research excellence, and such papers should if anything be given greater not lesser weight. But whether you agree with the motivation for the change or not is beside the point.

The real question is how can we be sure that any change in league table position for an institution from year to year are is caused by methodological tweaks rather than changes in “performance”, i.e. not by changes in the metrics but by changes in the way they are combined? Would you trust the outcome of a medical trial in which the response of two groups of patients (e.g. one given medication and the other placebo) were assessed with two different measurement techniques?

There is an obvious and easy way to test for the size of this effect, which is to construct a parallel set of league tables, with this year’s input data but last year’s methodology, which would make it easy to isolate changes in methodology from changes in the performance indicators. The Times Higher – along with other purveyors of similar statistical twaddle – refuses to do this. No scientifically literate person would accept the result of this kind of study unless the systematic effects can be shown to be under control. There is a very easy way for the Times Higher to address this question: all they need to do is publish a set of league tables using, say, the 2015/16 methodology and the 2016/17 data, for comparison with those constructed using this year’s methodology on the 2016/17 data. Any differences between these two tables will give a clear indication of the reliability (or otherwise) of the rankings.

I challenged the Times Higher to do this last year, and they refused. You can draw your own conclusions about why.

12 Responses to “The Worthless University Rankings”

  1. Anton Garrett Says:

    There is an amusing ambiguity in “The worthless university rankings”…

    • Presumably you mean that it could be confused with “the worthless-university rankings”. However, as written, it means what it should. Of course, people who omit the hyphen when writing might assume, wrongly, that the writer of what they are reading also omitted it.

  2. I think we already know the answer to the Coles test. The simple fact that most institutions change places in the rankings each year suggests that the methodology used is not reliable. In the real world, universities don’t change their standards over a period of months – and physics departments don’t become better than their peers overnight

    • There could well be, and indeed probably are, long-term changes, but in the short term everything is dominated by noise and systematics. Admitting that wouldn’t help the compilers sell copy, however, so they hide it.

      • Exactly. The pity is, rankings could be a useful way of highlighting longterm trends such as the lack of investment in third level education in Ireland in the last 2 decades, but the signal is getting buried in the noise

      • Worse, it also drives University managers to target short-term improvements by gaming the metrics.

  3. […] “The Times Higher World University Rankings, which were released this week. The main table can be found here and the methodology used to concoct them here. Here I wish to reiterate the objection I made last year to the way these tables are manipulated year on year to create an artificial ‘churn’ that renders them unreliable …” (more) […]

  4. in collaborations such as ATLAS or CMS there are approx. 3000 authors in alphabetical ordering. So how on earth can they determine this?:

    “where those that provide the most contributors to the paper receive a proportionately larger contribution.”

    I hope they didn’t just assume the first author made the most contribution….lucky people beginning with A if so!

    • It really does differ from field to field. In some fields, the main author is listed last. Reminds me of the beginning of Leave It to Beaver: “Starring Barbara Billingsley, Hugh Beaumont, Tony Dow, and Jerry Mathers as The Beaver.” If a rock group has a main musician, he usually introduces the others first, then lets himself be introduced.

      A name like Aarseth is not necessarily a blessing: some might just assume he is first because of alphabetical ordering, whereas he actually did most of the work!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: