That Was The REF That Was..

I feel obliged to comment on the results of the 2014 Research Excellence Framework (REF) that were announced today. Actually, I knew about them yesterday but the news was under embargo until one minute past midnight by which time I was tucked up in bed.

The results for the two Units of Assessment relevant to the School of Mathematical and Physical Sciences are available online here for Mathematical Sciences and here for Physics and Astronomy.

To give some background: the overall REF score for a Department is obtained by adding three different components: outputs (quality of research papers); impact (referrring to the impact beyond academia); and environment (which measures such things as grant income, numbers of PhD students and general infrastructure). These are weighted at 65%, 20% and 15% respectively.

Scores are assigned to these categories, e.g. for submitted outputs (usually four per staff member) on a scale of 4* (world-leading), 3* (internationally excellent), 2* (internationally recognised), 1* (nationally recognised) and unclassified and impact on a scale 4* (outstanding), 3* (very considerable), 2* (considerable), 1* (recognised but modest) and unclassified. Impact cases had to be submitted based on the number of staff submitted: two up to 15 staff, three between 15 and 25 and increasing in a like manner with increasing numbers.

The REF will control the allocation of funding in a manner yet to be decided in detail, but it is generally thought that anything scoring 2* or less will attract no funding (so the phrase “internationally recognised” really means “worthless” in the REF, as does “considerable” when applied to impact). It is also thought likely that funding will be heavily weighted towards 4* , perhaps with a ratio of 9:1 between 4* and 3*.

We knew that this REF would be difficult for the School and our fears were born out for both the Department of Mathematics or the Department of Physics and Astronomy because both departments grew considerably (by about 50%) during the course of 2013, largely in response to increased student numbers. New staff can bring outputs from elsewhere, but not impact. The research underpinning the impact has to have been done by staff working in the institution in question. And therein lies the rub for Sussex…

To take the Department of Physics and Astronomy, as an example, last year we increased staff numbers from about 23 to about 38. But the 15 new staff members could not bring any impact with them. Lacking sufficient impact cases to submit more, we were obliged to restrict our submission to fewer than 25. To make matters worse our impact cases were not graded very highly, with only 13.3% of the submission graded 4* and 13.4% graded 3*.

The outputs from Physics & Astronomy at Sussex were very good, with 93% graded 3* or 4*. That’s a higher fraction than Oxford, Cambridge, Imperial College and UCL in fact, and with a Grade Point Average of 3.10. Most other departments also submitted very good outputs – not surprisingly because the UK is actually pretty good at Physics – so the output scores are very highly bunched and a small difference in GPA means a large number of places in the rankings. The impact scores, however, have a much wider dispersion, with the result that despite the relatively small percentage contribution they have a large effect on overall rankings. As a consequence, overall, Sussex Physics & Astronomy slipped down from 14th in the RAE to 34th place in the REF (based on a Grade Point Average). Disappointing to say the least, but we’re not the only fallers. In the 2008 RAE the top-rated physics department was Lancaster; this time round they are 27th.

I now find myself in a situation eerily reminiscent of that I found myself facing in Cardiff after the 2008 Research Assessment Exercise, the forerunner of the REF. Having been through that experience I’m a hardened to disappointments and at least can take heart from Cardiff’s performance this time round. Spirits were very low there after the RAE, but a thorough post-mortem, astute investment in new research areas, and determined preparations for this REF have paid dividends: they have climbed to 6th place this time round. That gives me the chance not only to congratulate my former colleagues there for their excellent result but also to use them as an example for what we at Sussex have to do for next time. An even more remarkable success story is Strathclyde, 34th in the last RAE and now top of the REF table. Congratulations to them too!

Fortunately our strategy is already in hand. The new staff have already started working towards the next REF (widely thought to be likely to happen in 2020) and we are about to start a brand new research activity in experimental physics next year. We will be in a much better position to generate research impact as we diversify our portfolio so that it is not as strongly dominated by “blue skies” research, such as particle physics and astronomy, for which it is much harder to demonstrate economic impact.

I was fully aware of the challenges facing Physics & Astronomy at Sussex when I moved here in February 2013, but with the REF submission made later the same year there was little I could do to alter the situation. Fortunately the University of Sussex management realises that we have to play a long game in Physics and has been very supportive of our continued strategic growth. The result of the 2014 REF result is a setback but it does demonstrate that the stategy we have already embarked upon is the right one.

Roll on 2020!

11 Responses to “That Was The REF That Was..”

  1. Bryn Jones Says:

    I find the concept of impact difficult to appreciate. Does it include outreach, media appearances, and articles or talks aimed at a popular audience? (Astronomers tend to be quite strong with those.) What is the philosophical reason for research outputs moving with academics, but impacts remaining in the university departments? How much guidance is given to universities about these issues?

    • These questions are probably moot because it is mostly bullshit anyway. Why 65/25/15? Why 9:1? Is this any better than just making up some numbers on the spot?

    • telescoper Says:

      It includes anything that’s outside pure academic research: including commercial spin-offs, outreach, effect on government (e.g. changing policy), public engagement etc. There was guidance given, but I don’t think the way that it would be scored was particularly clear. It seems that the only impact that was actually graded 4* had some sort of international dimension, but that wasn’t clear in advance. At least not to me.

      • Bryn Jones Says:

        Presumably many astronomers put emphasis on outreach in the REF, and that is an activity that is often carried out on the initiative of individuals – it is something strongly attached to the academic, rather than the department.

        I can’t remember what the equivalent was in the 2008 RAE, but can’t remember making any reference to outreach or publicity.

      • telescoper Says:

        No, impact was new for the REF 2014. It didn’t exist in 2008.

      • Bryn Jones Says:

        No, nor any kind of reference to anything now included in impact, as far as I recall.

    • Dave Carter Says:

      My reading from various presentations on the REF pilot has been that the answer to Bryn’s questions are yes, no, and yes in that order. It also includes collaborations with industry, where experimental physics departments score well. But goalposts move, so what I had been told may no longer be current.

      As to why research outputs move but research generating impact does not, thats a real anomaly. It means for instance that if an academic who has recently moved to a new institute gives a public talk, they really should concentrate on their research since they moved.

      • Bryn Jones Says:

        The worry I’ve had about impact is, of course, that activities having direct economic impact, such as in engineering and applied science, would skew funding away from basic research. Hence my questions about the kind of impact on society that astronomers might have.

        And thanks for the answers to my questions above.

  2. And I agree with Peter that the guidance given was not particularly clear. The information that only impact with an international dimension was graded 4* is new to me, but very believable. And if we eventually get a breakdown of scores by case study then I think this will be clear.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: