Six (very) bad things about the REF

I see that Jon Butterworth has written a piece on the Grauniad website, entitled Six good things about the REF, the REF in question not being a black-clad figure of questionable parentage and visual acuity responsible for supervising a game of association football, but the Research Excellence Framework.

I agree with some of Jon’s comments and do believe that past Research Assessment Exercises have generally raised the quality of research in UK universities. I do however think that there are some very bad things about the way the REF is being implemented, and that these far outweigh the positives Jon mentions. In the interest of balance, therefore, I thought I’d respond with a list of six (very) bad things about the REF, and particularly how it applies to physics. I’ll keep them brief because I’ve blogged about most of them before:

  1. The rules positively encouraged universities to play games with selectivity. This is absurd. All academic staff on teaching and research contracts should be submitted if a true indication of research quality is to be obtained.
  2. The criteria for what constitutes 3* or 4* publications are vague and subjective, leaving everything in the hands of the panels. Worse, all paperwork will be shredded after the panel’s deliberations leaving no possibility for appeal. This absolutely stinks.
  3. How QR funding will be allocated on the basis of the REF is not made clear in advance of the submission. Nobody knows how heavily the funding will be skewed towards 4* and 3* submissions. Having encouraged departments to play games, therefore, the REF refuses to disclose the rules. It’s not even clear there will be any QR funding.
  4. The panels will be unable to perform a detailed peer review of submissions simply because there will be too many papers. Each panel will be expected to make decisions on many hundreds of papers, leaving time only for a cursory reading of each.
  5. Limiting the physics submission to 4 papers per person is ridiculous. This corresponds to a tiny fraction of the outputs of a typical physics researcher. If someone has written ten 4* publications in the REF period, why should these not be counted?
  6. Impact counts for a sizable fraction (20%) of the funding, but the rules governing what counts as “impact” are absurdly restrictive and clearly encourage short-term commercially-oriented boilerplate stuff at the expense of genuine long-term “blue skies” research.


Well, I got to six in just a few minutes and could easily get to sixty, but that will do for now. Perhaps you’d like to contribute your own bad things through the comments box?

4 Responses to “Six (very) bad things about the REF”

  1. Hah! Surely 1+2+3 lead any intelligent uni to the conclusion they should have the confidence to do what they think is right, rather than try and game a system which is (deliberately?) unclear.

    I think my biggest negative (and I agree there are many) was already highlighted by Jack Stilgoe on twitter – the way it damages interdisciplinarity, tbh.

    • I might accept your argument if I believed there was such a thing as an intelligent uni 😉 or, more seriously, whether there was a well-defined answer to the question what is “right” in this context….

      I agree entirely about interdisciplinarity though; that should have been in my six had I thought longer about it…

  2. 7. If we return all staff on T&R contracts, then game playing continues, with staff pushed onto Teaching and Scholarship contracts.

    8. The non linear reward for 2* to 4* (all “international quality research”) is for me the major problem.

    9. Large submissions, e.g., Clinical Medicine, are nearly career destroying, not the same as a single discipline/Dept submission, but necessary.

    10. This should be the first point: Paul Jump’s excellent summary of the evolution of RAE/REF, including fantastic insights from Peter Swinnerton-Dyer. Essential reading BEFORE writing about REF!

  3. I agree with quite a lot that Peter has written above.

    Perhaps the biggest concern I have is not with the REF itself, but how the results are likely to be applied. Currently HEFCE research funding based on the 2008 RAE is severely biased to 4* quality research compared with 3*, as noted in the comments above. I don’t see much changing with the REF.

    In general, this practice preferentially acknowledges the work of established academics who have been in post for at least several years, which may weight internal resource allocations by university departments to senior academics, regardless of the potential of younger or recently recruited academics. The practice also makes the REF performance critically important, because small differences in the proportion of 4* papers will have significant financial consequences. Presentation of REF submissions therefore becomes overly important and laborious.

    Having recently become an advocate for the interests of women in science, I must also criticise the effect of the REF on academics who take time out to start families. Although allowance can be made for time out from research, the practical effect is to lessen the status of people who have had breaks in their publication record. This has effects, for example, on the availability of departmental support to, and the promotion of, women academics.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: