The Dark Side of the REF

There’s a disturbing story in the latest Times Higher which argues that the University of Leicester has apparently reneged on a promise that non-submission to the forthcoming (2014)  Research Excellence Framework (REF) would not have negative career consequences. They have now said that except in exceptional circumstances, non-submitted academics will either be moved to a teaching-only contract (where there is a vacancy and they can demonstrate teaching excellence), or have their performance “managed”, with the threat of sacking if they don’t meet the specified targets.  I’d heard rumours of this on the grapevine (i.e. Twitter) before the Times Higher story was published. It’s very worrying to have it confirmed, as it raises all kinds of questions about what might happen in departments that turn out to have disappointing REF results .

There are (at least) two possible reasons for non-inclusion of the outputs of a researcher and it is important to distinguish between them. One is that the researcher hasn’t enough high-quality outputs to submit. In the absence of individual extenuating circumstances, researchers are expected to submit four “outputs” (in my discipline that means “research papers”) for assessment. That’s a pretty minimal level of productivity, actually;  such a number per year is a reasonable average for an active researcher in my field.  A person employed on a contract that specifies their duties as Teaching and Research may therefore be under-performing  if they can’t produce four papers over the period 2008-2013. I think some form of performance management  may be justifiable in this case, but the primary aim should be to help the individual rather than show them the door. We all have fallow periods in research, and it’s not appropriate to rush to sack anyone who experiences a lean time.   Andrew Wiles would have been considered `inactive’ had there been a REF in 1992 as he hadn’t published anything for years. Then he produced a proof of Fermat’s Last Theorem. Some things just take time.

A second reason for excluding researcher from the REF is that the institution concerned may be making a tactical submission. As the Times Higher article explains:

The memo suggests that academics would be spared repercussions if, among other reasons, the number of individuals submitted is “constrained” by the volume of case studies their department intends to enter to demonstrate research impact.

Institutions must submit one case study for every 10 scholars entered.

Maria Nedeva, professor of science and innovation dynamics and policy at Manchester Business School, said the tactic of deciding how many academics to submit based on impact case study numbers was “rife”.

(Incidentally, the second paragraph is not quite right. The number of case studies required depends on the number of staff submitted as follows: for fewer than 15 staff , TWO case studies;  for 15-24.99 staff it is THREE case studies – and then for each additional ten members of staff entered a further case study is required.)

e case study for every scholars included plus one, i.e. forThe statement at the end of the quote there is in line with my experience too.  The point is that the REF is not just a means of allocating relatively small amounts of so-called `QR’ research funding . Indeed, it remains entirely possible that no funding at all will be allocated following the 2014 exercise. The thinking then is that the number of staff submitted is largely irrelevant; all that will count is league table position.

This by no means the only example of the dangers that lurk when you take league tables too seriously.

If a department is required to submit, say, four impact cases if all staff are included in the REF submission, but only has three viable ones, it would not be unreasonable to submit fewer staff because their overall would be dragged down by a poor impact case even if the output quality of all staff is high.  There will certainly be highly active researchers in UK institutions, including many who hold sizable external research grants, whose outputs are not submitted to the REF. As the article points out, it would be very wrong for managers to penalize scholars who have been excluded because of this sort of game-playing. That’s certainly not going to happen in the School of Mathematical and Physical Sciences at Sussex University.  Not while I’m Head of School, anyway.

Moreover, even researchers whose “outputs” are not selected may still contribute to the “Environment” and/or “Impact” sections so they still, in a very real sense, do participate in their department’s REF submission.

My opinion? All this silliness could easily have been avoided by requiring all staff in all units of assessment to be submitted by all departments. You know, like would have happened if the system were actually designed to identify and reward research excellence. Instead, it’s yet another example of a bureaucratic machine that’s become entirely self-serving. It exists simply because it exists.  Research would be much better off without it.

8 Responses to “The Dark Side of the REF”

  1. Peter – I think its unfair to characterise REF as self-serving, because that implies a level of cynicism by real individuals which I don’t see. Otherwise, your analysis is spot on, and the solution – it should be compulsory for everybody to be entered – is the right one I think. We should campaign for this, assuming something RAE-REF-like survives to another round.

    • telescoper Says:

      I think the REF panels are just doing what, e.g., the AGP does, i.e. the best they can within very difficult constraints. My argument is with the people at HEFCE who *must* know how pointless the REF is.

  2. Anton Garrett Says:

    Glad I’m out of the rat race. Yes it’s public money and needs to be justified, but there is a perfectly good way to do that by seeing who successfully jumps over the hurdles that most others fall at. After you have got a First, a good PhD, held fecund postdoctoral research scholarships, etc, the optimal thing to do is to GIVE YOU THE RESOURCES YOU NEED AND LEAVE YOU ALONE to get on with it, with reviews every few years.

  3. Alexander Murphy Says:

    You said: “There are (at least) two possible reasons for non-inclusion of the outputs of a researcher and it is important to distinguish between them. One is that the researcher hasn’t enough high-quality outputs to submit. In the absence of individual extenuating circumstances, researchers are expected to submit four “outputs” (in my discipline that means “research papers”) for assessment. That’s a pretty minimal level of productivity, actually…”

    I think it important to make the distinction between ‘enough outputs’ and ‘enough high quality outputs’: Isn’t the most likely scenario that the researcher does indeed have >>3 papers, but that an internal process decides that they are not of sufficient quality, and thus the researcher is not submitted (with whatever consequences).

    As I understand it, there will never be ANY feedback on the finally assigned rankings (3*, 4* etc) for individual papers. So the inclusion/exclusion choice can never be (post-)validated.

    Is it reasonable for an institution to terminate a career based on internal assessment of research productivity, over a few year period, with rather artificial criteria set by an external process? Discuss…

    Required inclusion of all staff seems sensible.

    • i agree it is iniquitous not to submit all staff – or at least *all* staff who have contributed to the REF submission through research outputs or to the impact cases or environment.

      the fact that staff can be excluded on the basis of their number/quality of their outputs, but their work still be used in the impact cases or environment is particularly disturbing.

      i do think it is right that staff who are funded for some of their time to do research should be required to demonstrate that they are using that time appropriately.

    • telescoper Says:

      Alexander Murphy makes a good point. Unless you’re on the REF panel concerned you really don’t have much idea what they’re going to do in terms of grading. How on Earth, for example, can anyone know how to draw the line between:

      4* Quality that is world-leading in terms of originality, significance and rigour.

      and
      3* Quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence.

      Perhaps more importantly, how many papers will be classified as:

      2* Quality that is recognised internationally in terms of originality, significance and rigour.

      This category will attract no funding at all, so in effect these words mean “worthless”. Does this mean the panels will be more, or less, likely to grade a paper 2*?

      Answers on a postcard..

  4. […] “There’s a disturbing story in the latest Times Higher which argues that the University of Leicester has apparently reneged on a promise that non-submission to the forthcoming (2014) Research Excellence Framework (REF) would not have negative career consequences …” (more) […]

  5. […] papers. In his blog Peter Coles, Professor of Theoretical Astrophysics at Sussex University, has attacked the REF for becoming self-serving. Quoting from a Times Higher Education (THE) story he also writes of the practice in some […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: