REF moves the goalposts (again)
The topic of the dreaded 2014 Research Excellence Framework came up quite a few times in quite a few different contexts over the last few days, which reminded me that I should comment on a news item that appeared a week or so ago.
As you may or may not be aware, the REF is meant to assess the excellence of university departments in various disciplines and distribute its “QR” research funding accordingly. Institutions complete submissions which include details of relevant publications etc and then a panel sits in judgement. I’ve already blogged of all this: the panels clearly won’t have time to read every paper submitted in any detail at all, so the outcome is likely to be highly subjective. Moreover, HEFCE’s insane policy to award the bulk of its research funds to only the very highest grade (4* – “internationally excellent”) means that small variations in judged quality will turn into enormous discrepancies in the level of research funding. The whole thing is madness, but there seems no way to inject sanity into the process as the deadline for submissions remorselessly approaches.
Now another wrinkle has appeared on the already furrowed brows of those preparing REF submissions. The system allows departments to select staff to be entered; it’s not necessary for everyone to go in. Indeed if only the very best researchers are entered then the typical score for the department will be high, so it will appear higher up in the league tables, and since the cash goes primarily to the top dogs then this might produce almost as much money as including a few less highly rated researchers.
On the other hand, this is a slightly dangerous strategy because it presupposes that one can predict which researchers and what research will be awarded the highest grade. A department will come a cropper if all its high fliers are deemed by the REF panels to be turkeys.
In Wales there’s something that makes this whole system even more absurd, which is that it’s almost certain that there will be no QR funding at all. Welsh universities are spending millions preparing for the REF despite the fact that they’ll get no money even if they do stunningly well. The incentive in Wales is therefore even stronger than it is in England to submit only the high-fliers, as it’s only the position in the league tables that will count.
The problem with a department adopting the strategy of being very selective is that it could have a very negative effect on the career development of younger researchers if they are not included in their departments REF submission. As well as taking the risk that people who manage to convince their Head of School that they are bound to get four stars in the REF may not have the same success with the various grey eminences who make the decision that really matters.
Previous incarnations of the REF (namely the Research Assessment Exercises of 2008 and 2001) did not publish explicit information about exactly how many eligible staff were omitted from the submissions, largely because departments were extremely creative in finding ways of hiding staff they didn’t want to include.
Now however it appears there are plans that the Higher Education Statistics Agency (HESA) will publish its own figures on how many staff it thinks are eligible for inclusion in each department. I’m not sure how accurate these figures will be but they will change the game, in that they will allow compilers of league tables to draw up lists of the departments that prefer playing games to just allowing the REF panels to judge the quality of their research.
I wonder how many universities are hastily revising their submission plans in the light of this new twist?
Follow @telescoper
January 18, 2013 at 6:03 pm
This just gets sillier and sillier.
I am beginning to understand friends and colleagues who have left the country to avoid this kind of thing.
January 19, 2013 at 8:55 am
Reblogged this on To the left of centre and commented:
I had considered writing about this myself. I noticed an article in the Times Higher Education a few days ago about HESA publishing contextual staff data. I’ve written a number of times about my concerns about REF2014 (see, for example, REF2014: Good or Bad and The negative impact of REF) and it is amazing – to me at least – that at this late stage they are deciding on procedures. Rather than writing my own post, I thought I would simply reblog this as it mostly says what I was going to say myself.
January 19, 2013 at 4:14 pm
I’m accustomed to hearing folk criticise STFC following the failure of their dept to win a suitable grant. Presumably what they really mean is “the university robber barons on AGP didn’t rate my science”.
Criticisms of the REF seems similar. Who is making all these REF rules, if not the robber barons that run most of our elite universities?
Most of the commentators on this blog are from those very universities – the winners, loosely speaking – and even they’re not happy, so God help the rest!
January 19, 2013 at 5:40 pm
The REF rules are made by the good folk at HEFCE not by universities, although as you say the VCs of some of the latter will be happy whatever the rules are, as long as their institutions get the dosh.
January 24, 2013 at 12:33 pm
One thing which would help would be if HESA would tell us which staff it thinks are eligible before we have to make the submission.
January 24, 2013 at 12:35 pm
And in general, with the REF, I don’t mind them moving the goalposts, but it would be good if they let us know when they do, and they told us where the goalposts have gone.