Come off it, REF!

Yesterday we all trooped off to the Millennium Stadium in Cardiff for a Staff Away Day. We didn’t actually get to play on the pitch of course, which wasn’t even there, as it had been removed to reveal a vast expanse of soil. Instead we were installed in the “Dragon Suite” for a discussion about our preparation for the forthcoming Research Excellence Framework.

Obviously I can’t post anything about our internal deliberations, but I’m sure departments up and down the United Kingdom are doing similar things so I thought I’d mention a few things which are already in the public domain and my personal reactions to them. I should also say that the opinions I express below are my own and not necessarily those of anyone else at Cardiff.

The first thing is the scale of the task facing members of the panel undertaking this assessment. Each research active member of staff is requested to submit four research publications (“outputs”) to the panel, and we are told that each of these will be read by at least two panel members. The panel comprises 20 members.

As a rough guess I’d say that the UK has about 40 Physics departments, and the average number of research-active staff in each is probably about 40. That gives about 1600 individuals for the REF. Actually the number of category A staff submitted to the 2008 RAE was 1,685.57 FTE (Full-Time Equivalent), pretty  close to this figure. At 4 outputs per person that gives 6400 papers to be read. We’re told that each will be read by at least two members of the panel, so that gives an overall job size of 12800 paper-readings. There are 20 members of the panel, so that means that between 29th November 2013 (the deadline for submissions) and the announcement of the results in December 2014 each member of the panel will have to have read 640 research papers. That’s an average of about two a day…

Incidentally, as I’ve mentioned before, the Physics REF panel includes representatives from institutions in England, Scotland and Northern Ireland, but not Wales. The decision to exclude representation from Welsh physics departments was a disgrace, in my view.

Now we are told the panel will use their expert judgment to decide which outputs belong to the following categories:

  • 4*  World Leading
  • 3* Internationally Excellent
  • 2* Internationally Recognized
  • 1* Nationally Recognized
  • U   Unclassified

There is an expectation that the so-called QR  funding allocated as a result of the 2013 REF will be heavily weighted towards 4*, with perhaps a small allocation to 3* and probably nothing at all for lower grades. “Internationally recognized” research is probably worthless in the view of HEFCE, in other words. Will the papers belonging to the category “Not really understood by the panel member” suffer the same fate?

The panel members will apparently know enough about every single one of the papers they are going to read in order to place them  into one of the above categories, especially the crucial ones “world-leading” or “internationally excellent”, both of which are obviously defined in a completely transparent and objective manner. Not.

We are told that after forming this judgement based on their expertise the panel members will “check” the citation information for the papers. This will be done using the SCOPUS service provided (no doubt at considerable cost) by   Elsevier, which by sheer coincidence also happens to be a purveyor of ridiculously overpriced academic journals. I’ve just checked the citation information for some of my papers on SCOPUS, and found an alarming number of errors. No doubt Elsevier are  on a nice little earner peddling meaningless data for the HECFE bean-counters, but I haven’t any confidence that it will add much value to the assessment process.

There have been high-profile statements to the effect that the REF will take no account of where the relevant “outputs”  are published, including a recent pronouncement by David Willetts. On the face of it, that would suggest that a paper published in the spirit of Open Access in a free archive would not be disadvantaged. However, I very much doubt that will be the case.

I think if you look at the volume of work facing the REF panel members it’s pretty clear that citation statistics will be much more important for the Physics panel than we’ve been led to believe. The panel simply won’t have the time or the breadth of understanding to do an in-depth assessment of every paper, so will inevitably in many cases be led by bibliometric information. The fact that SCOPUS doesn’t cover the arXiv means that citation information will be entirely missing from papers just published there.

The involvement of  a company like Elsevier in this system just demonstrates the extent to which the machinery of research assessment is driven by the academic publishing industry. The REF is now pretty much the only reason why we have to use traditional journals. It would be better for research, better for public accountability and better economically if we all published our research free of charge in open archives. It wouldn’t be good for academic publishing houses, however, so they’re naturally very keen to keep things just the way they are. The saddest thing is that we’re all so cowed by the system that we see no alternative but to participate in this scam.

Incidentally we were told before the 2008 Research Assessment Exercise that citation data would emphatically not be used;  we were also told afterwards that citation data had been used by the Physics panel. That’s just one of the reasons why I’m very sceptical about the veracity of some of the pronouncements coming out from the REF establishment. Who knows what they actually do behind closed doors?  All the documentation is shredded after the results are published. Who can trust such a system?

To put it bluntly, the apparatus of research assessment has done what most bureaucracies eventually do; it has become  entirely self-serving. It is imposing increasingly  ridiculous administrative burdens on researchers, inventing increasingly  arbitrary assessment criteria and wasting increasing amounts of money on red tape which should actually be going to fund research.

And that’s all just about “outputs”. I haven’t even started on “impact”….

13 Responses to “Come off it, REF!”

  1. Interesting about SCOPUS. Last I heard the only panel that had announced was the Computing panel, who decided to use web of science.

  2. […] “Yesterday we all trooped off to the Millennium Stadium in Cardiff for a Staff Away Day. We didn’t actually get to play on the pitch of course, which wasn’t even there, as it had been removed to reveal a vast expanse of soil … (more) […]

  3. interesting – i just went and checked out scopus (or at least the minimal access i can get in the absence of any institutional login…)

    it looks as though scopus has 98% of my refereed publications (pretty good) but only 28% of the citations which NASA ADS reports for these papers or 34% of the citations from refereed publications. so its missing roughly two thirds (my recollection is WoS is a bit better, but it still doesn’t do as well as ADS). when i can get access i’ll have a look what the distribution is in date/journal/etc.

    i imagine there will be a lot of complaints.

    • telescoper Says:

      Some of my papers are missing only about 30%, others more like 90%. It probably depends which journals they’re being cited from.

    • i went back and checked “Web of Knowledge” and its actually improved quite considerably since 2007. it now agrees to within 1-3% with ADS on citations.

      apparently HEFCE are in discussion about how to allow access to SCOPUS for the majority of UK institutions who don’t currently have it. let’s hope this doesn’t involve more payments.

  4. Anton Garrett Says:

    You just know that any initiative with the word “excellence” in its title is going to be bullshit. (When the University of Cambridge cricket team’s performance against the counties finally became too embarrrassing it was rebranded as a “Centre of Excellence”.) Although I must say I see no reason for Elsevier not to be involved, in a sort of PPI. (To continue the cricketing analogy, private TV companies are involved in replay decisions.) Did you know how Elsevier got its name, incidentally?

    No system is perfect but this one does sound pretty lousy, particularly at recognising innovative ideas.

  5. In spite of its flaws I value peer review though dislike it being tied to paid journals.

    IMO consortia of university groups should form to provide a peer review service for papers submitted to the arxiv. An author can choose which consortium would review his/her paper and the reader would know that minimum checks had been made.

    There would administrative costs involved but these could be easily managed. I’m a member of a large collaboration and we have an internal peer review system, designed and staffed by physcists working part time on the task, which performs well. If done efficiently we could probably save money if we, at the same time, stop paying subscriptions to the journals.

    In this scenario we could have a diversity of models. For example, some consortia may implement anonymous peer review and others have named reviewers. It will soon become clear which consortia “publish” papers which are well regarded and which don’t. Furthermore, there is no requirement that a consortium must exist indefinitely. The papers are stored safely on the arxiv and there could presumably be a common repository for correspondence related to the papers.

    To implement this model would be an experiment and could go badly wrong. Then again the current system is outmoded and sick and I doubt that even a failed experiment would deliver a worse service.

  6. Regarding the database, our community uses Spires/Inspire. Its madness to choose another one which is . Its even more ridiculous to choose another one, not least since we have to pay for it.

  7. […] thought I’d post a quick follow-up to last week’s item about the Research Excellence Framework (REF). You will recall that in that post I expressed […]

  8. […] This, together with the post I reblogged earlier this morning, suggested that I should re-hash an article I wrote some time ago about the arithmetic of the REF, and how it will clearly not do what it says […]

  9. […] which include details of relevant publications etc and then a panel sits in judgement. I’ve already blogged of all this: the panels clearly won’t have time to read every paper submitted in any detail […]

  10. […] the REF will be primarily based on peer review. If you are then I suggest you read on because, as I have blogged about before, although peer review is fine in principle, the way that it will be implemented as part of the REF […]

  11. […] the REF will be primarily based on peer review. If you are then I suggest you read on because, as I have blogged about before, although peer review is fine in principle, the way that it will be implemented as part of the REF […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: