Advice for the REF Panels

I thought I’d post a quick follow-up to last week’s item about the Research Excellence Framework (REF). You will recall that in that post I expressed serious doubts about the ability of the REF panel members to carry out a reliable assessment of the “ouputs” being submitted to this exercise, primarily because of the scale of the task in front of them. Each will have to read hundreds of papers, many of them far outside their own area of expertise. In the hope that it’s not too late to influence their approach, I thought I’d offer a few concrete suggestions as to how things might be improved. Most of my comments refer specifically to the Physics panel, but I have a feeling the themes I’ve addressed may apply in other disciplines.

The first area of  concern relates to citations, which we are told will be used during the assesment, although we’re not told precisely how this will be done. I’ve spent a few hours over the last few days looking at the accuracy and reliability various bibliometric databases and have come to the firm conclusion that Google Scholar is by far the best, certainly better than SCOPUS or Web of Knowledge. It’s also completely free. NASA/ADS is also free, and good for astronomy, but probably less complete for the rest of physics. I therefore urge the panel to ditch its commitment to use SCOPUS and adopt Google Scholar instead.

But choosing a sensible database is only part of the solution. Can citations be used sensibly at all for recently published papers? REF submissions must have been published no earlier than 2008 and the deadline is in 2013, so the longest time any paper can have had to garner citations will be five years. I think that’s OK for papers published early in the REF window, but obviously citations for those published in 2012 or 2013 won’t be as numerous.

However, the good thing about Google Scholar (and ADS) is that they include citations from the arXiv as well as from papers already published. Important papers get cited pretty much as soon as they appear on the arXiv, so including these citations will improve the process. That’s another strong argument for using Google Scholar.

The big problem with citation information is that citation rates vary significantly from field to field sit will be very difficult to use bibliometric data in a formulaic sense, but frankly it’s the only way the panel has to assess papers that lie far from their own expertise. Unless anyone else has a suggestion?

I suspect that what some panel members will do is to look beyond the four publications to guide their assessment. They might, for example, be tempted to look up the H-index of the author if they don’t know the area very well. “I don’t really understand the paper by Professor Poindexter but he has an H-index of 95 so is obviously a good chap and his work is probably therefore world-leading”. That sort of thing.

I think this approach would be very wrong indeed. For a start, it seriously disadvantages early career researchers who haven’t had time to build up a back catalogue of high-impact papers. Secondly, and more fundamentally still, it is contrary to the stated aim of the REF, which is to assess only the research carried out in the assessment period, i.e. 2008 to 2013. The H-index would include papers going back far further than 2008.

But as I pointed out in my previous post, it’s going to be impossible for the panel to perform accurate assessments of all the papers they are given: there will just be far too many and too diverse in content. They will obviously therefore have to do something other than what the rest of the community has been told they will do. It’s a sorry state of affairs that dishonesty is built into the system, but there you go. Given that the panel will be forced to cheat, let me suggest that they at least do so fairly. Better than using the H-index of each individual, use the H-index calculated over the REF period only. That will at least ensure that only research done in the REF period will count towards the REF assessment.

Another bone of contention is the assessment of the level of contribution authors have made to each paper, in other words the question of attribution. In astronomy and particle physics, many important papers have very long author lists and may be submitted to the REF by many different authors in different institutions. We are told that what the panel will do is judge whether a given individual has made a “significant” contribution to the paper. If so, that author will be accredited with the score given to the paper. If not, the grade assigned will be the lowest and that author will get no credit at all. Under this scheme one could be an author on a 4* paper but be graded “U”.

This is fair enough, in that it will penalise the “lurkers” who have made a career by attaching their names to papers on which they have made negligible contributions. We know that such people exist. But how will the panel decide what contribution is significant and what isn’t? What is the criterion?

Take the following example. Suppose the Higgs Boson is discovered at the LHC duringthe REF period. Just about every particle physics group in the UK will have authors on the ensuing paper, but the list is likely to be immensely long and include people who performed many different roles. Who decides where to draw the line on “significance”. I really don’t know the answer to this one, but a possibility might be to found in the use of the textual commentary that accompanies the submission of a research output. At present we are told that this should be used to explain what the author’s contribution to the paper was, but as far as I’m aware there is no mechanism to stop individuals hyping up their involvement.What I mean is I don’t think the panel will check for consistency between commentaries submitted by different people for the same institution.

I’d suggest that consortia  should be required to produce a standard form of words for the textual commentary, which will be used by every individual submitting the given paper and which lists all the other individuals in the UK submitting that paper as one of their four outputs. This will require co-authors to come to an agreement about their relative contributions in advance, which will no doubt lead to a lot of argument, but it seems to me the fairest way to do it. If the collaboration does not produce such an agreement then I suggest that paper be graded “U” throughout the exercise. This idea doesn’t answer the question “what does significant mean?”, but will at least put a stop to the worst of the game-playing that plagued the previous Research Assessment Exercise.

Another aspect of this relates to a question I asked several members of the Physics panel for the 2008 Research Assessment Exercise. Suppose Professor A at Oxbridge University and Dr B from The University of Neasden are co-authors on a paper and both choose to submit it as part of the REF return. Is there a mechanism to check that the grade given to the same piece of work is the same for both institutions? I never got a satisfactory answer in advance of the RAE but afterwards it became clear that the answer was “no”. I think that’s indefensible. I’d advise the panel to identify cases where the same paper is submitted by more than one institution and ensure that the grades they give are consistent.

Finally there’s the biggest problem. What on Earth does a grade like “4* (World Leading)” mean in the first place? This is clearly crucial because almost all the QR funding (in England at any rate) will be allocated to this grade. The percentage of outputs placed in this category varied enormously from field to field in the 2008 RAE and there is very strong evidence that the Physics panel judged much more harshly than the others. I don’t know what went on behind closed doors last time but whatever it was, it turned out to be very detrimental to the health of Physics as a discipline and the low fraction of 4* grades certainly did not present a fair reflection of the UK’s international standing in this area.

Ideally the REF panel could look at papers that were awarded 4* grades last time to see how the scoring went. Unfortunately, however, the previous panel shredded all this information, in order, one suspects, to avoid legal challenges. This more than any other individual act has led to deep suspicions amongs the Physics and Astronomy community about how the exercise was run. If I were in a position of influence I would urge the panel not to destroy the evidence. Most of us are mature enough to take disappointments in good grace as long as we trust the system.  After all, we’re used to unsuccessful grant applications nowadays.

That’s about twice as much as I was planning to write so I’ll end on that, but if anyone else has concrete suggestions on how to repair the REF  please file them through the comments box. They’ll probably be ignored, but you never know. Some members of the panel might take them on board.


19 Responses to “Advice for the REF Panels”

  1. Monica Grady Says:

    I like the idea that consortia decide amongst themselves the relative significance of each author’s contribution to a piece of research. But we only have another 2 years before the cut off date – I suspect it might take longer than 2 years for some consortia to come to agreement.

  2. Chris Brunt Says:

    Regarding ‘authorship’ (though this will be of no use for the REF) – this is clearly out of control lately. At the very least, I’d like to see a two tier system of attribution: “authors” and “contributors”. This is done informally at present by putting a few authors first and the rest as alphabetical list. Why not formalise this? Authors on the front, contributors on a list at the end. This used to be called “acknowledgments” I believe.

    Some kind of contribution statement or even perhaps just a “score” would be nice, but probably impractical.

  3. Indeed – just getting UK only consortia to agree will be difficult, but the problem expands massively when this goes international. Why should Prof Bigname from Germany, PI of an experiment at CERN, waste the time of his collaboration sorting out those details just to satisfy UK bureaucracy when that time could be spent doing science instead?

    • telescoper Says:

      It’s only relevant for the UK contributors, so perhaps it could work without the foreign ones being involved at all?

      • Daniel Mortlock Says:

        But for this to work the international collaboration would still need to decide on what fraction of the authorship goes to the UK researchers. Otherwise the UK members of such collaborations could claim “significant” contributions to papers even if there was no big contribution from anyone in this country, which would be a huge (and obviously unfair) advantage.

      • telescoper Says:

        Quite. My problem is that if an international consortium does not have a UK PI I don’t see how the UK contribution to it can be regarded as “world-leading” anyway; it’s not even “consortium-leading”.

  4. For the REF, which compares many different fields, you also have to cope with different approaches to authorship lists across different disciplines. I was shocked and surprised to find out just last week that the most important author on molecular biology papers is not the first but the last. So the race in that field is to be the last author on a paper, exactly opposite to astrophysics. The first author, to quote my biology colleague, will just be a technician.

  5. telescoper Says:

    Often in astronomy it’s the people who built the instruments that come last. Without them there would be no observations, but their contribution to writing the science papers might indeed be small.

  6. Dave Carter Says:

    My reading from responses given by panel members is:

    1) Author order will not be used to determine the significance of a submitter’s contribution.
    2) If there are four authors or less the contribution is deemed to be significant
    3) If there are five authors or more you need to justify your contribution in the text box.

    I have not asked directly your question of whether there is a mechanism to ensure that authors from different institutions submitting the same paper get the same grade (or unclassified if one has not made a significant contribution). But there should be. I suspect that there will have to be otherwise there will be all sorts of legal challenges.

    One thing which works against the high energy groups more than us is that there is no concept of “institutional significance”, you have to ascribe the paper to a particular author (within a given UoA) and then argue that that author’s contribution is significant. So if you have a really good paper, which has say 8 authors, all within a single institution, you cannot automatically use that paper even as an output of the first author. You still have a case to argue, and 100 words to argue it in.

  7. Alan Heavens Says:

    The ‘all-or-nothing’ credit for a significant paper in which an author makes a marginally significant contribution is clearly problematic, and if the panels stick with it, the criteria for what counts as significant need to be very clear and published in advance. If a contribution-related assessment is allowed, then having different ratings for different authors is clearly fine, but that is not what is proposed – as you say, it is the paper which is judged, so there has to be a mechanism for ensuring that it is judged consistently. Insisting on a role statement should make things fairer within Physics, but remember that other panels do not have the same requirement, and a consequence of insisting that we have extra constraints within Physics may result in different disciplines being treated differently, with potentially unfair outcomes across subjects. The last point I would make is that the disappearance of the narrative – a statement summarising the significance of a paper – will make the panel’s job much harder and prone to error. With the best will in the world it will be very hard for panel members to judge the importance or significance of all papers across the discipline on the basis of the paper alone, bolstered by some potentially unreliable citation information.

  8. telescoper Says:

    A more fundamental question is how we’ve come to a situation where it’s even thinkable that someone could be an author of a serious scientific publication without having made a “significant” contribution to it. The fact this is even an issue demonstrates that the whole system is deeply flawed. Nevertheless, this is the system that the REF Panels have to work with.

    I agree with Alan’s suggestion that the Panel must publish clear instructions on what it is going to judge “significant”. If it doesn’t, departments may end up being punished for submitting works to which they justifiably feel they made a significant contribution but the panel arbirarily decided otherwise.

    • Dave Carter Says:


      If they have written the grant application and managed the grant, and for the reasons you give, I think that is a good rule. Same with telescope time, if you write an application and get the time you should be on a paper which results.

    • Dave Carter Says:

      Its a different thing though, making a contribution which justifies authorship and making a contribution which justifies intellectual leadership, at the level that the paper could not have happened or would have significantly less impact without you. The latter I think is what the REF panels will look for. But if you have a paper with hundreds or even thousands of authors on an LHC result, many of those authors will be technical staff who did not write a word or plot a graph, but spent years building bits of kit which enabled the experiment to succeed. Authorship on a key paper is a very small recompense for years of work for far less than academics get paid, and I would not begrudge them that.

  9. telescoper Says:

    For my own papers Google Scholar and NASA/ADS are fairly similar, and I’ve had comments from colleagues to the same effect.

  10. […] όπως το impactfactorτου περιοδικού και το περιβόητο h-index, καθώς και τον βαθμό στον οποίο επηρεάζουν την […]

  11. […] nos quejamos del ritmo al que métricas tales como el factor de impacto de las revistas y el famoso índice h están aumentando su dominio absoluto sobre la evaluación de la investigación. Y, a pesar de […]

  12. […] regularly moan about the rate at which metrics such as the journal impact factor and the notorious h-index are increasing their stranglehold on the assessment of research. And, yet, as the hypothetical […]

  13. […] regularly moan about the rate at which metrics such as the journal impact factor and the notorious h-index are increasing their stranglehold on the assessment of research. And, yet, as the hypothetical […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: