The Authorized Version

Following on from my previous post about the 2008 Research Assessment Exercise, I’ve been told that Cardiff University’s preferred measure of research activity is not the simple grade point average that I computed there, but an index of research power which is the average multiplied by the number of staff submitted.

Partly out of interest and partly so as not to incur the wrath of the University Thought Police I recalculated the list sorted by the official measure. So here is the authorized version, as sanctioned by the powers that be:

1. University of Cambridge 402.6
2. University of Oxford 371.3
3. Imperial College London 348.7
4. University College London 277.8
5. University of Manchester 215.3
6. University of Durham 191.1
7. University of Edinburgh 169.4
8. University of Warwick 132.6
9. University of Nottingham 126.7
10. University of Glasgow 125.8
11. Queen’s University Belfast 125.0
12. University of Bristol 121.9
13. University of Southampton 120.0
14. University of Birmingham 117.7
15. University of Leicester 114.8
16. University of St Andrews 91.8
17. University of Liverpool 91.7
18. University of Leeds 90.5
19. Queen Mary, University of London 87.5
20. University of Sheffield 86.6
21. Lancaster University 76.6
22. Cardiff University 75.9
23. University of Exeter 75.6
24. University of Strathclyde 74.4
25. University of Hertfordshire 72.8
26. Royal Holloway, University of London 71.3
27. University of Surrey 69.4
28. University of York 67.6
29. University of Bath 57.6
30. University of Sussex 54.0
31. Swansea University 52.9
32. Heriot-Watt University 51.7
33. University of Central Lancashire 51.1
34. Loughborough University 41.9
35. King’s College London 41.8
36. Liverpool John Moores University 39.6
37. Aberystwyth University 35.7
38. Keele University 22.5
39. Armagh Observatory 16.9
40. University of the West of Scotland 6.7
41. University of Kent 6.6
42. University of Brighton 2.3

Well, it’s actually quite surprising how much things change. I don’t think it means very much, but 22nd certainly sounds much better than 35th.

But, being a Newcastle United supporter, I’ve never been a great fan of league tables.

About these ads

30 Responses to “The Authorized Version”

  1. Michael Merrifield Says:

    Yes, Nottingham is also keen on this metric. Very strange, since it gives the impression that huge departments stuffed with hundreds of mediocre researchers are somehow better than average-sized ones with excellent researchers.

  2. telescoper Says:

    I’m not sure of the correct units for “research power”..is it Watts per paper?

  3. Doesn’t really help Cardiff, does it? This is basically a ‘never mind the quality, feel the width’ metric. You could come top, in principle, by having a thousand mediocre researchers.

    The previous metric is also seriously flawed of course, with some universities submitting only a fraction of their academic staff. Is there a version somewhere revealing which physics departments played this game?

  4. telescoper Says:

    It helps in the sense that it’s probably a closer indication of how the money will be allocated to the University and in the end that’s what this is all about.

    The system has many flaws but I’m happy to congratulate other departments being smart enough to play it well. Although we don’t know what’s going to happen with money, the peculiarties of being in Wales mean that the actual funds are pretty much irrelevant for Cardiff. Last RAE Cardiff got much less out of getting a grade 5 than it would have done had it been in England, since HEFCEW wanted to spread funds around in Wales rather than concentrate it here.

    Oh, and if you want to post any further comments please leave your name and a proper email address or I may have to delete your contributions.

  5. Anton Garrett Says:

    The only way to judge research merit is by expert qualitative opinion. But any experts appointed will, by definition, be of the same calibre as the better people they are judging, so there will always be complaints – not infrequently legitimate. Moving to a quantitative syatem is a cop-out designed to deflect criticism onto the system of numerical assignment.

    I quit the system just as Appraisal was coming in. I had options to remain in the university research system but was too picky and I have not regretted the change. It is easy for government and internal university administrators to make small incremental changes to the administrative load of researchers, but the cumulative effect after a decade is that there is little time left in which to get research done.

    Government should be reminded that academic scientists (I am not speaking for rubbish like Media Studies or academic prostitution like Management Studies) who reach a certain level of seniority have successfully jumped over multiple hurdles at which most have fallen, and deserve to be given time (in particular) to get on with their research. As for internal administrators, I suggest that universities take a deep breath and sack a large proportion of them. Long ago, they were hired to take the burden of administration off researchers, but it doesn’t work: by inventing unnecessary tasks, they actually *increase* the load. That is why the administrator-to-doctor ratio is four times higher in the inefficient NHS than in (efficient) private medicine. Doing admin is unavoidable and the optimal solution for dealing it is close to DIY.

    There are deeper problems still. Isupport government funding of a certain amount of science, but on moral grounds university scientists need to be able to give a justification for receiving taxpayers’ money. Also, since World War II there has been an explosion of research (as measured by the volume of published papers), but the proportion of it that is significant has gone down; the great bulk of published science papers are correct but useless. I think it is not coincidence that this has happened at the same time as a great increase in government funding. Yet we want to be able to get intelligent people from poor backgrounds into research – apart from anything else, it is an immoral waste of national resource not to do this.

    I’m not sure what to do next, but recognition of the problem is stage 1…

    Anton

  6. The RAE results are very interesting.

    The position of Cardiff University is somewhat disappointing, as you said, especially when it is remembered that in 2001 the old University of Wales Cardiff scored significantly better, being placed in the top ten British institutions in some league tables. When Cardiff University was formed in 2004, the expectation was that combining the research activities of University of Wales Cardiff with those of the University of Wales College of Medicine would create a new university with a much larger research base. Somehow, adding the two together has given an institution that has been placed much lower in the rankings in the 2008 RAE even when the total research volume is considered (ranked 22nd today).

    The old university structure in Wales had a number of effectively-independent institutions within the framework of the federal University of Wales. This had some merits, though it never really succeeded in fostering a fully cooperative spirit between the individual institutions. The formation of Cardiff University in 2004 outside the University of Wales broke this, and the remaining University of Wales has been fragmenting since then. This is producing a new structure to Welsh higher education in which there will be close to a dozen separate institutions, all competing against each other for students and funding, most of which will not have a critical mass for successful leadership in research.

    There was an expectation that the one potential Britain-wide success of the new Welsh structure might be Cardiff University, because it would have the critical mass required to attract numbers of good students and to develop a strong reputation in research. There was a chance that Wales would have one premiership university (or do I mean Russell Group?) in Cardiff, a few universities in the second division, and the rest struggling as primarily teaching-only local universities. Cardiff University’s lower than expected performance in the RAE will harm its image as a leading British university, and therefore its ability to attract good students. It looks more like a first division university rather than one in the premiership.

    My fear now is that the old Welsh university structure is dead, and that the new structure does not appear to have even one university that can compete with the best of the British universities. There appear to be a lot of rather small, local universities lacking the size to succeed on their own. My preferred option would have been to have kept the University of Wales, but remove the central bureaucratic elements and replace them with inter-institution collaboration to support research and share teaching resources. But nobody would have listened to me. Instead there is fragmentation, parochialism and an inability to compete at the highest level within Britain and internationally.

    One positive issue among this is what you mentioned about the separate funding structure for Wales giving some protection in all this.

    However, it’s still a mess.

  7. The University’s spin on this is that the old University of Wales College of Medicine was much lower down the rankings in 2001 (48th place) and if you add the two together the combined ranking hasn’t changed much.

    However this misses the point that many of the English and Scottish universities have improved while Cardiff appears to have gone backwards, especially in Physics. One of the reasons for that must surely be that HEFCEW did not pass on as much funding for the 2001 Grade 5 as HEFCE did to English universities.

    The overall problem seems to me that the Welsh system is very inward-looking and we are in danger of continuing to lose ground unless we recognise that substantial investment is needed not just to keep departments open but to make them competitive on the world stage. I don’t think the problem lies with small universities as these have been shown to be successful in England, but they have to be led by people prepared to make bold decisions. I think lack of imaginative leadership is the big problem at the moment, especially in Cardiff which has been treading water since 2001 while the rest of the UK has moved on.

  8. Michael Merrifield Says:

    The previous metric is also seriously flawed of course, with some universities submitting only a fraction of their academic staff. Is there a version somewhere revealing which physics departments played this game?

    I believe that there were some legal shinanigans which meant that league tables revealing the fraction of academics returned are not allowed. However, I am happy to report that Nottingham Physics returned all its academic staff in attaining its 2nd place in the country (did I mention…).

  9. Mike,

    Your point reinforces what I was trying to say above. Nottingham entered about 32 academic staff in the 2001 RAE and now that number has grown to about 45. That’s partly a result of staff getting grants and external income, but also a sustained investment by the University of Nottingham, bringing in more staff (and giving them low teaching loads) as well as investing in infrastructure. Remember that quite a chunk of the RAE score is for “research environment” which I’m sure you did well on. These things cost the University a lot of money and have paid off in very spectacular style in terms of the RAE score. I think they were kick-started by the boost in getting a grade 5 in the last RAE. It’s not obvious the HEFCE QR funds will pay back even a fraction of this, but the indirect benefit in terms of publicity and kudos are undeniable.

    Here in Cardiff it’s a different story, although we also got a grade 5 in the last RAE there hasn’t been anything like the injection that Nottingham has received. At least part of the reason for that is we didn’t get anything like the funds from winning grade 5 last time, and partly because Cardiff University’s management team is much less imaginative than Nottingham’s.

    Peter

  10. Adrian Burd Says:

    Anton,
    I agree wholeheartedly with some of what you say. Having recently waded through the American system of getting tenure, I now find myself lumbered with an upper level university administration filled with micro-managers par excellence.

    To my knowledge, there is no equivalent in the US of the RAE and I left the UK too long ago to recall if it was in place then or not. Individual universities here will get qualitative reviews (both internal and external) of their departments, but as was noted above, that has its problems as well. However, in my experience of such things, participants in such review panels are quite good at both giving due credit as well as well-founded criticism of the departments they are reviewing. The resulting documents can (and I stress, can) be used to effectively correct deficiencies in departments as well as leverage to gain a bigger slice of the pie from the university administration (whether in terms of faculty lines or other resources).

    In my opinion, an enlightened university administration realizes that its existence depends (in part) on the effectiveness of its faculty – here in the US that effectiveness is measured both in terms of teaching and research, but both in the common currency of dollars. So an enlightened administration will make decisions that will allow their faculty to spend their time doing their jobs.

    Adrian (who is currently spending too much of his time trying to convince an administrator that he as only one DPhil from Sussex University, and that he does not have a Master of Science degree from Cambridge!)

  11. […] In the Dark A blog about the Universe, and all that surrounds it « The Authorized Version […]

  12. Andrew Liddle Says:

    Peter,

    First, I’m sorry and surprised to see Cardiff so low down the tables, particularly the `grade point average’ table. The contrast between that table and the research power table is very interesting, though being from a university (another of your ex’s!) that was fortunate to score well, yet is very small in size, you’ll understand that I favour the GPA table. It is right though that research power gives a true sense of the scale of the research activity, and hence is a useful measure.

    I am skeptical though that research power is that close an indicator of ultimate funding. I think the funding agencies, particularly HEFCE with its strong drive for selectivity, have got themselves a real problem here. In the previous system, funding per faculty member in a 5* department was substantially more than in a 5, which was itself several times that in a grade 4 department. The GPA by contrast has a dynamical range only from 1.8 to 2.9, with the vast majority in the narrower range 2.3 to 2.9. Any linear function of GPA correlates extremely weakly with past selectivity. As far as I can see the only statistic which has the sort of sensitivity HEFCE likes is the fraction of 4* researchers. Even the fraction of 3*, by contrast, hardly varies across the departments. So, either HEFCE has to abandon strong selectivity (to howls of anguish from former 5* departments), or most of the money will be driven by the 4* fraction alone. Since only approximately 20% of the typical profiles are in 4*, that means 4* has to be worth say 10 times as much as anything else to generate selectivity. So, I suggest an alternative table, the `prospective funding’ table, which is simply the fraction of 4* researchers times the volume submitted. A small constant could be added to account for funding for 3* and 2* at a much lower level. It would be interesting to see if that changes things significantly yet again. I know we at Sussex slide down a bit (though I suppose not as much as in the research power table).

    This would, I’m afraid, be bad news for Cardiff, but most likely HEFCEW (and the Scottish equivalent SFC) will be much less aggressively selective than their English cousin.

    Your point about investment in research, comparing Nottingham to Cardiff, is an interesting one. However, it clearly wouldn’t make sense for *every* university to grow its faculty by 50%, as Nottingham did, since the total pot for the subject nationally is more or less fixed. Nevertheless, `growth’ and `success’ appear to be intertwined in the way people think about university departments (the same sort of thinking which in economics requires the present generation to blow the entire resources of the planet). I wonder if there is another way.

    best,

    Andrew

  13. Andrew

    Looking at the last RAE though I see Cardiff only put in 23 whereas this time it was 32.30 so there has been an increase in staff here too. Clearly that in itself isn’t the deciding factor is what staff, what infrastructure and what time and other resources they are given.

    I think at least part of Cardiff’s problem is that several groups are involved in long-term things (space instruments, gravitational waves) that haven’t paid off yet, hence the low level of output rating. But I very much doubt that’s the whole story. The most disappointing thing is how little 4* we have alongside our decent showing at 3*.

    Clearly there can’t be unlimited growth in numbers but also bear in mind that the previous RAE had 50 submissions for physics and this one only had 42…

    ..which brings me to your other point. One argument that I heard after the last RAE is that research selectivity has by now been achieved so there should not be a further steep weighting like you suggest. I think you could argue that grades 2* to 4* all would have been included in grades 5 and above in the last RAE so do you really want to spread this range out again? You could argue that those with GPA 2.9 are really only just a bit better than those with 2.5 so should just get a bit more funding. Otherwise you have to do down departments that are actually pretty good. Do we really want to end up with only a handful of physics departments?

    I don’t know what the thinking is for HEFCE (or indeed if there is any thinking) – and it’s anyone’s guess what HEFCEW will do – but there is certainly a logic to the idea that the ratings should not be fiddled with too much.

    There is another point that even those departments that have done well should think about as it will affect the entire physics community. This is the fact that the top scoring physics department has a GPA of 2.9 and the average is 2.66. The top chemistry department, for example, has 3.20 and the mean is 2.79. The tables give countless other examples of higher scores for the best departments in other fields. Indeed most fields have a highest score significantly higher than physics. I don’t know whether this is intentional or not, or even whether the physics panel knew that it would come out this way, but it gives a clear message to our political masters: physics in the UK is not as good as many other fields. This is going to make it very tough to make a case for increased investment in this area in the current climate.

    Finally, is there another way? Perhaps. Take research funding away from HEFCE and give it to the research councils in a ring-fenced way. Let them distribute it through research grants at 100% of the full economic cost. Discuss.

    Peter

  14. Michael Merrifield Says:

    So, I suggest an alternative table, the `prospective funding’ table, which is simply the fraction of 4* researchers times the volume submitted.

    Given the ongoing drive toward selectivity, I suspect that even that won’t be enough, and some non-linear factor will be applied.

    but it gives a clear message to our political masters: physics in the UK is not as good as many other fields.

    Actually, Peter, the fraction of top grades is almost identical between physics and chemistry, so the message is that excellence is more uniformly spread across the country in our discipline than theirs

  15. Bryn Jones Says:

    I expect that some of the rationale behind replacing the old 1-5* system with the new 1*, 2*, 3*, 4* system was to bring in greater selectivity, and that political masters may see nothing wrong with targeting funds even more. Whether this would be largely based on the 4* score or would include some 3* contribution will only emerge with time when the methods are published.

    It is interesting to compare physics (+astronomy) at Cardiff University with other Welsh universities. Swansea appears to done better in physics in the 2008 RAE than Cardiff in terms of grade points, although with fewer staff (21 FTE staff vs. 32). This is despite Swansea getting grade 4 in 2001 and Cardiff 5. Both Cardiff and Swansea have done better than Aberystwyth in physics.

    I also note that Oxford and Cambridge managed to enter such large numbers of staff, presumably with the help of the extra funding for college posts, but that Imperial and UCL also managed rather large numbers of staff without the help of a college system.

  16. Mike,

    Looking at the chemistry table I see that about 1/3 of chemistry departments in the UK got a GPA exceeding 2.9 whereas that score would have been top in physics. It all depends how you count things, but I’m not sure politicians are very sophisticated in this respect!

    My only other point is that if they do a very non-linear weighting using numbers then I hope they have the sense not to do it with numbers weighted to the nearest 5%!

    Peter

  17. telescoper Says:

    Bryn

    Yes, it’s interesting about Swansea. They only submitted 11.6 FTE last time but had 20.75 this time. I don’t know exactly what happened with the funding last time but I think their grade 5 was worth a lot more to them that Cardiff’s was, as ours was capped. If HEFCEW had applied the HEFCE formula last time, Cardiff would have had 75% of all the research funding in Wales. Instead it got about 50%. I think it’s fairly clear the capping of Cardiff’s Grade 5 funding must have benefitted other departments that got Grade 5 and perhaps Swansea was among them. Anyway, good luck to them I say. Just because we did badly, doesn’t mean I begrudge the success of people who have done better.

    It’s just like football. I support Newcastle but I accept that there are better teams and I quite enjoy watching the top teams play and don’t get too jealous. Except for Sunderland of course. But then they don’t win that often. And they don’t do physics.

    Peter

  18. Andrew Liddle Says:

    Peter,

    I do feel that research funding in England has been too selective in the past, and that the Scots and Welsh have largely been more sensible, though the dominance of Cardiff is problematic for the latter. Nevertheless, to abandon such strong selectivity now, in the likely absence of a funding injection, would imply a significant transfer of funds from the former 5* departments to the others. Of course it would also mean that the highest ranked departments this time around will benefit less than they had hoped. That wouldn’t bother me, but expect an outcry from those who would be affected, who are of course amongst the most influential in the community.

    It was interesting to learn that Cardiff had in fact increased its numbers quite significantly. We at Sussex by contrast stayed at about 20, more or less unchanged from last time (incidentally submitting 20 out of a total of 21 eligible staff). Have you or anyone else seen any comparison of the total number of physics staff submitted between 2001 and 2008 (I know that over all subjects the number is 15% — it would be interesting to know if the physics number shared or even exceeded that growth).

    best,

    Andrew

  19. Michael Merrifield Says:

    I’m not sure of the correct units for “research power”..is it Watts per paper?

    Well, since it is at least in part a measure of how much dead wood there is in a department, I would be inclined to go for that fine old English unit, the faggot, but in these politically correct times it might be misconstrued.

  20. telescoper Says:

    Well, that says it all.

    And a Merry Xmas to you too.

  21. […] shortly going to be climbing aboard the Deadwood Stage (with some other faggots) in order to spend Christmas with my friends in the  […]

  22. telescoper Says:

    In view of Mike Merrifield’s offensive remarks, I have blocked him from making further posts on this blog.

  23. Unfortunately, this block was imposed before I was able to apologize for the offense that my comment had caused. I will desist from any further postings on this blog, but would appreciate the courtesy of being allowed to convey my apologies to any I may have offended.

    The comment was intended to be a lighthearted, if somewhat risque, play on words, which anyone who knows Peter Coles will know is something that he has been known to indulge in himself. It was not a dig at anyone, irrespective of their sexuality, and, as I am sure anyone who knows me will confirm, I am not exactly the world’s greatest homophobe.

    I can only apologize that the remark not come across as intended.

    Mike M.

  24. telescoper Says:

    I have decided unblock the above “apology” from Mike Merrifield.

    Readers of this blog can decide for themselves whether they believe this is sincere and who it is who has grounds to complain about lack of “courtesy”. I know what I think.

    I hope this will be the end of this matter.

    • telescoper Says:

      The Moving Finger writes; and, having writ,
      Moves on; nor all your Piety nor Wit
      Shall lure it back to cancel half a Line,
      Nor all your Tears wash out a Word of it.

  25. […] some of which relate to comments posted on my previous items about the RAE results (here and here) until I terminated the […]

  26. […] gave the results of the latest Research Assessment Exercise (which I’ve blogged about here, there and everywhere). To my dismay he announced that HEFCW are indeed going to use the 0:1:3:7 weighting […]

  27. […] think I’ve made it clear (here, here, here, here and here) that I think the RAE was a bit of a botch generally and that Physics was particularly […]

  28. […] In this first post, he listed straight weighted mean scores (in which Cardiff came 35th). In a second post, he introduced “research power”, meaning volume times score, which brought Cardiff up […]

  29. […] blogged about the RAE results before: here, there, elsewhere, et cetera and passim. Andy Lawrence (e-astronomer) has now written a blog post about the latest […]

Comments are closed.

Follow

Get every new post delivered to your Inbox.

Join 3,792 other followers

%d bloggers like this: