Post Mortem

Finally the full details of the Physics panel’s deliberations during the 2008 Research Assessment Exercise have been published in the form of sub-profiles, showing the breakdown of the overall scores into various components, including the rating attached to “outputs” (i.e. papers), “environment” and “esteem”; for the jargon see the RAE guidelines for submissions.

 I’ve blogged about the RAE results before: here, there, elsewhere, et cetera and passim. Andy Lawrence (e-astronomer) has now written a blog post about the latest publications from HEFCE  (commenting on the Cardiff situation with a generosity that contrasts with the offensive attitude displayed by one of my former colleagues).  Andy has also produced a graph which makes for very interesting reading:


I’ve used my meagre graphical skills to indicate the location of Cardiff on the figure between the thick solid lines. Note the enormous gap between the panel’s assessment of our outputs (2.22) compared to the score for esteem (2.74).

I’ve mentioned before that apparently not a single one of the papers submitted by Cardiff’s excellent Astronomy Instrumentation Group was graded as 4* (world leading). Among the papers submitted by this group were several highly cited ones relating to an important Cosmic Microwave Background experiment called BOOMERANG. The panel probably judged that Cardiff hadn’t played a sufficiently prominent role in this collaboration to merit a 4*, which seems to be a completely perverse conclusion. The experiment wouldn’t have been possible at all without the Cardiff group.

Notwithstanding my disgruntlement at the particularly and peculiarly harsh assessment of Cardiff’s physics submission, there is also an indication of a more general problem. Notice how at the top right, a large number of departments has an output score seriously lagging their other score (by about 0.4 or more).

The counterexample to this trend is Loughborough, which has a very small but clearly good research activity in physics, and which scored 2.66 on its outputs but only 1.1 on environment. They are easily identified on the graph as an extreme outlier below the general trend.

Although there is no reason to expect a perfect correlation between the different elements of the overall assessment, it looks to me like the Physics panel decided to let the output score for the strong departments saturate at a level of about 2.8 whereas other panels were much more generous.

Why did they do this?

Answers on a postcard (or, better, via the comments box), please.

4 Responses to “Post Mortem”

  1. Bryn Jones Says:

    I note that you are confident that the Cardiff instrumentalists did not have any of their research submissions rated 4*, despite the RAE policy of not publishing the results for individual researchers. This is presumably based on some assessment of how the instrmentalists’ submissions would have affected the overall Cardiff physics RAE “research outputs” result.

    It would be very disappointing if this is true. The Cardiff instrumentalists have carried out a lot of research that is internationally significant and this should have been recognised in the RAE.

    I am a little puzzled that the “environment” and “esteem” categories have been broken down into the four grades in the same way as the research outputs were. These two categories considered the overall quality of the departments, rather than of individual researchers, and as such we might have expected they would be expressed as a single statistic for each department (or as results for each of a number of subcategories).

    One question that should be asked in Wales and Scotland is what would happen were the funding that is currently given to the STFC instead devolved. That is, would Scotland and Wales be better off if a proportion of the Treasury funding that now goes to the STFC instead were given to the devolved administrations, and those administrations were responsible for supporting research in the STFC areas and buying into STFC-supported facilities as needed. Devolution of research funding in the STFC areas might allow Wales and Scotland to escape some of the consequences of the STFC funding mess. I’m not advocating this as a policy, just that the question needs to be considered, if for no other reason than to concentrate minds in Swindon.

  2. telescoper Says:


    We know that none of the AIG outputs were 4* because the rather short summary we were given explicitly said so. The only 4* output in our submission was in astronomy and, if memory serves, opto-electronics. The gravitational physics groups also failed to register any 4* no doubt because they are working within international consortia.

    I don’t know how the esteem and environment things work out either.

    I don’t think devolving little pots of money will help Wales and Scotland. The latter actually does very well at getting research council money, in fact proportionately better than English Universities at winning science funding. They would lose out if the pot were divided on a pro rata basis.
    Wales does less well, but the devolved pot would be tiny and likely to be raided by the WAG for other things. There is little or no regional influence
    in research funding decisions (except for the RAE dosh). In think it’s best that good science be the prime factor. However, if the WAG continues to underinvest in science in Wales it will become still harder to get research council funding and the rot will definitely set in.


  3. Bryn Jones Says:

    I see: the RAE feedback was detailed enough in the case of physics/astronomy in Cardiff for some conclusions to be drawn about who achieved what. I suspected that the RAE procedure was devised – with results only being available at a summary level – to prevent questions being raised about detailed panel decisions. The panel has gone beyond that in this case.

    The Cardiff instrumentation group does therefore have good cause to feel they have been treated unfairly.

    My point about devolving research council funding is that it is a question that needs to be considered: it might not help at all, but the issue should be there in the background.

    • telescoper Says:


      They gave us a bland paragraph or so of feedback, but it was sufficient to work out that information.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: