Archive for RAE

Follow My Leader

Posted in Education, Science Politics with tags , , , on February 2, 2011 by telescoper

News broke today about  the dreaded “Letter from HEFCE” to heads of English higher education institutions informing them of their funding allocations for the near future. The cuts outlined in the funding letter were not entirely unexpected, but have nevertheless generated quite a lot of reaction – see, e.g., here. For those of you who can’t be bothered to read the Circular Letter, it is of the form:

Dear Vice-Chancellor or Principal,

You’re screwed.

Yours Sincerely,


One of the ramifications of the decisions made by the Higher Education Funding Council for England (for that is HEFCE) is that funding will be further cut for research rated a “mere” 2* in the 2008 Research Assessment Exercise. For those of you who prefer meaningless words to meaningless numbers-with-asterisks, 2* is translated in the HEFCE dictionary as “internationally recognised”. Obviously a waste of money then. By 2013 the only research that will be funded at all under the QR formula will be 3* (“internationally excellent) and 4* (world leading). Of course “world leading” won’t include “nationally leading”, as by then there won’t be anyone in England left to follow the survivors; everyone left after the cull will be doing their best to be a  “leader” in order to avoid being labelled a “follower”,  a crime for which the punishment is, apparently, death.

I only hope that this mad policy dismantling the Great Pyramind of Research from the bottom upwards isn’t pursued here in Wales too, otherwise I’ll soon be sitting next to the radiator in the local Wetherspoons drinking cheap beer with the rest of the prematurely terminated. For the time being, until HEFCW decides what do do, we in Wales have a stay of execution.

All this reminds me of a story I heard at a conference dinner years ago. I happened to be sitting next to someone Mrs X – the spouse of a physicist who shall remain nameless – who had for a time worked on undergraduate admissions for a Humanities Department of a  big American Ivy League university. One application for a place had been accompanied by a lukewarm recommendation from a high school teacher, containing a statement along the lines of John is definitely a follower rather than a leader. On this slim evidence of potential, my dining companion Mrs X decided to offer little Johnny a place.

Bewildered to hear that John had been admitted, the aforementioned teacher telephoned the Department and was put through to Mrs X. “But I thought I made it clear in my reference what I thought! Was the decision to let him in some sort of administrative error?”

“No,” said Mrs X.

“Let me explain. Every year this Department takes in about 500 undergraduate students. Virtually every  recommendation letter says this student is a leader rather than a follower. In the light of this huge number of leaders, I feel it is my duty every year to ensure that the Department contains at least one follower.”


Unravelling Cable

Posted in Politics, Science Politics with tags , , , , , , , on September 8, 2010 by telescoper

I woke up this morning with the Vince Cable Blues, owing to an item on the BBC News concerning a speech by the Secretary of State for Business, Innovation and Skills that clearly signals that the forthcoming Comprehensive Spending Review will entail big cuts to the UK’s science budget.

It was a depressing way to start the day, but I for one wasn’t particularly surprised by the news. We all know big cuts are coming, the only remaining questions are “how big?” and “where?”. However, when the text of the speech was released, I was shocked by what it revealed about the Secretary of State’s grasp of his brief.  Read it for yourself and see if you agree with me.

Vince Cable: Out of his Depth

Of course there are the obligatory  platitudes about the quality of the UK’s scientific research, a lot of flannel about the importance of “blue skies” thinking, before he settles on the utilitarian line favoured by the Treasury mandarins who no doubt wrote his speech for him: greater concentration of research funding into areas that are “theoretically outstanding” (judged how?) or “commercially useful” (when?). In fact one wonders what the point of this speech was, as it said very little that was specific except that the government is going to cut science. We knew that already.

For what it’s worth I’ll repeat my own view that “commercially useful” research should not be funded by the taxpayer through research grants. If it’s going to pay off in the short term it should be funded by private investors or venture capitalists of some sort. Dragon’s Den, even. When the public purse is so heavily constrained, it should only be asked to fund those things that can’t in practice be funded any other way. That means long-term, speculative, curiosity driven research. You know, science.

So was Cable’s speech was feeble-minded, riddled with clichés, and totally lacking in depth or detail? Yes.  Was it surprising? No.

What was surprising, at least to me, is Cable’s deliberate use of spurious numbers to back up his argument. For example,

Its is worth noting in the last RAE 54 per cent of submitted work was defined as world class and that is the area where funding should be concentrated.

This appears to be what Cable  was referring to when he stated on the BBC Radio 4 Today Programme that  “45% of research grants were not of excellent standard”.

For one thing, there’s a difference between a research grant and the money allocated by HEFCE through the Research Assessment Exercise (RAE); more about that in a moment. Moreover, at least in England,  RAE funding is only allocated to grades 3* and 4* anyway, so the concentration he talks about is already happening. The comment is made all the more meaningless, however, because the 54% was actually imposed on the assessment panels anyway; they were told to match the outcome of their deliberations to a target profile. The figure quoted is therefore hardly an objective measure of the quality of scientific research in the UK.

When it comes to research grants – usually obtained from one of the Research Councils, such as the Science and Technology Facilities Council (STFC) scientists apply for funding and their proposals are assessed by panels. In the case of STFC I can assure every one that the only proposals funded are those graded excellent, and there isn’t anything like enough money to fund all the proposals graded that way. Further cuts will simply mean that  even more excellent research will have to be scrapped, and even more excellent scientists will  go abroad.

This basic misunderstanding convinces me that Vince Cable is completely out of his depth in this job. That’s very unfortunate because it means he will probably be susceptible to manipulation by the dark side (i.e. the anti-science lobby in Whitehall). Already  someone – most likely a Civil Service mandarin with an axe to grind – seems to have  duped him into thinking that 45% of  taxpayer’s money funds mediocre research. What with him already singing so enthusiastically from the Treasury hymn sheet, I fear they have got him exactly where they want him. Rarely has a new arrival in the Whitehall jungle gone native so quickly.

Another remark of his that was quoted today is that “the bar will have to be raised somewhat” in terms of  science funding.  At the next General Election I hope the British people, especially those foolish enough to opt for the Liberal Democrats last time, will “raise the bar” when it comes to deciding who is worthy of their vote. I’m sure of one thing, though. The fraction of British politicians who are “mediocre” is an awful lot higher than 45%.



Posted in Science Politics with tags , , , on September 28, 2009 by telescoper

No sooner has the dust settled on the  2008 Research Assessment Exercise (RAE) when the Higher Education Funding Council for England (HEFCE) has tabled its proposals for a new system called the Research Excellence Framework (REF) in a 56-page consultation document that you can download and peruse at your leisure.

I won’t try to give a complete account of the new system except to say that apart from the change of acronym there won’t be much different. Many of us hoped that the new framework would involve a lighter touch than the RAE, so we could actually get on with research instead of filling in forms all our lives. Fat chance. You can call me cynical if you like, but I think it’s obvious that once you set up a monstrous bureaucratical nightmare like the RAE it is almost impossible to kill it off. Things like this gather their own momentum and become completely self-serving. The apparatus of research assessment no longer exists to fulfil a particular purpose. It exists because it exists.

It might be useful however to summarise the main changes:

  1. The number of Units of Assessment and sub-panels is to be reduced from 67 to 30 and the number of main assessment panels from 15 to 4. This move is bound to prove controversial as it will clearly reduce the number of specialists involved in the quality appraisal side of things. However, the last RAE produced clear anomalies in the assessment carried out by different panels: physics overall did very poorly compared to other disciplines, for example. Having fewer panels might make it easier to calibrate different subjects. Might.
  2. In REF the overall assessments are going to be based on three elements: research output (60%); impact (25%); and environment (15%). In the last RAE each panel was free to vary the relative contribution of different components to the overall score. Although the “research output” category is similar to the last RAE, it is now proposed to include citation measures in the overall assessment. Officially, that is. It’s an open secret that panel members did look at citations last time anyway.  Citation impact will however be used only for certain science and engineering subjects.  “Impact” is a new element and its introduction is  in line with the government’s agenda to pump research funds into things which will generate wealth, so this measure will probably shaft fundamental physics. “Environment” includes things like postgraduate numbers, research funding and the like; this is also similar to the RAE.
  3. A roughly similar number of experts will be involved as in RAE 2008 – so it will be similarly expensive to run.
  4. The consultation document asks whether the number of outputs submitted per person should be reduced from four to three, and also whether “substantive outputs” (whatever they are) should be “double-weighted”.
  5. The results will be presented in terms of “profiles” as in 2008, with the percentage of activity at each level being given.
  6. The consultation also suggests honing the description of “world-leading” (4*) and “internationally excellent” (3*) to achieve greater discrimination at the top end of the scale. This is deeply worrying, as well as completely absurd. The last RAE applied a steeply rising funding formula to the scores so that 4*:3*:2*:1* was weighted 7:3:1:0. However the fraction of  work in each category is subject to considerable uncertainty, amplified by the strong weighting.  If the categories are divided further then I can see an even steeper weighting emerging, with the likely outcome that small variations in the (subjective) assessment will lead to drastic variations in funding. Among the inevitable consequences of this will be that  some excellent research will lose out.

No doubt university administrators across the United Kingdom will already be plotting how best to play the new system. I think we need to remember, though, that deep cuts in public spending have been promised by both major political parties and there is a general election due next year. I can see the overall  budget for university research being slashed so we’ll be fighting for shares of a shrinking pot. Killing off the bureaucracy would save money, but somehow I doubt that will be on the agenda.

Post Mortem

Posted in Science Politics with tags , , , , on April 6, 2009 by telescoper

Finally the full details of the Physics panel’s deliberations during the 2008 Research Assessment Exercise have been published in the form of sub-profiles, showing the breakdown of the overall scores into various components, including the rating attached to “outputs” (i.e. papers), “environment” and “esteem”; for the jargon see the RAE guidelines for submissions.

 I’ve blogged about the RAE results before: here, there, elsewhere, et cetera and passim. Andy Lawrence (e-astronomer) has now written a blog post about the latest publications from HEFCE  (commenting on the Cardiff situation with a generosity that contrasts with the offensive attitude displayed by one of my former colleagues).  Andy has also produced a graph which makes for very interesting reading:


I’ve used my meagre graphical skills to indicate the location of Cardiff on the figure between the thick solid lines. Note the enormous gap between the panel’s assessment of our outputs (2.22) compared to the score for esteem (2.74).

I’ve mentioned before that apparently not a single one of the papers submitted by Cardiff’s excellent Astronomy Instrumentation Group was graded as 4* (world leading). Among the papers submitted by this group were several highly cited ones relating to an important Cosmic Microwave Background experiment called BOOMERANG. The panel probably judged that Cardiff hadn’t played a sufficiently prominent role in this collaboration to merit a 4*, which seems to be a completely perverse conclusion. The experiment wouldn’t have been possible at all without the Cardiff group.

Notwithstanding my disgruntlement at the particularly and peculiarly harsh assessment of Cardiff’s physics submission, there is also an indication of a more general problem. Notice how at the top right, a large number of departments has an output score seriously lagging their other score (by about 0.4 or more).

The counterexample to this trend is Loughborough, which has a very small but clearly good research activity in physics, and which scored 2.66 on its outputs but only 1.1 on environment. They are easily identified on the graph as an extreme outlier below the general trend.

Although there is no reason to expect a perfect correlation between the different elements of the overall assessment, it looks to me like the Physics panel decided to let the output score for the strong departments saturate at a level of about 2.8 whereas other panels were much more generous.

Why did they do this?

Answers on a postcard (or, better, via the comments box), please.