Archive for Research Assessment Exercise

Hard Cash

Posted in Science Politics, Uncategorized with tags , , on March 19, 2009 by telescoper

The Higher Education Funding Council for Wales (HEFCW) has finally announced its cash allocations for Welsh Universities over the period 2009-10. The settlement of English Universities (produced by HEFCE) has been public for quite a while already.

On the back of a poor showing in the Research Assessment Exercise (RAE) by Cardiff University we were all braced for a cut in our recurrent grant, which has indeed turned out to be the case. Our total grant for teaching and research has been cut in cash terms by about 1.3% with most of the hit coming in the QR money that was allocated according to the RAE. This cut amounts to losing about £2M from the University’s budget and, including inflation, is more like a 3% cut in real terms.

That sounds bad enough (even the fact that there is a minus sign is pretty poor), but there are exacerbating factors on top. First, the National Pay Agreement has given University staff large pay rises over the past year or so. Given the large fraction of a University’s budget that goes on salaries, this means that a positive change in the grant would have been required to keep pace with the increased cost of staff wages. I’m ignoring other sources of income, of course, such as external research grants and endowments but the latter are less important to us in Cardiff than they are, for example, in Oxbridge. Moreover, the recent dire performance of the various University pension schemes has led to the proposal – virtually certain to be agreed – that the employers’ contributions should rise by 2%. This also has a big effect on the University’s budget.

The particular implications of all this for the School of Physics & Astronomy are yet to be worked out in detail, but a safe working assumption is an effective cut in our own budget of about 10%. Unless we can drastically increase our external income then some of our planned activity will have to be curtailed. With STFC having a budget crisis of its own, there seems little prospect of increasing our income from that source so it looks like we’re in for a challenging time.

There were winners in Wales, notably Swansea which has enjoyed a cash increase of about 10%, and some even bigger losers than Cardiff such as Lampeter, already a struggling institution, which has to endure a cut of 9% in its HEFCW grant.

The funding allocations for English Universities have been handled a bit differently to Wales, partly by the introduction of transitional relief to assuage the pain of some large Universities who would have suffered large drops in grant. HEFCE also ring-fenced funding for Science Technology and Medicine (STEM) subjects which helped out places like Imperial College, who would otherwise have had a cut; as it is, their allocation is up by 0.1% in cash. There was no attempt by HEFCW to implement this type of damage limitation, although it did put some extra money into STEM subjects from “other resources”.

It’s interesting to note that Cardiff’s share of the QR funds is actually steady at about 50% which is roughly where as a result of the previous exercise. Application of the English formula in 2001 would have given Cardiff 75% of the QR funding in Wales, which was decided to be politically unacceptable so it was capped at 50%. I think HEFCW used the English formula this time because it kept Cardiff at the level HEFCW wanted it at…

Furthermore the settlement for England as a whole is a tad more generous than Wales. The overall cash settlement for Welsh Universities is up by about 1.66% over last year, whereas that for England is up by 4.1%. The origin of the difference is in the QR funds which in England are up by 7.7% in cash terms but rise by a much lower amount in Wales. This isn’t HEFCW’s fault of course: it has to work with the funds allocated to it by the Welsh Assembly.

Among the English Universities to have done well overall are two that I used to work at. The University of Nottingham has a total grant that has increased by about 9.6% and Queen Mary has trumped that with 10.4%. However, another of my previous haunts, the University of Sussex is one of the few English institutions to have a cash cut like Cardiff’s. Their total grant is cut by 1.4%, which is a tough deal for them. I think the ring-fencing of STEM subjects probably hasn’t helped Sussex as much as some other institutions, as its traditional research strengths are in Arts and Humanities. The biggest loser in England is the troubled Thames Valley University, which has a cash cut of 11.7%. Ouch!

I think I’ve made it clear (here, here, here, here and here) that I think the RAE was a bit of a botch generally and that Physics was particularly badly done by. The outcome has certainly hit Cardiff School of Physics & Astronomy hard. I still can’t understand why our research was rated so poorly. Nature papers with over a thousand citations were not graded 4* by the panel, or at least not when submitted from Cardiff.

When I moved here, I had dreams of building up a nice little cosmology group but it looks like there’s not much chance of this happening, unless we find some way of getting some more money into Welsh physics. Welsh University Physics Alliance anyone?

But the cards have now been dealt. At least we know what sort of hand we’ve got. Now we have to get on playing it as best we can.

A Welsh Affair

Posted in Science Politics with tags , , , , on February 26, 2009 by telescoper

Today I had the “pleasure” of attending a day-long conference called Funding, Risk and Innovation: Wales’s Engagement with Science Policy organized by the Institute of Welsh Affairs and sponsored by, amongst other organizations, the Institute of Physics. I had hoped that this would give me an insight into the landscape of Welsh science politics which might bear fruit in the future. As if.

Unfortunately, but alas predictably, there wasn’t much of interest. I think the first presentation of the day perfectly  illustrated the whole problem. Opening the batting was Ieuan Wyn Jones, Deputy First Minister and Minister for Economic Development, from the Welsh Assembly Government or WAG.  He gave a motherhood-and-apple-pie talk about how important science was to the future of Wales, took a few questions and then left. Those of us scientists who had gone to the meeting hoping for engagement between  politicians and scientists were left to discuss things between ourselves. Hardly the point.

Next was Phil Gummett, Chief Executive of HEFCW who gave the results of the latest Research Assessment Exercise (which I’ve blogged about here, there and everywhere). To my dismay he announced that HEFCW are indeed going to use the 0:1:3:7 weighting formula adopted by HEFCE, but has found a bit more cash which it will add to the pot of money allocated to 4*. However, unlike in the case of English universities, HEFCW is not going to apply any protection to STEM subjects (Science, Technology & Medicine). In the case of my own department at Cardiff University, which got a very low assessment  of 4* research, this is very bad news.

When I got home this evening I read the same news in the Times Higher. I could have found this out without wasting a day sitting  in a ghastly conference room in the soulless Cardiff Novotel. Still, the lunch wasn’t bad.

Phil Gummett struck me as quite a reasonable chap who is trying to do the right thing, but whose hands are tied by the Welsh Assembly which has decided that Higher Education in Wales is not as high a priority as Further Education, with the result that the funds available to HEFCW for research is less than it would be for English universities. University STEM departments in Wales altogether receive about £10M less from HEFCW than they would get from HEFCE if they were in England. For physics, this will probably get worse after the 2008 RAE.

The reason for this pessimism is that, as I’ve noted before, Physics did rather badly in the RAE compared to other discplines, with a significantly lower fraction of work assessed at 4* (world-leading). Since the funding formula is heavily weighted by the 4* element, physics will suffer relative to other disciplines. HEFCW will not attempt to correct this. I think the Chair of the Physics panel, Sir John Pendry, must shoulder at least some of the blame for the gross anomaly that this represents.

It remains to be seen what happens to physics nationally, but I fear the RAE may undo a decade of very effective positive campaigning about the importance of physics. I’ve already heard from various Heads of Physics departments around the country (even those who have done well in the RAE)  who have been asked by their Vice-Chancellors why they have done so much worse than other disciplines.

The final thing he said was that HEFCW would make its allocations as block grants to the Universities concerned and that they should make their own decisions as to how to allocate the funds to the departments. This sort of thing always annoys me. It’s admitting that the formula is probably stupid, so passing the buck to the institutions to sort out the mess themselves.

I spoke to a nice lady from Cardiff University’s planning department in the afternoon who said that they weren’t sure how they were going to allocate funds to Schools after the HEFCW grant was announced, and that the University as a whole was probably going to lose out in research funds, despite having 54% of all the 4* research in Wales.

The big problem is the funding gap caused by the WAG’s policy. Devolution has had a negative effect in science funding in Wales, while in Scotland it has had the opposite effect. The Scottish parliament seems much more interested in science than does the Welsh Assembly. Indeed, per capita, Scottish Universities have a much heavier level of research investment even than those in England, which in turn are much higher than in Wales.

EPSRC‘ recently allocated £82M to UK universities to fund  doctoral training centres. In all, it allocated grants to 45 universities. Wales has 5% of the UK population, but not a single grant went to a Welsh university. Of the 1200 or so students these centres will train, not a single one will be in Wales.  I can’t believe the Scottish assembly would have let such an outcome happen in Scotland.

Further strangulation of research funds is inevitable unless the WAG is persuaded to change its mind about the importance of science. But if the politicians don’t stay to listen to the arguments, how will this happen?

Over lunch I chatted to various physicists from Swansea University. Several of them had come to the meeting, but I was the only representative from Cardiff. There was a strong steer from the RAE panel for physics in terms of closer collaboration so we chatted a bit about possibilities for that. I think the consensus was that we’re probably going to be bounced down the road anyway so the best way forward would be to come up with a plan of our own instead of having someone else’s.

I promise not to mention the RAE again, until the final allocations are published in April!

Physics Funding by Numbers

Posted in Science Politics with tags , , , , , on January 29, 2009 by telescoper

I just read today that HEFCE has decided on the way funds will be allocated for research following the 2008 Research Assessment Exercise. I have blogged about this previously (here, there and elsewhere), but to give you a quick reminder the exercise basically graded all research in UK universities on a scale from 4* (world-leading) to 1* (nationally recognized), producing for each department a profile giving the fraction of research in each category.

HEFCE has decided that English universities will be funded according to a formula that includes everything from 2* up to 4* but with a weighting 1:3:7.  Those graded 1* and unclassified get no funding at all. How they arrived at this formula is anyone’s guess. Personally I think it’s a bit harsh on 2* which is supposed to be internationally recognized research, but there you go.

Assuming there is also a multiplier for volume (i.e. the number of people submitted) we can now easily produce another version of the physics research league table which reveals the relative amount of money each will get. I don’t know the overall normalisation, of course.

The table shows the number of staff submitted (second column) and the overall fundability factor based on a 7:3:1 weighting of the published profile multiplied by the figure in column 2. This is like the “research power” table I showed here, only with a different and much steeper weighting (7,3,1,0) versus (4,3,2,1).

1. University of Cambridge 141.25 459.1
2. University of Oxford 140.10 392.3
3. Imperial College London 126.80 380.4
4. University College London 101.03 298.0
5. University of Manchester 82.80 227.7
6. University of Durham 69.50 205.0
7. University of Edinburgh 60.50 184.5
8. University of Nottingham 44.45 144.5
9. University of Glasgow 45.75 135.0
10. University of Warwick 51.00 130.1
11. University of Bristol 46.00 128.8
12. University of Birmingham 43.60 126.4
13. University of Southampton 45.30 120.0
14. Queen’s University Belfast 50.00 115.0
15. University of Leicester 45.00 114.8
16. University of St Andrews 32.20 104.7
17. University of Liverpool 34.60 96.9
18. University of Sheffield 31.50 92.9
19. University of Leeds 35.50 88.8
20. Lancaster University 26.40 88.4
21. Queen Mary, University of London 34.98 85.7
22. University of Exeter 28.00 77.0
23. University of Hertfordshire 28.00 72.8
24. University of York 26.00 67.6
25. Royal Holloway, University of London 27.96 67.1
26. University of Surrey 27.20 65.3
27. Cardiff University 32.30 64.6
28. University of Bath 20.20 63.6
29. University of Strathclyde 31.67 60.2
30. University of Sussex 20.00 55.0
31. Heriot-Watt University 19.50 51.7
32. Swansea University 20.75 48.8
33. Loughborough University 17.10 41.9
34. University of Central Lancashire 22.20 41.1
35. King’s College London 16.40 38.5
36. Liverpool John Moores University 16.50 35.5
37. Aberystwyth University 18.33 23.8
38. Keele University 10.00 18.0
39. Armagh Observatory 7.50 13.1
40. University of Kent 3.00 4.5
41. University of the West of Scotland 3.70 4.1
42. University of Brighton 1.00 1.8

It looks to me that the fraction of funds going to the big three at the top will probably be reduced quite significantly, although apparently there are  funds set aside to smooth over any catastrophic changes. I’d hazard a guess that things won’t change much for those in the middle.

I’ve left the Welsh and Scottish universities in the list for comparison, but there is no guarantee that HEFCW and SFC will use the same formula for Wales and Scotland as HEFCE did for England. I have no idea what is going to happen to Cardiff University’s funding at the moment.

Another bit of news worthing putting in here is that HEFCE has protected funding for STEM subjects (Science, Technology and Medicine) so that the apparently poor showing of some science subjects (especially physics) compared to, e.g., Economics will not necessarily mean that physics as a whole will suffer. How this works out in practice remains to be seen.

Apparently also the detailed breakdowns of how the final profiles were reached will go public soon. That will make for some interesting reading, although apparently everything relating to individual researchers will be shredded to prevent problems with the data protection act.

The Authorized Version

Posted in Science Politics with tags , on December 18, 2008 by telescoper

Following on from my previous post about the 2008 Research Assessment Exercise, I’ve been told that Cardiff University’s preferred measure of research activity is not the simple grade point average that I computed there, but an index of research power which is the average multiplied by the number of staff submitted.

Partly out of interest and partly so as not to incur the wrath of the University Thought Police I recalculated the list sorted by the official measure. So here is the authorized version, as sanctioned by the powers that be:

1. University of Cambridge 402.6
2. University of Oxford 371.3
3. Imperial College London 348.7
4. University College London 277.8
5. University of Manchester 215.3
6. University of Durham 191.1
7. University of Edinburgh 169.4
8. University of Warwick 132.6
9. University of Nottingham 126.7
10. University of Glasgow 125.8
11. Queen’s University Belfast 125.0
12. University of Bristol 121.9
13. University of Southampton 120.0
14. University of Birmingham 117.7
15. University of Leicester 114.8
16. University of St Andrews 91.8
17. University of Liverpool 91.7
18. University of Leeds 90.5
19. Queen Mary, University of London 87.5
20. University of Sheffield 86.6
21. Lancaster University 76.6
22. Cardiff University 75.9
23. University of Exeter 75.6
24. University of Strathclyde 74.4
25. University of Hertfordshire 72.8
26. Royal Holloway, University of London 71.3
27. University of Surrey 69.4
28. University of York 67.6
29. University of Bath 57.6
30. University of Sussex 54.0
31. Swansea University 52.9
32. Heriot-Watt University 51.7
33. University of Central Lancashire 51.1
34. Loughborough University 41.9
35. King’s College London 41.8
36. Liverpool John Moores University 39.6
37. Aberystwyth University 35.7
38. Keele University 22.5
39. Armagh Observatory 16.9
40. University of the West of Scotland 6.7
41. University of Kent 6.6
42. University of Brighton 2.3

Well, it’s actually quite surprising how much things change. I don’t think it means very much, but 22nd certainly sounds much better than 35th.

But, being a Newcastle United supporter, I’ve never been a great fan of league tables.

Res Judicata

Posted in Science Politics with tags , , , , on December 18, 2008 by telescoper

Today is the day people working in British Universities have waited for in a mixture of hope and apprehension for several years. The results of the 2008 Research Assessment Exercise (RAE) were published at 0.01am GMT today (18th December).

I had a look just after midnight and the webserver crashed, but only for a few minutes and I soon got back in and found the bad news. The relevant one for me as an astrophysicist is the table for Unit of Assessment 19 which is Physics & Astronomy. Results are given as a list of numbers, consisting of the number of staff entered (not necessarily an integer, for accounting reasons) followed by the percentage of work judged by the panel to be in each of four categories explained in the following excerpt from the RAE website

The quality profiles displayed on this website are the results of the 2008 Research Assessment Exercise (RAE2008), the sixth assessment in this current format of the quality of research conducted in UK Higher Education Institutions (HEIs). The UK funding bodies for England, Northern Ireland, Scotland and Wales will use the RAE2008 results to distribute funding for research from 2009-10.

The results follow an expert review process conducted by assessment panels throughout 2008. Research in all subjects was assessed against agreed quality standards within a common framework that recognised appropriate variations between subjects in terms of both the research submitted and the assessment criteria.

Submissions were made in a standard form that included both quantitative and descriptive elements. Full details of the contents of, and arrangements for making, submissions were published in ‘Guidance on submissions‘ (RAE 03/2005).

The RAE quality profiles present in blocks of 5% the proportion of each submission judged by the panels to have met each of the quality levels defined below. Work that fell below national quality or was not recognised as research was unclassified.

4* Quality that is world-leading in terms of originality, significance and rigour.
3* Quality that is internationally excellent in terms of originality, significance and rigour but which nonetheless falls short of the highest standards of excellence.
2* Quality that is recognised internationally in terms of originality, significance and rigour.
1* Quality that is recognised nationally in terms of originality, significance and rigour.
Unclassified Quality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment.

The ‘international’ criterion equates to a level of excellence that it was reasonable to expect for the UOA, even though there may be no current examples of such a level in the UK or elsewhere. It should be noted that ‘national’ and ‘international’ refer to standards, not to the nature or geographical scope of particular subjects.

For my own department, the School of Physics & Astronomy, at Cardiff University, I found the following

Cardiff University (32.30) 5 45 30 20

which means that we entered 32.30 people, but only 5% of the work was judged to be at the top level (4*), 45% at 3*, 30% at 2* and 20% at 1*. On their own these figures don’t mean very much but one can do a quick comparison with the rest of the table to see that for us this is an enormous disappointment. We have a much lower fraction of 4* than the majority of departments, and also a significantly higher fraction of 1*. These findings are very worrying.

If I were working an English University with these results I would be very concerned about their financial implications, but it’s a bit more complicated with us being here in Wales. The numbers given in the table are translated into money by the funding councils and Wales has its own one of these (HEFCW, different from the English HEFCE). There are many fewer physics departments in Wales and we’re not competing with the bigger English ones for funding. We don’t yet know how much our research funds will be cut. It might not be as bad as if we were in England, but it’s clearly not good. We won’t know how much dosh will be involved until March 2009. t’s not just a matter of funding, it’s also the national and international perception of the department in the physics community.

I can see there will be a post mortem to find out what went wrong, as most of us were confident of a much better outcome. Perhaps the format of the RAE (focussing on research papers as the measure of output) is not favourable to a department with so many instrument builders in it?

But with the economy in deep recession making further cuts in research funding likely in the future, and our major external funder (STFC) already struggling to make ends meet, this poor showing in the RAE this has cast a gloomy shadow over Christmas.

Of course many places did much better, including my old department at Nottingham which has

University of Nottingham (44.45) 25 40 30 5

which can be interestingly compared with Cambridge, who have

University of Cambridge (141.25) 25 40 30 5

You can see that apart from the different numbers of staff the profile is exactly the same. I’m sure their publicity machine will pick up on this so I won’t be the last to mention it! Well done, Nottingham!

It will be interesting to see what the newspapers make of the new RAE results. They are significantly more complicated than previous versions which just gave a single number. The scope for flexibility in generating league tables is clearly greatly enhanced by this complexity so we can bet the hacks will have a field day. I thought I’d get a headstart by doing a straightforward ranking using a simple weighted average using 4=4*, 3=3*, etc and then sorting them by the average thus obtained:

1. Lancaster University 2.9
2. University of Bath 2.85
3. University of Cambridge 2.85
4. University of Nottingham 2.85
5. University of St Andrews 2.85
6. University of Edinburgh 2.8
7. University of Durham 2.75
8. Imperial College London 2.75
9. University of Sheffield 2.75
10. University College London 2.75
11. University of Glasgow 2.75
12. University of Birmingham 2.7
13. University of Exeter 2.7
14. University of Sussex 2.7
15. University of Bristol 2.65
16. University of Liverpool 2.65
17. University of Oxford 2.65
18. University of Southampton 2.65
19. Heriot-Watt University 2.65
20. University of Hertfordshire 2.6
21. University of Manchester 2.6
22. University of Warwick 2.6
23. University of York 2.6
24. King’s College London 2.55
25. University of Leeds 2.55
26. University of Leicester 2.55
27. Royal Holloway, University of London 2.55
28. University of Surrey 2.55
29. Swansea University 2.55
30. Queen Mary, University of London 2.5
31. Queen’s University Belfast 2.5
32. Loughborough University 2.45
33. Liverpool John Moores University 2.4
34. University of Strathclyde 2.35
35. Cardiff University 2.35
36. University of Brighton 2.3
37. University of Central Lancashire 2.3
38. Keele University 2.25
39. Armagh Observatory 2.25
40. University of Kent 2.2
41. Aberystwyth University 1.95
42. University of the West of Scotland 1.8

So you can see we are languishing at 35th place out of 42.

This is supposed to be the last RAE and we don’t know what is going to replace it. I don’t at all object to the principle that research funding should be peer-assessed but this particular exercise was enormously expensive in the effort spent at Universities preparing for it, not to mention the ridiculous burden placed on the panel of having to read all those papers.