Archive for REF

That Was The REF That Was..

Posted in Finance, Science Politics with tags , , , , , , on December 18, 2014 by telescoper

I feel obliged to comment on the results of the 2014 Research Excellence Framework (REF) that were announced today. Actually, I knew about them yesterday but the news was under embargo until one minute past midnight by which time I was tucked up in bed.

The results for the two Units of Assessment relevant to the School of Mathematical and Physical Sciences are available online here for Mathematical Sciences and here for Physics and Astronomy.

To give some background: the overall REF score for a Department is obtained by adding three different components: outputs (quality of research papers); impact (referrring to the impact beyond academia); and environment (which measures such things as grant income, numbers of PhD students and general infrastructure). These are weighted at 65%, 20% and 15% respectively.

Scores are assigned to these categories, e.g. for submitted outputs (usually four per staff member) on a scale of 4* (world-leading), 3* (internationally excellent), 2* (internationally recognised), 1* (nationally recognised) and unclassified and impact on a scale 4* (outstanding), 3* (very considerable), 2* (considerable), 1* (recognised but modest) and unclassified. Impact cases had to be submitted based on the number of staff submitted: two up to 15 staff, three between 15 and 25 and increasing in a like manner with increasing numbers.

The REF will control the allocation of funding in a manner yet to be decided in detail, but it is generally thought that anything scoring 2* or less will attract no funding (so the phrase “internationally recognised” really means “worthless” in the REF, as does “considerable” when applied to impact). It is also thought likely that funding will be heavily weighted towards 4* , perhaps with a ratio of 9:1 between 4* and 3*.

We knew that this REF would be difficult for the School and our fears were born out for both the Department of Mathematics or the Department of Physics and Astronomy because both departments grew considerably (by about 50%) during the course of 2013, largely in response to increased student numbers. New staff can bring outputs from elsewhere, but not impact. The research underpinning the impact has to have been done by staff working in the institution in question. And therein lies the rub for Sussex…

To take the Department of Physics and Astronomy, as an example, last year we increased staff numbers from about 23 to about 38. But the 15 new staff members could not bring any impact with them. Lacking sufficient impact cases to submit more, we were obliged to restrict our submission to fewer than 25. To make matters worse our impact cases were not graded very highly, with only 13.3% of the submission graded 4* and 13.4% graded 3*.

The outputs from Physics & Astronomy at Sussex were very good, with 93% graded 3* or 4*. That’s a higher fraction than Oxford, Cambridge, Imperial College and UCL in fact, and with a Grade Point Average of 3.10. Most other departments also submitted very good outputs – not surprisingly because the UK is actually pretty good at Physics – so the output scores are very highly bunched and a small difference in GPA means a large number of places in the rankings. The impact scores, however, have a much wider dispersion, with the result that despite the relatively small percentage contribution they have a large effect on overall rankings. As a consequence, overall, Sussex Physics & Astronomy slipped down from 14th in the RAE to 34th place in the REF (based on a Grade Point Average). Disappointing to say the least, but we’re not the only fallers. In the 2008 RAE the top-rated physics department was Lancaster; this time round they are 27th.

I now find myself in a situation eerily reminiscent of that I found myself facing in Cardiff after the 2008 Research Assessment Exercise, the forerunner of the REF. Having been through that experience I’m a hardened to disappointments and at least can take heart from Cardiff’s performance this time round. Spirits were very low there after the RAE, but a thorough post-mortem, astute investment in new research areas, and determined preparations for this REF have paid dividends: they have climbed to 6th place this time round. That gives me the chance not only to congratulate my former colleagues there for their excellent result but also to use them as an example for what we at Sussex have to do for next time. An even more remarkable success story is Strathclyde, 34th in the last RAE and now top of the REF table. Congratulations to them too!

Fortunately our strategy is already in hand. The new staff have already started working towards the next REF (widely thought to be likely to happen in 2020) and we are about to start a brand new research activity in experimental physics next year. We will be in a much better position to generate research impact as we diversify our portfolio so that it is not as strongly dominated by “blue skies” research, such as particle physics and astronomy, for which it is much harder to demonstrate economic impact.

I was fully aware of the challenges facing Physics & Astronomy at Sussex when I moved here in February 2013, but with the REF submission made later the same year there was little I could do to alter the situation. Fortunately the University of Sussex management realises that we have to play a long game in Physics and has been very supportive of our continued strategic growth. The result of the 2014 REF result is a setback but it does demonstrate that the stategy we have already embarked upon is the right one.

Roll on 2020!

The Stifling Effect of REF Impact

Posted in Education, Politics with tags , , , on April 22, 2014 by telescoper

Well, I’m back to civilization (more or less) and with my plan to watch a day of cricket at Sophia Gardens thwarted by the rain I decided to pop into an internet café and do a quick post about one of the rants that has been simmering on the back burner while I’ve been taking a break.

Just before the Easter vacation I had lunch with some colleagues from the Department of Physics & Astronomy at the University of Sussex. One of the things that came up was the changing fortunes of the department. After years of under-investment from the University administration,  it was at one time at such a low ebb  that it was in real danger of being closed down (despite its undoubted strengths in research and teaching).  Fortunately help came in the form of SEPnet, which provided funds to support new initiatives in Physics not only in Sussex but across the South East. Moreover, the University administration had belatedly realized that a huge part of the institutional standing in tables of international research rankings was being generated by the Department of Physics & Astronomy. In the nick of time, the necessary resources were invested and the tide was turned and there has been steady growth in staff and student numbers since.

As Head of the School of Mathematical and Physical Sciences I have had to deal with the budget for the Department of Physics & Astronomy. Just a decade ago very few physics departments in the UK were  financially solvent and most had to rely on generous subsidies from University funds to stay open. Those that did not receive such support were closed down, a fate which Sussex narrowly avoided but which befell, for example, the physics departments at Reading and Newcastle.

As I blogged about previously, the renaissance of Sussex physics seems not to be unique. Admissions to physics departments across the country are growing at a healthy rate, to the extent that new departments are being formed at, e.g. Lincoln and Portsmouth. None of this could have been imagined just ten years ago.

So will this new-found optimism be reflected in the founding of even more new physics departments? One would hope so, as I think it’s a scandal that there are only around 40 UK universities with physics departments. Call me old-fashioned but I think a university without a physics department is not a university at all. Thinking about this over the weekend however I realized that any new physics department is going to have grave problems under the system of allocating research funding known as the Research Excellence Framework.

A large slice (20%) of the funding allocated by the 2014 REF will be based on “Impact” which, roughly speaking, means the effect the research can be demonstrated to have had outside the world of academic research. This isn’t the largest component – 65% is allocated on the basic of the quality of “Outputs” (research papers etc) – but is a big chunk and will probably be very important in determining league table positions. It is probably going to be even larger in future versions of the REF.

Now here’s the rub. When an academic changes institution (as I have recently done, for example) he/she can take his/her outputs to the new institution. Thus, papers I wrote while at Cardiff could be submitted to the REF from Sussex. This is not the case with “impact”. The official guidance on submissions states:

Impact: The sub-panels will assess the ‘reach and significance’ of impacts on the economy, society and/or culture that were underpinned by excellent research conducted in the submitted unit, as well as the submitted unit’s approach to enabling impact from its research. This element will carry a weighting of 20 per cent.

The emphasis is mine.

The period during which the underpinning research must have been published is quite generous in length: 1 January 1993 to 31 December 2013. This is clearly intended to recognize the fact that some research take a long time to generate measurable impact. The problem is that the underpinning research must have been done within the submitting unit; it can’t be brought in from elsewhere. If the unit is new and did not exist for most of this period,then it is much harder to generate impact no matter how brilliant the staff it recruits. Any new departments in physics, or any other subject for that matter, will have to focus on research that can generate impact very rapidly indeed if it is to compete in the next REF, expected in 2018 or thereabouts. That is a powerful disincentive for universities to invest in research that may take many years to come to fruition. Five years is a particularly short time in experimental physics.

 

 

Open Access Repositories should be based on Subject, not Institution

Posted in Open Access with tags , , , , on September 18, 2013 by telescoper

Just when we thought that the powers that be might be starting to see the light on Open Access, round comes another circular from the Higher Education Funding Council for England (HEFCE) that shows that policymakers have an unlimited ability to get things wrong at the most basic level.

The document concerned opens a “Consultation on open access in the post-2014 Research Excellence Framework” by putting forward a number of proposals.

Now it’s depressing enough that the entire document is predicated on the assumption that there will be another Research Excellence Framework, perhaps in 2020. The current REF is such a disaster that one might have hoped somebody up there would have decided that enough is enough. But then we thought that about Research Assessment Exercise in 2008. Even the most pessimistic of us hope that the REF would have a “lighter touch” than the RAE, but as it has turned out it’s many times worse both in the time it has taken to prepare submissions and the ridiculous game-playing and dodgy employment practices that it has encouraged among participation institutions.

I hope there’s still time to drive a stake through the heart of the runaway bureaucracy that keeps imposing this idiocies on us. After all, a new Chief Executive about to take over at HEFCE. Perhaps a new broom will be wielded? I hope against hope.

However, setting all that to one side, I had a look at the proposals for Open Access after 2014 contained in the document. Here is the summary of the proposals:

HEFCE

Aarrgh!

Why on Earth should the proposals favour national institutional repositories over international subject-based ones? A shining example of the latter is the arXiv which has, for Physics and Astronomy, become the basic resource for researchers around the world; it’s a one-stop shop at which one can access research from all around the world. By contrast, having work in the same field stored over a plethora of institutional repositories will serve no useful purpose at all, because UK research will not  treated in the same way as work from other countries and in any case individual repositoes will lead to an absurd level of duplication of infrastructure and other resources. This requirement is particularly indefensible in Physics and Astronomy, as it would require us to duplicate in (probably inferior) institutional repositories what we already do with the arXiv.

The UK Funding Councils need realize that the solution to many of the challenges of Open Access has already been found. In fact, the European Research Council seems to have acknowledged this and is now directly funding the arXiv. The UK Research Councils should be required to construct similar archives for their disciplines. That shouldn’t be difficult, because all the hard work has already been done. There is a working model.

I’ll be responding to the consultation document in no uncertain terms. The Royal Astronomical Society is also collating responses for a collective submission. We have to resist these, and other proposals such as another REF, which are being foisted on us by people who have no idea what they’re doing and no idea what damage they’re causing.

The Dark Side of the REF

Posted in Finance, Science Politics with tags , , , , , , , , on August 8, 2013 by telescoper

There’s a disturbing story in the latest Times Higher which argues that the University of Leicester has apparently reneged on a promise that non-submission to the forthcoming (2014)  Research Excellence Framework (REF) would not have negative career consequences. They have now said that except in exceptional circumstances, non-submitted academics will either be moved to a teaching-only contract (where there is a vacancy and they can demonstrate teaching excellence), or have their performance “managed”, with the threat of sacking if they don’t meet the specified targets.  I’d heard rumours of this on the grapevine (i.e. Twitter) before the Times Higher story was published. It’s very worrying to have it confirmed, as it raises all kinds of questions about what might happen in departments that turn out to have disappointing REF results .

There are (at least) two possible reasons for non-inclusion of the outputs of a researcher and it is important to distinguish between them. One is that the researcher hasn’t enough high-quality outputs to submit. In the absence of individual extenuating circumstances, researchers are expected to submit four “outputs” (in my discipline that means “research papers”) for assessment. That’s a pretty minimal level of productivity, actually;  such a number per year is a reasonable average for an active researcher in my field.  A person employed on a contract that specifies their duties as Teaching and Research may therefore be under-performing  if they can’t produce four papers over the period 2008-2013. I think some form of performance management  may be justifiable in this case, but the primary aim should be to help the individual rather than show them the door. We all have fallow periods in research, and it’s not appropriate to rush to sack anyone who experiences a lean time.   Andrew Wiles would have been considered `inactive’ had there been a REF in 1992 as he hadn’t published anything for years. Then he produced a proof of Fermat’s Last Theorem. Some things just take time.

A second reason for excluding researcher from the REF is that the institution concerned may be making a tactical submission. As the Times Higher article explains:

The memo suggests that academics would be spared repercussions if, among other reasons, the number of individuals submitted is “constrained” by the volume of case studies their department intends to enter to demonstrate research impact.

Institutions must submit one case study for every 10 scholars entered.

Maria Nedeva, professor of science and innovation dynamics and policy at Manchester Business School, said the tactic of deciding how many academics to submit based on impact case study numbers was “rife”.

(Incidentally, the second paragraph is not quite right. The number of case studies required depends on the number of staff submitted as follows: for fewer than 15 staff , TWO case studies;  for 15-24.99 staff it is THREE case studies – and then for each additional ten members of staff entered a further case study is required.)

e case study for every scholars included plus one, i.e. forThe statement at the end of the quote there is in line with my experience too.  The point is that the REF is not just a means of allocating relatively small amounts of so-called `QR’ research funding . Indeed, it remains entirely possible that no funding at all will be allocated following the 2014 exercise. The thinking then is that the number of staff submitted is largely irrelevant; all that will count is league table position.

This by no means the only example of the dangers that lurk when you take league tables too seriously.

If a department is required to submit, say, four impact cases if all staff are included in the REF submission, but only has three viable ones, it would not be unreasonable to submit fewer staff because their overall would be dragged down by a poor impact case even if the output quality of all staff is high.  There will certainly be highly active researchers in UK institutions, including many who hold sizable external research grants, whose outputs are not submitted to the REF. As the article points out, it would be very wrong for managers to penalize scholars who have been excluded because of this sort of game-playing. That’s certainly not going to happen in the School of Mathematical and Physical Sciences at Sussex University.  Not while I’m Head of School, anyway.

Moreover, even researchers whose “outputs” are not selected may still contribute to the “Environment” and/or “Impact” sections so they still, in a very real sense, do participate in their department’s REF submission.

My opinion? All this silliness could easily have been avoided by requiring all staff in all units of assessment to be submitted by all departments. You know, like would have happened if the system were actually designed to identify and reward research excellence. Instead, it’s yet another example of a bureaucratic machine that’s become entirely self-serving. It exists simply because it exists.  Research would be much better off without it.

University Research Funding: Will the Axe Fall on QR?

Posted in Finance, Politics, Science Politics with tags , , , on June 25, 2013 by telescoper

As we tremble in anticipation of this week’s Spending Review, which will determine the budgets for Science and Higher Education in 2015/16, there’s fairly strong evidence that Chancellor of the Exchequer George Osborne is looking to save about £11.5 billion of public spending. Given that funding for some Whitehall departments is ring-fenced there is considerable speculation that the axe will fall heavily on the Department for Business, Innovation and Skills (BIS), which seems likely to have to make over £1 billion of savings.

But where will these savings in the BIS budget be made? The government has made noises that it will protect science funding (at least in cash terms) so big cuts in the larger research council budgets appear unlikely. However, Treasury officials have been rumoured as thinking that the Universities are now “awash with money” and should therefore be cut. On the other hand, incoming for University teaching now largely comes from fees so there’s very little of the HEFCE teaching budget to cut further.

Now here’s the rub. The part of HEFCE’s budget that deals with research amounts to about £1.6 billion per year. This, the so-called `QR’ funding, is currently being distributed to Higher Education Institutions according to the outcome of the 2008 Research Assessment Exercise (RAE). This year we
are preparing submissions to a new system called the Research Excellence Framework (REF), which was always intended to be used to distribute QR funding from 2015/16 onwards. But what if the government decides that the only way to balance the books is to remove the QR funding stream entirely?

The 2012 funding letter from HEFCE states explains that it is distributing

£1,558 million for research. The ring-fenced settlement for science and research means that we will be able to maintain overall funding, in cash terms, until 2014-15.

But this does not include the period covered by the spending review, so it’s perfectly possible that the “ring fence” could be removed, or at least re-interpreted as a result of this spending review.

The government could argue that QR and Research Council grant income correlate so well that there’s no need to continue with the current dual funding system, by which the Research Councils provide grants for specific projects and programmes and the higher education funding bodies provide block grant funding to universities via the QR line. It could also argue that the high fees being charged mean that Universities will be able to cope with these cuts without undue hardship. There is a precedent in Wales, where HEFCW will not be awarding any QR funding after the 2013 REF, so why shouldn’t England do the same? They could also get away with the argument that this money isn’t part of the ring fence mentioned above.

The only problem is that so many institutions have spent so much time on preparations for the REF that cancelling the funding associated with it will mean all that effort is wasted. Indeed, the only remaining justification (if it can be called that) for Universities participating in the REF is for position in various league tables, which is rather a lot of expense for something of extremely debatable value.

Anyway, if I were a gambling man (which I am, actually) I don’t think I’d be betting against this outcome. Predictions are very difficult, especially about the future, but this one is mine. And I hope it’s proved wrong…

REF moves the goalposts (again)

Posted in Bad Statistics, Education, Science Politics with tags , , , on January 18, 2013 by telescoper

The topic of the dreaded 2014 Research Excellence Framework came up quite a few times in quite a few different contexts over the last few days, which reminded me that I should comment on a news item that appeared a week or so ago.

As you may or may not be aware, the REF is meant to assess the excellence of university departments in various disciplines and distribute its “QR” research funding accordingly.  Institutions complete submissions which include details of relevant publications etc and then a panel sits in judgement. I’ve already blogged of all this: the panels clearly won’t have time to read every paper submitted in any detail at all, so the outcome is likely to be highly subjective. Moreover, HEFCE’s insane policy to award the bulk of its research funds to only the very highest grade (4* – “internationally excellent”) means that small variations in judged quality will turn into enormous discrepancies in the level of research funding. The whole thing is madness, but there seems no way to inject sanity into the process as the deadline for submissions remorselessly approaches.

Now another wrinkle has appeared on the already furrowed brows of those preparing REF submissions. The system allows departments to select staff to be entered; it’s not necessary for everyone to go in. Indeed if only the very best researchers are entered then the typical score for the department will be high, so it will appear  higher up  in the league tables, and since the cash goes primarily to the top dogs then this might produce almost as much money as including a few less highly rated researchers.

On the other hand, this is a slightly dangerous strategy because it presupposes that one can predict which researchers and what research will be awarded the highest grade. A department will come a cropper if all its high fliers are deemed by the REF panels to be turkeys.

In Wales there’s something that makes this whole system even more absurd, which is that it’s almost certain that there will be no QR funding at all. Welsh universities are spending millions preparing for the REF despite the fact that they’ll get no money even if they do stunningly well. The incentive in Wales is therefore even stronger than it is in England to submit only the high-fliers, as it’s only the position in the league tables that will count.

The problem with a department adopting the strategy of being very selective is that it could have a very  negative effect on the career development of younger researchers if they are not included in their departments REF submission. As well as taking the risk that people who manage to convince their Head of School that they are bound to get four stars in the REF may not have the same success with the various grey eminences who make the decision that really matters.

Previous incarnations of the REF (namely the Research Assessment Exercises of 2008 and 2001) did not publish explicit information about exactly how many eligible staff were omitted from the submissions, largely because departments were extremely creative in finding ways of hiding staff they didn’t want to include.

Now however it appears there are plans that the Higher Education Statistics Agency (HESA) will publish its own figures on how many staff it thinks are eligible for inclusion in each department. I’m not sure how accurate these figures will be but they will change the game, in that they will allow compilers of league tables to draw up lists of the departments that prefer playing games to   just allowing the REF panels to  judge the quality of their research.

I wonder how many universities are hastily revising their submission plans in the light of this new twist?

The Quality of Physics

Posted in Science Politics with tags , , , on February 21, 2012 by telescoper

Just time for a quick post this lunchtime,  in between meetings and exercise classes. My eye was drawn this morning to an article about a lengthy report from the Institute of Physics that gives an international comparison of citation impact in physics and related fields.

According to the IOP website..

Although the UK is ranked seventh in a list of key competitor countries for the quantity of its physics research output – measured by the number of papers published – the UK is second only to Canada, and now higher than the US, when ranked on the average quality of the UK’s physics research output – measured by the average number of times research papers are cited around world.

The piece also goes on to note that the UK’s share of the total number of research papers written has decreased

For the UK, however, its proportionate decrease in output – from 7.1% of the world’s physics research in 2001 to 6.4% in 2010 – has been accompanied by a celebratory increase in overall, average quality – with the average number of citations of UK research papers rising from 1.24 in 2001 to 1.72 in 2010.

This, of course, assumes that citations measure “quality” but I’ve got no time to argue that point today. What I will do is put up a couple of interesting figures from the report.  This one shows that Space Science in the UK (including Astronomy and Astrophysics) holds a much bigger share of the total world output of papers than other disciplines (by a factor of about three):

While this one shows that the “citation impact” for Physics and Space Science roughly track each other…

..apart from the downturn right at the end of the window for space sciences, which, one imagines, might be a result of decisions taken by the management of the Science and Technology Facilities Council  over that period.

Our political leaders will be tempted to portray the steady increase of citation impact across fields as a sign of improved quality arising from the various research assessment exercises.  But I don’t think it’s as simple as that. It seems that many developing countries – especially China – are producing more and more scientific papers. This inevitably drives the UK’s share of world productivity down, because our capacity is not increasing. If anything it’s going down, in fact, owing to recent funding cuts. However, the more papers there are, the more reference lists there are, and the more citations there will be. The increase in citation rates may therefore just be a form of inflation.

Anyway, you can download the entire report here (PDF). I’m sure there will be other reactions to it so, as usual, please feel free to comment via the box below…

Follow

Get every new post delivered to your Inbox.

Join 3,807 other followers