Archive for REF

The Stifling Effect of REF Impact

Posted in Education, Politics with tags , , , on April 22, 2014 by telescoper

Well, I’m back to civilization (more or less) and with my plan to watch a day of cricket at Sophia Gardens thwarted by the rain I decided to pop into an internet café and do a quick post about one of the rants that has been simmering on the back burner while I’ve been taking a break.

Just before the Easter vacation I had lunch with some colleagues from the Department of Physics & Astronomy at the University of Sussex. One of the things that came up was the changing fortunes of the department. After years of under-investment from the University administration,  it was at one time at such a low ebb  that it was in real danger of being closed down (despite its undoubted strengths in research and teaching).  Fortunately help came in the form of SEPnet, which provided funds to support new initiatives in Physics not only in Sussex but across the South East. Moreover, the University administration had belatedly realized that a huge part of the institutional standing in tables of international research rankings was being generated by the Department of Physics & Astronomy. In the nick of time, the necessary resources were invested and the tide was turned and there has been steady growth in staff and student numbers since.

As Head of the School of Mathematical and Physical Sciences I have had to deal with the budget for the Department of Physics & Astronomy. Just a decade ago very few physics departments in the UK were  financially solvent and most had to rely on generous subsidies from University funds to stay open. Those that did not receive such support were closed down, a fate which Sussex narrowly avoided but which befell, for example, the physics departments at Reading and Newcastle.

As I blogged about previously, the renaissance of Sussex physics seems not to be unique. Admissions to physics departments across the country are growing at a healthy rate, to the extent that new departments are being formed at, e.g. Lincoln and Portsmouth. None of this could have been imagined just ten years ago.

So will this new-found optimism be reflected in the founding of even more new physics departments? One would hope so, as I think it’s a scandal that there are only around 40 UK universities with physics departments. Call me old-fashioned but I think a university without a physics department is not a university at all. Thinking about this over the weekend however I realized that any new physics department is going to have grave problems under the system of allocating research funding known as the Research Excellence Framework.

A large slice (20%) of the funding allocated by the 2014 REF will be based on “Impact” which, roughly speaking, means the effect the research can be demonstrated to have had outside the world of academic research. This isn’t the largest component – 65% is allocated on the basic of the quality of “Outputs” (research papers etc) – but is a big chunk and will probably be very important in determining league table positions. It is probably going to be even larger in future versions of the REF.

Now here’s the rub. When an academic changes institution (as I have recently done, for example) he/she can take his/her outputs to the new institution. Thus, papers I wrote while at Cardiff could be submitted to the REF from Sussex. This is not the case with “impact”. The official guidance on submissions states:

Impact: The sub-panels will assess the ‘reach and significance’ of impacts on the economy, society and/or culture that were underpinned by excellent research conducted in the submitted unit, as well as the submitted unit’s approach to enabling impact from its research. This element will carry a weighting of 20 per cent.

The emphasis is mine.

The period during which the underpinning research must have been published is quite generous in length: 1 January 1993 to 31 December 2013. This is clearly intended to recognize the fact that some research take a long time to generate measurable impact. The problem is that the underpinning research must have been done within the submitting unit; it can’t be brought in from elsewhere. If the unit is new and did not exist for most of this period,then it is much harder to generate impact no matter how brilliant the staff it recruits. Any new departments in physics, or any other subject for that matter, will have to focus on research that can generate impact very rapidly indeed if it is to compete in the next REF, expected in 2018 or thereabouts. That is a powerful disincentive for universities to invest in research that may take many years to come to fruition. Five years is a particularly short time in experimental physics.

 

 

Open Access Repositories should be based on Subject, not Institution

Posted in Open Access with tags , , , , on September 18, 2013 by telescoper

Just when we thought that the powers that be might be starting to see the light on Open Access, round comes another circular from the Higher Education Funding Council for England (HEFCE) that shows that policymakers have an unlimited ability to get things wrong at the most basic level.

The document concerned opens a “Consultation on open access in the post-2014 Research Excellence Framework” by putting forward a number of proposals.

Now it’s depressing enough that the entire document is predicated on the assumption that there will be another Research Excellence Framework, perhaps in 2020. The current REF is such a disaster that one might have hoped somebody up there would have decided that enough is enough. But then we thought that about Research Assessment Exercise in 2008. Even the most pessimistic of us hope that the REF would have a “lighter touch” than the RAE, but as it has turned out it’s many times worse both in the time it has taken to prepare submissions and the ridiculous game-playing and dodgy employment practices that it has encouraged among participation institutions.

I hope there’s still time to drive a stake through the heart of the runaway bureaucracy that keeps imposing this idiocies on us. After all, a new Chief Executive about to take over at HEFCE. Perhaps a new broom will be wielded? I hope against hope.

However, setting all that to one side, I had a look at the proposals for Open Access after 2014 contained in the document. Here is the summary of the proposals:

HEFCE

Aarrgh!

Why on Earth should the proposals favour national institutional repositories over international subject-based ones? A shining example of the latter is the arXiv which has, for Physics and Astronomy, become the basic resource for researchers around the world; it’s a one-stop shop at which one can access research from all around the world. By contrast, having work in the same field stored over a plethora of institutional repositories will serve no useful purpose at all, because UK research will not  treated in the same way as work from other countries and in any case individual repositoes will lead to an absurd level of duplication of infrastructure and other resources. This requirement is particularly indefensible in Physics and Astronomy, as it would require us to duplicate in (probably inferior) institutional repositories what we already do with the arXiv.

The UK Funding Councils need realize that the solution to many of the challenges of Open Access has already been found. In fact, the European Research Council seems to have acknowledged this and is now directly funding the arXiv. The UK Research Councils should be required to construct similar archives for their disciplines. That shouldn’t be difficult, because all the hard work has already been done. There is a working model.

I’ll be responding to the consultation document in no uncertain terms. The Royal Astronomical Society is also collating responses for a collective submission. We have to resist these, and other proposals such as another REF, which are being foisted on us by people who have no idea what they’re doing and no idea what damage they’re causing.

The Dark Side of the REF

Posted in Finance, Science Politics with tags , , , , , , , , on August 8, 2013 by telescoper

There’s a disturbing story in the latest Times Higher which argues that the University of Leicester has apparently reneged on a promise that non-submission to the forthcoming (2014)  Research Excellence Framework (REF) would not have negative career consequences. They have now said that except in exceptional circumstances, non-submitted academics will either be moved to a teaching-only contract (where there is a vacancy and they can demonstrate teaching excellence), or have their performance “managed”, with the threat of sacking if they don’t meet the specified targets.  I’d heard rumours of this on the grapevine (i.e. Twitter) before the Times Higher story was published. It’s very worrying to have it confirmed, as it raises all kinds of questions about what might happen in departments that turn out to have disappointing REF results .

There are (at least) two possible reasons for non-inclusion of the outputs of a researcher and it is important to distinguish between them. One is that the researcher hasn’t enough high-quality outputs to submit. In the absence of individual extenuating circumstances, researchers are expected to submit four “outputs” (in my discipline that means “research papers”) for assessment. That’s a pretty minimal level of productivity, actually;  such a number per year is a reasonable average for an active researcher in my field.  A person employed on a contract that specifies their duties as Teaching and Research may therefore be under-performing  if they can’t produce four papers over the period 2008-2013. I think some form of performance management  may be justifiable in this case, but the primary aim should be to help the individual rather than show them the door. We all have fallow periods in research, and it’s not appropriate to rush to sack anyone who experiences a lean time.   Andrew Wiles would have been considered `inactive’ had there been a REF in 1992 as he hadn’t published anything for years. Then he produced a proof of Fermat’s Last Theorem. Some things just take time.

A second reason for excluding researcher from the REF is that the institution concerned may be making a tactical submission. As the Times Higher article explains:

The memo suggests that academics would be spared repercussions if, among other reasons, the number of individuals submitted is “constrained” by the volume of case studies their department intends to enter to demonstrate research impact.

Institutions must submit one case study for every 10 scholars entered.

Maria Nedeva, professor of science and innovation dynamics and policy at Manchester Business School, said the tactic of deciding how many academics to submit based on impact case study numbers was “rife”.

(Incidentally, the second paragraph is not quite right. The number of case studies required depends on the number of staff submitted as follows: for fewer than 15 staff , TWO case studies;  for 15-24.99 staff it is THREE case studies – and then for each additional ten members of staff entered a further case study is required.)

e case study for every scholars included plus one, i.e. forThe statement at the end of the quote there is in line with my experience too.  The point is that the REF is not just a means of allocating relatively small amounts of so-called `QR’ research funding . Indeed, it remains entirely possible that no funding at all will be allocated following the 2014 exercise. The thinking then is that the number of staff submitted is largely irrelevant; all that will count is league table position.

This by no means the only example of the dangers that lurk when you take league tables too seriously.

If a department is required to submit, say, four impact cases if all staff are included in the REF submission, but only has three viable ones, it would not be unreasonable to submit fewer staff because their overall would be dragged down by a poor impact case even if the output quality of all staff is high.  There will certainly be highly active researchers in UK institutions, including many who hold sizable external research grants, whose outputs are not submitted to the REF. As the article points out, it would be very wrong for managers to penalize scholars who have been excluded because of this sort of game-playing. That’s certainly not going to happen in the School of Mathematical and Physical Sciences at Sussex University.  Not while I’m Head of School, anyway.

Moreover, even researchers whose “outputs” are not selected may still contribute to the “Environment” and/or “Impact” sections so they still, in a very real sense, do participate in their department’s REF submission.

My opinion? All this silliness could easily have been avoided by requiring all staff in all units of assessment to be submitted by all departments. You know, like would have happened if the system were actually designed to identify and reward research excellence. Instead, it’s yet another example of a bureaucratic machine that’s become entirely self-serving. It exists simply because it exists.  Research would be much better off without it.

University Research Funding: Will the Axe Fall on QR?

Posted in Finance, Politics, Science Politics with tags , , , on June 25, 2013 by telescoper

As we tremble in anticipation of this week’s Spending Review, which will determine the budgets for Science and Higher Education in 2015/16, there’s fairly strong evidence that Chancellor of the Exchequer George Osborne is looking to save about £11.5 billion of public spending. Given that funding for some Whitehall departments is ring-fenced there is considerable speculation that the axe will fall heavily on the Department for Business, Innovation and Skills (BIS), which seems likely to have to make over £1 billion of savings.

But where will these savings in the BIS budget be made? The government has made noises that it will protect science funding (at least in cash terms) so big cuts in the larger research council budgets appear unlikely. However, Treasury officials have been rumoured as thinking that the Universities are now “awash with money” and should therefore be cut. On the other hand, incoming for University teaching now largely comes from fees so there’s very little of the HEFCE teaching budget to cut further.

Now here’s the rub. The part of HEFCE’s budget that deals with research amounts to about £1.6 billion per year. This, the so-called `QR’ funding, is currently being distributed to Higher Education Institutions according to the outcome of the 2008 Research Assessment Exercise (RAE). This year we
are preparing submissions to a new system called the Research Excellence Framework (REF), which was always intended to be used to distribute QR funding from 2015/16 onwards. But what if the government decides that the only way to balance the books is to remove the QR funding stream entirely?

The 2012 funding letter from HEFCE states explains that it is distributing

£1,558 million for research. The ring-fenced settlement for science and research means that we will be able to maintain overall funding, in cash terms, until 2014-15.

But this does not include the period covered by the spending review, so it’s perfectly possible that the “ring fence” could be removed, or at least re-interpreted as a result of this spending review.

The government could argue that QR and Research Council grant income correlate so well that there’s no need to continue with the current dual funding system, by which the Research Councils provide grants for specific projects and programmes and the higher education funding bodies provide block grant funding to universities via the QR line. It could also argue that the high fees being charged mean that Universities will be able to cope with these cuts without undue hardship. There is a precedent in Wales, where HEFCW will not be awarding any QR funding after the 2013 REF, so why shouldn’t England do the same? They could also get away with the argument that this money isn’t part of the ring fence mentioned above.

The only problem is that so many institutions have spent so much time on preparations for the REF that cancelling the funding associated with it will mean all that effort is wasted. Indeed, the only remaining justification (if it can be called that) for Universities participating in the REF is for position in various league tables, which is rather a lot of expense for something of extremely debatable value.

Anyway, if I were a gambling man (which I am, actually) I don’t think I’d be betting against this outcome. Predictions are very difficult, especially about the future, but this one is mine. And I hope it’s proved wrong…

REF moves the goalposts (again)

Posted in Bad Statistics, Education, Science Politics with tags , , , on January 18, 2013 by telescoper

The topic of the dreaded 2014 Research Excellence Framework came up quite a few times in quite a few different contexts over the last few days, which reminded me that I should comment on a news item that appeared a week or so ago.

As you may or may not be aware, the REF is meant to assess the excellence of university departments in various disciplines and distribute its “QR” research funding accordingly.  Institutions complete submissions which include details of relevant publications etc and then a panel sits in judgement. I’ve already blogged of all this: the panels clearly won’t have time to read every paper submitted in any detail at all, so the outcome is likely to be highly subjective. Moreover, HEFCE’s insane policy to award the bulk of its research funds to only the very highest grade (4* – “internationally excellent”) means that small variations in judged quality will turn into enormous discrepancies in the level of research funding. The whole thing is madness, but there seems no way to inject sanity into the process as the deadline for submissions remorselessly approaches.

Now another wrinkle has appeared on the already furrowed brows of those preparing REF submissions. The system allows departments to select staff to be entered; it’s not necessary for everyone to go in. Indeed if only the very best researchers are entered then the typical score for the department will be high, so it will appear  higher up  in the league tables, and since the cash goes primarily to the top dogs then this might produce almost as much money as including a few less highly rated researchers.

On the other hand, this is a slightly dangerous strategy because it presupposes that one can predict which researchers and what research will be awarded the highest grade. A department will come a cropper if all its high fliers are deemed by the REF panels to be turkeys.

In Wales there’s something that makes this whole system even more absurd, which is that it’s almost certain that there will be no QR funding at all. Welsh universities are spending millions preparing for the REF despite the fact that they’ll get no money even if they do stunningly well. The incentive in Wales is therefore even stronger than it is in England to submit only the high-fliers, as it’s only the position in the league tables that will count.

The problem with a department adopting the strategy of being very selective is that it could have a very  negative effect on the career development of younger researchers if they are not included in their departments REF submission. As well as taking the risk that people who manage to convince their Head of School that they are bound to get four stars in the REF may not have the same success with the various grey eminences who make the decision that really matters.

Previous incarnations of the REF (namely the Research Assessment Exercises of 2008 and 2001) did not publish explicit information about exactly how many eligible staff were omitted from the submissions, largely because departments were extremely creative in finding ways of hiding staff they didn’t want to include.

Now however it appears there are plans that the Higher Education Statistics Agency (HESA) will publish its own figures on how many staff it thinks are eligible for inclusion in each department. I’m not sure how accurate these figures will be but they will change the game, in that they will allow compilers of league tables to draw up lists of the departments that prefer playing games to   just allowing the REF panels to  judge the quality of their research.

I wonder how many universities are hastily revising their submission plans in the light of this new twist?

The Quality of Physics

Posted in Science Politics with tags , , , on February 21, 2012 by telescoper

Just time for a quick post this lunchtime,  in between meetings and exercise classes. My eye was drawn this morning to an article about a lengthy report from the Institute of Physics that gives an international comparison of citation impact in physics and related fields.

According to the IOP website..

Although the UK is ranked seventh in a list of key competitor countries for the quantity of its physics research output – measured by the number of papers published – the UK is second only to Canada, and now higher than the US, when ranked on the average quality of the UK’s physics research output – measured by the average number of times research papers are cited around world.

The piece also goes on to note that the UK’s share of the total number of research papers written has decreased

For the UK, however, its proportionate decrease in output – from 7.1% of the world’s physics research in 2001 to 6.4% in 2010 – has been accompanied by a celebratory increase in overall, average quality – with the average number of citations of UK research papers rising from 1.24 in 2001 to 1.72 in 2010.

This, of course, assumes that citations measure “quality” but I’ve got no time to argue that point today. What I will do is put up a couple of interesting figures from the report.  This one shows that Space Science in the UK (including Astronomy and Astrophysics) holds a much bigger share of the total world output of papers than other disciplines (by a factor of about three):

While this one shows that the “citation impact” for Physics and Space Science roughly track each other…

..apart from the downturn right at the end of the window for space sciences, which, one imagines, might be a result of decisions taken by the management of the Science and Technology Facilities Council  over that period.

Our political leaders will be tempted to portray the steady increase of citation impact across fields as a sign of improved quality arising from the various research assessment exercises.  But I don’t think it’s as simple as that. It seems that many developing countries – especially China – are producing more and more scientific papers. This inevitably drives the UK’s share of world productivity down, because our capacity is not increasing. If anything it’s going down, in fact, owing to recent funding cuts. However, the more papers there are, the more reference lists there are, and the more citations there will be. The increase in citation rates may therefore just be a form of inflation.

Anyway, you can download the entire report here (PDF). I’m sure there will be other reactions to it so, as usual, please feel free to comment via the box below…

A Modest Proposal

Posted in Education, Science Politics with tags , , , , on March 7, 2011 by telescoper

Last week I posted a short item about the looming Kafka-esque nightmare that is the Research Excellence Framework. A few people commented to me in private that although they hate the REF and accept that it’s ridiculously expensive and time-consuming, they didn’t see any alternative. I’ve been thinking about it and thought I’d make a suggestion. Feel free to shoot it down in flames through the box at the end, but I’ll begin with a short introduction.

Those of you old enough to remember will know that before 1992 (when the old `polytechnics’ were given the go-ahead to call themselves `universities’) the University Funding Council – the forerunner of HEFCE – allocated research funding to universities by a simple formula related to the number of undergraduate students. When the number of universities suddenly increased this was no longer sustainable, so the funding agency began a series of Research Assessment Exercises to assign research funds (now called QR funding) based on the outcome. This prevented research money going to departments that weren’t active in research, most (but not all) of which were in the ex-Polys. Over the years the apparatus of research assessment has become larger, more burdensome, and incomprehensibly obsessed with “research concentration”. Like most bureaucracies it has lost sight of its original purpose and has now become something that exists purely for its own sake.

It’s especially indefensible at this time of deep cuts to university budgets that we are being forced to waste an increasingly large fraction of our decreasing budgets on staff-time that accomplishes nothing useful except pandering to the bean counters.

My proposal is to abandon the latest manifestation of research assessment mania, i.e. the REF, and return to a simple formula, much like the pre-1992 system,  except that QR funding should be based on research student rather than undergraduate numbers.

There’s an obvious risk of game-playing, and this idea would only stand a chance of working at all if the formula involved the number of successfully completed research degrees over a given period .

I can also see an argument  that four-year undergraduate students (e.g. MPhys or MSci students) also be included in the formula, as most of these involve a project that requires a strong research environment.

Among the advantages of this scheme are that it’s simple, easy to administer, would not spread QR funding in non-research departments, and would not waste hundreds of millions of pounds on bureaucracy that would be better spent on research. It would also maintain the current “dual support” system for research.

I’m sure you’ll point out disadvantages through the comments box!


Share/Bookmark

Follow

Get every new post delivered to your Inbox.

Join 3,747 other followers