Archive for impact

The Stifling Effect of REF Impact

Posted in Education, Politics with tags , , , on April 22, 2014 by telescoper

Well, I’m back to civilization (more or less) and with my plan to watch a day of cricket at Sophia Gardens thwarted by the rain I decided to pop into an internet café and do a quick post about one of the rants that has been simmering on the back burner while I’ve been taking a break.

Just before the Easter vacation I had lunch with some colleagues from the Department of Physics & Astronomy at the University of Sussex. One of the things that came up was the changing fortunes of the department. After years of under-investment from the University administration,  it was at one time at such a low ebb  that it was in real danger of being closed down (despite its undoubted strengths in research and teaching).  Fortunately help came in the form of SEPnet, which provided funds to support new initiatives in Physics not only in Sussex but across the South East. Moreover, the University administration had belatedly realized that a huge part of the institutional standing in tables of international research rankings was being generated by the Department of Physics & Astronomy. In the nick of time, the necessary resources were invested and the tide was turned and there has been steady growth in staff and student numbers since.

As Head of the School of Mathematical and Physical Sciences I have had to deal with the budget for the Department of Physics & Astronomy. Just a decade ago very few physics departments in the UK were  financially solvent and most had to rely on generous subsidies from University funds to stay open. Those that did not receive such support were closed down, a fate which Sussex narrowly avoided but which befell, for example, the physics departments at Reading and Newcastle.

As I blogged about previously, the renaissance of Sussex physics seems not to be unique. Admissions to physics departments across the country are growing at a healthy rate, to the extent that new departments are being formed at, e.g. Lincoln and Portsmouth. None of this could have been imagined just ten years ago.

So will this new-found optimism be reflected in the founding of even more new physics departments? One would hope so, as I think it’s a scandal that there are only around 40 UK universities with physics departments. Call me old-fashioned but I think a university without a physics department is not a university at all. Thinking about this over the weekend however I realized that any new physics department is going to have grave problems under the system of allocating research funding known as the Research Excellence Framework.

A large slice (20%) of the funding allocated by the 2014 REF will be based on “Impact” which, roughly speaking, means the effect the research can be demonstrated to have had outside the world of academic research. This isn’t the largest component – 65% is allocated on the basic of the quality of “Outputs” (research papers etc) – but is a big chunk and will probably be very important in determining league table positions. It is probably going to be even larger in future versions of the REF.

Now here’s the rub. When an academic changes institution (as I have recently done, for example) he/she can take his/her outputs to the new institution. Thus, papers I wrote while at Cardiff could be submitted to the REF from Sussex. This is not the case with “impact”. The official guidance on submissions states:

Impact: The sub-panels will assess the ‘reach and significance’ of impacts on the economy, society and/or culture that were underpinned by excellent research conducted in the submitted unit, as well as the submitted unit’s approach to enabling impact from its research. This element will carry a weighting of 20 per cent.

The emphasis is mine.

The period during which the underpinning research must have been published is quite generous in length: 1 January 1993 to 31 December 2013. This is clearly intended to recognize the fact that some research take a long time to generate measurable impact. The problem is that the underpinning research must have been done within the submitting unit; it can’t be brought in from elsewhere. If the unit is new and did not exist for most of this period,then it is much harder to generate impact no matter how brilliant the staff it recruits. Any new departments in physics, or any other subject for that matter, will have to focus on research that can generate impact very rapidly indeed if it is to compete in the next REF, expected in 2018 or thereabouts. That is a powerful disincentive for universities to invest in research that may take many years to come to fruition. Five years is a particularly short time in experimental physics.

 

 

Advertisements

Six (very) bad things about the REF

Posted in Education, Science Politics with tags , , , , on November 22, 2013 by telescoper

I see that Jon Butterworth has written a piece on the Grauniad website, entitled Six good things about the REF, the REF in question not being a black-clad figure of questionable parentage and visual acuity responsible for supervising a game of association football, but the Research Excellence Framework.

I agree with some of Jon’s comments and do believe that past Research Assessment Exercises have generally raised the quality of research in UK universities. I do however think that there are some very bad things about the way the REF is being implemented, and that these far outweigh the positives Jon mentions. In the interest of balance, therefore, I thought I’d respond with a list of six (very) bad things about the REF, and particularly how it applies to physics. I’ll keep them brief because I’ve blogged about most of them before:

  1. The rules positively encouraged universities to play games with selectivity. This is absurd. All academic staff on teaching and research contracts should be submitted if a true indication of research quality is to be obtained.
  2. The criteria for what constitutes 3* or 4* publications are vague and subjective, leaving everything in the hands of the panels. Worse, all paperwork will be shredded after the panel’s deliberations leaving no possibility for appeal. This absolutely stinks.
  3. How QR funding will be allocated on the basis of the REF is not made clear in advance of the submission. Nobody knows how heavily the funding will be skewed towards 4* and 3* submissions. Having encouraged departments to play games, therefore, the REF refuses to disclose the rules. It’s not even clear there will be any QR funding.
  4. The panels will be unable to perform a detailed peer review of submissions simply because there will be too many papers. Each panel will be expected to make decisions on many hundreds of papers, leaving time only for a cursory reading of each.
  5. Limiting the physics submission to 4 papers per person is ridiculous. This corresponds to a tiny fraction of the outputs of a typical physics researcher. If someone has written ten 4* publications in the REF period, why should these not be counted?
  6. Impact counts for a sizable fraction (20%) of the funding, but the rules governing what counts as “impact” are absurdly restrictive and clearly encourage short-term commercially-oriented boilerplate stuff at the expense of genuine long-term “blue skies” research.

 

Well, I got to six in just a few minutes and could easily get to sixty, but that will do for now. Perhaps you’d like to contribute your own bad things through the comments box?

The Dark Side of the REF

Posted in Finance, Science Politics with tags , , , , , , , , on August 8, 2013 by telescoper

There’s a disturbing story in the latest Times Higher which argues that the University of Leicester has apparently reneged on a promise that non-submission to the forthcoming (2014)  Research Excellence Framework (REF) would not have negative career consequences. They have now said that except in exceptional circumstances, non-submitted academics will either be moved to a teaching-only contract (where there is a vacancy and they can demonstrate teaching excellence), or have their performance “managed”, with the threat of sacking if they don’t meet the specified targets.  I’d heard rumours of this on the grapevine (i.e. Twitter) before the Times Higher story was published. It’s very worrying to have it confirmed, as it raises all kinds of questions about what might happen in departments that turn out to have disappointing REF results .

There are (at least) two possible reasons for non-inclusion of the outputs of a researcher and it is important to distinguish between them. One is that the researcher hasn’t enough high-quality outputs to submit. In the absence of individual extenuating circumstances, researchers are expected to submit four “outputs” (in my discipline that means “research papers”) for assessment. That’s a pretty minimal level of productivity, actually;  such a number per year is a reasonable average for an active researcher in my field.  A person employed on a contract that specifies their duties as Teaching and Research may therefore be under-performing  if they can’t produce four papers over the period 2008-2013. I think some form of performance management  may be justifiable in this case, but the primary aim should be to help the individual rather than show them the door. We all have fallow periods in research, and it’s not appropriate to rush to sack anyone who experiences a lean time.   Andrew Wiles would have been considered `inactive’ had there been a REF in 1992 as he hadn’t published anything for years. Then he produced a proof of Fermat’s Last Theorem. Some things just take time.

A second reason for excluding researcher from the REF is that the institution concerned may be making a tactical submission. As the Times Higher article explains:

The memo suggests that academics would be spared repercussions if, among other reasons, the number of individuals submitted is “constrained” by the volume of case studies their department intends to enter to demonstrate research impact.

Institutions must submit one case study for every 10 scholars entered.

Maria Nedeva, professor of science and innovation dynamics and policy at Manchester Business School, said the tactic of deciding how many academics to submit based on impact case study numbers was “rife”.

(Incidentally, the second paragraph is not quite right. The number of case studies required depends on the number of staff submitted as follows: for fewer than 15 staff , TWO case studies;  for 15-24.99 staff it is THREE case studies – and then for each additional ten members of staff entered a further case study is required.)

e case study for every scholars included plus one, i.e. forThe statement at the end of the quote there is in line with my experience too.  The point is that the REF is not just a means of allocating relatively small amounts of so-called `QR’ research funding . Indeed, it remains entirely possible that no funding at all will be allocated following the 2014 exercise. The thinking then is that the number of staff submitted is largely irrelevant; all that will count is league table position.

This by no means the only example of the dangers that lurk when you take league tables too seriously.

If a department is required to submit, say, four impact cases if all staff are included in the REF submission, but only has three viable ones, it would not be unreasonable to submit fewer staff because their overall would be dragged down by a poor impact case even if the output quality of all staff is high.  There will certainly be highly active researchers in UK institutions, including many who hold sizable external research grants, whose outputs are not submitted to the REF. As the article points out, it would be very wrong for managers to penalize scholars who have been excluded because of this sort of game-playing. That’s certainly not going to happen in the School of Mathematical and Physical Sciences at Sussex University.  Not while I’m Head of School, anyway.

Moreover, even researchers whose “outputs” are not selected may still contribute to the “Environment” and/or “Impact” sections so they still, in a very real sense, do participate in their department’s REF submission.

My opinion? All this silliness could easily have been avoided by requiring all staff in all units of assessment to be submitted by all departments. You know, like would have happened if the system were actually designed to identify and reward research excellence. Instead, it’s yet another example of a bureaucratic machine that’s become entirely self-serving. It exists simply because it exists.  Research would be much better off without it.