Archive for Research Excellence Framework

That Was The REF That Was..

Posted in Finance, Science Politics with tags , , , , , , on December 18, 2014 by telescoper

I feel obliged to comment on the results of the 2014 Research Excellence Framework (REF) that were announced today. Actually, I knew about them yesterday but the news was under embargo until one minute past midnight by which time I was tucked up in bed.

The results for the two Units of Assessment relevant to the School of Mathematical and Physical Sciences are available online here for Mathematical Sciences and here for Physics and Astronomy.

To give some background: the overall REF score for a Department is obtained by adding three different components: outputs (quality of research papers); impact (referrring to the impact beyond academia); and environment (which measures such things as grant income, numbers of PhD students and general infrastructure). These are weighted at 65%, 20% and 15% respectively.

Scores are assigned to these categories, e.g. for submitted outputs (usually four per staff member) on a scale of 4* (world-leading), 3* (internationally excellent), 2* (internationally recognised), 1* (nationally recognised) and unclassified and impact on a scale 4* (outstanding), 3* (very considerable), 2* (considerable), 1* (recognised but modest) and unclassified. Impact cases had to be submitted based on the number of staff submitted: two up to 15 staff, three between 15 and 25 and increasing in a like manner with increasing numbers.

The REF will control the allocation of funding in a manner yet to be decided in detail, but it is generally thought that anything scoring 2* or less will attract no funding (so the phrase “internationally recognised” really means “worthless” in the REF, as does “considerable” when applied to impact). It is also thought likely that funding will be heavily weighted towards 4* , perhaps with a ratio of 9:1 between 4* and 3*.

We knew that this REF would be difficult for the School and our fears were born out for both the Department of Mathematics or the Department of Physics and Astronomy because both departments grew considerably (by about 50%) during the course of 2013, largely in response to increased student numbers. New staff can bring outputs from elsewhere, but not impact. The research underpinning the impact has to have been done by staff working in the institution in question. And therein lies the rub for Sussex…

To take the Department of Physics and Astronomy, as an example, last year we increased staff numbers from about 23 to about 38. But the 15 new staff members could not bring any impact with them. Lacking sufficient impact cases to submit more, we were obliged to restrict our submission to fewer than 25. To make matters worse our impact cases were not graded very highly, with only 13.3% of the submission graded 4* and 13.4% graded 3*.

The outputs from Physics & Astronomy at Sussex were very good, with 93% graded 3* or 4*. That’s a higher fraction than Oxford, Cambridge, Imperial College and UCL in fact, and with a Grade Point Average of 3.10. Most other departments also submitted very good outputs – not surprisingly because the UK is actually pretty good at Physics – so the output scores are very highly bunched and a small difference in GPA means a large number of places in the rankings. The impact scores, however, have a much wider dispersion, with the result that despite the relatively small percentage contribution they have a large effect on overall rankings. As a consequence, overall, Sussex Physics & Astronomy slipped down from 14th in the RAE to 34th place in the REF (based on a Grade Point Average). Disappointing to say the least, but we’re not the only fallers. In the 2008 RAE the top-rated physics department was Lancaster; this time round they are 27th.

I now find myself in a situation eerily reminiscent of that I found myself facing in Cardiff after the 2008 Research Assessment Exercise, the forerunner of the REF. Having been through that experience I’m a hardened to disappointments and at least can take heart from Cardiff’s performance this time round. Spirits were very low there after the RAE, but a thorough post-mortem, astute investment in new research areas, and determined preparations for this REF have paid dividends: they have climbed to 6th place this time round. That gives me the chance not only to congratulate my former colleagues there for their excellent result but also to use them as an example for what we at Sussex have to do for next time. An even more remarkable success story is Strathclyde, 34th in the last RAE and now top of the REF table. Congratulations to them too!

Fortunately our strategy is already in hand. The new staff have already started working towards the next REF (widely thought to be likely to happen in 2020) and we are about to start a brand new research activity in experimental physics next year. We will be in a much better position to generate research impact as we diversify our portfolio so that it is not as strongly dominated by “blue skies” research, such as particle physics and astronomy, for which it is much harder to demonstrate economic impact.

I was fully aware of the challenges facing Physics & Astronomy at Sussex when I moved here in February 2013, but with the REF submission made later the same year there was little I could do to alter the situation. Fortunately the University of Sussex management realises that we have to play a long game in Physics and has been very supportive of our continued strategic growth. The result of the 2014 REF result is a setback but it does demonstrate that the stategy we have already embarked upon is the right one.

Roll on 2020!

Advertisements

Anthem for Doomed Academics

Posted in Poetry, Science Politics with tags , , on December 17, 2014 by telescoper

Well, not long now until the announcement of the results of the 2014 Research Excellence Framework are known publicly. I’ll post something in the way of a personal reflection tomorrow, as long as I haven’t thrown myself off Brighton Pier by then. In the meantime, I couldn’t resist sharing this brilliant parody of Wilfred Owen I found via Twitter…

Stumbling with Confidence

(This has been written as the momentous results of the Research Excellence Framework, known to all and sundry as the dreaded REF, are about to be announced, and as careers hang in the balance depending on who are the winners and losers.)

Anthem for Doomed Academics

(with apologies to Wilfred Owen)

What lasting hell for these who try as authors?
Only the monstrous anger of the dons.
Only the stuttering academic’s crippled cursor
Can patter out career horizons.
No metrics now for them; no citations nor reviews;
Nor any voice of warning save the choirs, –
The shrill, demented choirs of wailing peers;
And lost opportunities calling them from sad HEIs.
What meetings may be held to speed them all?
Not in the hand of managers but in their eyes
Shall shine the unholy glimmers of goodbyes.
The cost of student fees shall be their pall;
Their inheritance the frustrations…

View original post 11 more words

Mathematics and Meningococcal Meningitis

Posted in Education, Science Politics with tags , , , , on June 9, 2014 by telescoper

Last week I attended a very enjoyable and informative event entitled Excellence with Impact that showcased some of the research that the University of Sussex submitted to the 2014 Research Excellence Framework. One of the case studies came from the Department of Mathematics which is part of the School of Mathematical and Physical Sciences (of which I am Head) so I thought I would showcase it here too:

The description from Youtube reads

Meningococcal meningitis is a debilitating and deadly disease, causing an estimated 10,000 deaths annually in endemic areas of sub-Saharan Africa. A novel mathematical model developed by Sussex researcher Dr Konstantin Blyuss and colleagues has helped explain the patterns of the dynamics of meningococcal meningitis in endemic areas. This model is now being used by epidemiologists and clinical scientists to design and deliver efficient public-health policies to combat this devastating disease.

You can find out more by following this link.

A Note to the Physics REF Panel

Posted in Education, Finance, Science Politics with tags , , , on April 23, 2014 by telescoper

I’ve just been skimming through an interesting report about the strength of UK physics research. One of the conclusions of said document is that UK physics research is the best in the world in terms of quality.

I couldn’t resist a brief post to point this out to any members of the Physics panel involved in the 2014 Research Excellence Framework. My motivation for doing this is that the Physics panel of the 2008 Research Assessment Exercise evidently came to the conclusion that UK physics research wasn’t very good at all, awarding a very much lower fraction of 4* (world-leading) grades than many other disciplines, including chemistry. I’ve never understood why the Panel arrived at such a low opinion of its own discipline, but there you go..

Physics departments across the country have fought very hard to recover from the financial and reputational damage inflicted by the 2008 RAE panel’s judgement. Let’s just hope there isn’t another unwarranted slap in the face in store when the 2014 results are announced later this year…

 

UPDATE: I’m grateful to Paul Crowther for pointing out a surprising fact based on a talk given by the Chairman of the Physics RAE Panel in 2008, Sir John Pendry. Here are the slides in full, but the pertinent fact is the distribution of 4*, 3* and 2* grades across disciplines shown in this table:

Bl8OKv0CYAE6EMq

You can see that they are in fact broadly, similar across disciplines. However, what is clear is that the highest scoring departments in Chemistry did much better than the highest-scoring in Physics; for example top of the table for Physics was Lancaster with 25% of its outputs graded 4* while top in Chemistry was Cambridge with 40%. Is it really justifiable that the top physics departments were so much worse than the top chemistry departments? Suspicion remains that the Physics scores were downgraded systematically to produce the uncannily similar profiles shown in the table. Since all the RAE documents have been shredded, we’ll never know whether that happened or not…

The Stifling Effect of REF Impact

Posted in Education, Politics with tags , , , on April 22, 2014 by telescoper

Well, I’m back to civilization (more or less) and with my plan to watch a day of cricket at Sophia Gardens thwarted by the rain I decided to pop into an internet café and do a quick post about one of the rants that has been simmering on the back burner while I’ve been taking a break.

Just before the Easter vacation I had lunch with some colleagues from the Department of Physics & Astronomy at the University of Sussex. One of the things that came up was the changing fortunes of the department. After years of under-investment from the University administration,  it was at one time at such a low ebb  that it was in real danger of being closed down (despite its undoubted strengths in research and teaching).  Fortunately help came in the form of SEPnet, which provided funds to support new initiatives in Physics not only in Sussex but across the South East. Moreover, the University administration had belatedly realized that a huge part of the institutional standing in tables of international research rankings was being generated by the Department of Physics & Astronomy. In the nick of time, the necessary resources were invested and the tide was turned and there has been steady growth in staff and student numbers since.

As Head of the School of Mathematical and Physical Sciences I have had to deal with the budget for the Department of Physics & Astronomy. Just a decade ago very few physics departments in the UK were  financially solvent and most had to rely on generous subsidies from University funds to stay open. Those that did not receive such support were closed down, a fate which Sussex narrowly avoided but which befell, for example, the physics departments at Reading and Newcastle.

As I blogged about previously, the renaissance of Sussex physics seems not to be unique. Admissions to physics departments across the country are growing at a healthy rate, to the extent that new departments are being formed at, e.g. Lincoln and Portsmouth. None of this could have been imagined just ten years ago.

So will this new-found optimism be reflected in the founding of even more new physics departments? One would hope so, as I think it’s a scandal that there are only around 40 UK universities with physics departments. Call me old-fashioned but I think a university without a physics department is not a university at all. Thinking about this over the weekend however I realized that any new physics department is going to have grave problems under the system of allocating research funding known as the Research Excellence Framework.

A large slice (20%) of the funding allocated by the 2014 REF will be based on “Impact” which, roughly speaking, means the effect the research can be demonstrated to have had outside the world of academic research. This isn’t the largest component – 65% is allocated on the basic of the quality of “Outputs” (research papers etc) – but is a big chunk and will probably be very important in determining league table positions. It is probably going to be even larger in future versions of the REF.

Now here’s the rub. When an academic changes institution (as I have recently done, for example) he/she can take his/her outputs to the new institution. Thus, papers I wrote while at Cardiff could be submitted to the REF from Sussex. This is not the case with “impact”. The official guidance on submissions states:

Impact: The sub-panels will assess the ‘reach and significance’ of impacts on the economy, society and/or culture that were underpinned by excellent research conducted in the submitted unit, as well as the submitted unit’s approach to enabling impact from its research. This element will carry a weighting of 20 per cent.

The emphasis is mine.

The period during which the underpinning research must have been published is quite generous in length: 1 January 1993 to 31 December 2013. This is clearly intended to recognize the fact that some research take a long time to generate measurable impact. The problem is that the underpinning research must have been done within the submitting unit; it can’t be brought in from elsewhere. If the unit is new and did not exist for most of this period,then it is much harder to generate impact no matter how brilliant the staff it recruits. Any new departments in physics, or any other subject for that matter, will have to focus on research that can generate impact very rapidly indeed if it is to compete in the next REF, expected in 2018 or thereabouts. That is a powerful disincentive for universities to invest in research that may take many years to come to fruition. Five years is a particularly short time in experimental physics.

 

 

Six (very) bad things about the REF

Posted in Education, Science Politics with tags , , , , on November 22, 2013 by telescoper

I see that Jon Butterworth has written a piece on the Grauniad website, entitled Six good things about the REF, the REF in question not being a black-clad figure of questionable parentage and visual acuity responsible for supervising a game of association football, but the Research Excellence Framework.

I agree with some of Jon’s comments and do believe that past Research Assessment Exercises have generally raised the quality of research in UK universities. I do however think that there are some very bad things about the way the REF is being implemented, and that these far outweigh the positives Jon mentions. In the interest of balance, therefore, I thought I’d respond with a list of six (very) bad things about the REF, and particularly how it applies to physics. I’ll keep them brief because I’ve blogged about most of them before:

  1. The rules positively encouraged universities to play games with selectivity. This is absurd. All academic staff on teaching and research contracts should be submitted if a true indication of research quality is to be obtained.
  2. The criteria for what constitutes 3* or 4* publications are vague and subjective, leaving everything in the hands of the panels. Worse, all paperwork will be shredded after the panel’s deliberations leaving no possibility for appeal. This absolutely stinks.
  3. How QR funding will be allocated on the basis of the REF is not made clear in advance of the submission. Nobody knows how heavily the funding will be skewed towards 4* and 3* submissions. Having encouraged departments to play games, therefore, the REF refuses to disclose the rules. It’s not even clear there will be any QR funding.
  4. The panels will be unable to perform a detailed peer review of submissions simply because there will be too many papers. Each panel will be expected to make decisions on many hundreds of papers, leaving time only for a cursory reading of each.
  5. Limiting the physics submission to 4 papers per person is ridiculous. This corresponds to a tiny fraction of the outputs of a typical physics researcher. If someone has written ten 4* publications in the REF period, why should these not be counted?
  6. Impact counts for a sizable fraction (20%) of the funding, but the rules governing what counts as “impact” are absurdly restrictive and clearly encourage short-term commercially-oriented boilerplate stuff at the expense of genuine long-term “blue skies” research.

 

Well, I got to six in just a few minutes and could easily get to sixty, but that will do for now. Perhaps you’d like to contribute your own bad things through the comments box?

The Dark Side of the REF

Posted in Finance, Science Politics with tags , , , , , , , , on August 8, 2013 by telescoper

There’s a disturbing story in the latest Times Higher which argues that the University of Leicester has apparently reneged on a promise that non-submission to the forthcoming (2014)  Research Excellence Framework (REF) would not have negative career consequences. They have now said that except in exceptional circumstances, non-submitted academics will either be moved to a teaching-only contract (where there is a vacancy and they can demonstrate teaching excellence), or have their performance “managed”, with the threat of sacking if they don’t meet the specified targets.  I’d heard rumours of this on the grapevine (i.e. Twitter) before the Times Higher story was published. It’s very worrying to have it confirmed, as it raises all kinds of questions about what might happen in departments that turn out to have disappointing REF results .

There are (at least) two possible reasons for non-inclusion of the outputs of a researcher and it is important to distinguish between them. One is that the researcher hasn’t enough high-quality outputs to submit. In the absence of individual extenuating circumstances, researchers are expected to submit four “outputs” (in my discipline that means “research papers”) for assessment. That’s a pretty minimal level of productivity, actually;  such a number per year is a reasonable average for an active researcher in my field.  A person employed on a contract that specifies their duties as Teaching and Research may therefore be under-performing  if they can’t produce four papers over the period 2008-2013. I think some form of performance management  may be justifiable in this case, but the primary aim should be to help the individual rather than show them the door. We all have fallow periods in research, and it’s not appropriate to rush to sack anyone who experiences a lean time.   Andrew Wiles would have been considered `inactive’ had there been a REF in 1992 as he hadn’t published anything for years. Then he produced a proof of Fermat’s Last Theorem. Some things just take time.

A second reason for excluding researcher from the REF is that the institution concerned may be making a tactical submission. As the Times Higher article explains:

The memo suggests that academics would be spared repercussions if, among other reasons, the number of individuals submitted is “constrained” by the volume of case studies their department intends to enter to demonstrate research impact.

Institutions must submit one case study for every 10 scholars entered.

Maria Nedeva, professor of science and innovation dynamics and policy at Manchester Business School, said the tactic of deciding how many academics to submit based on impact case study numbers was “rife”.

(Incidentally, the second paragraph is not quite right. The number of case studies required depends on the number of staff submitted as follows: for fewer than 15 staff , TWO case studies;  for 15-24.99 staff it is THREE case studies – and then for each additional ten members of staff entered a further case study is required.)

e case study for every scholars included plus one, i.e. forThe statement at the end of the quote there is in line with my experience too.  The point is that the REF is not just a means of allocating relatively small amounts of so-called `QR’ research funding . Indeed, it remains entirely possible that no funding at all will be allocated following the 2014 exercise. The thinking then is that the number of staff submitted is largely irrelevant; all that will count is league table position.

This by no means the only example of the dangers that lurk when you take league tables too seriously.

If a department is required to submit, say, four impact cases if all staff are included in the REF submission, but only has three viable ones, it would not be unreasonable to submit fewer staff because their overall would be dragged down by a poor impact case even if the output quality of all staff is high.  There will certainly be highly active researchers in UK institutions, including many who hold sizable external research grants, whose outputs are not submitted to the REF. As the article points out, it would be very wrong for managers to penalize scholars who have been excluded because of this sort of game-playing. That’s certainly not going to happen in the School of Mathematical and Physical Sciences at Sussex University.  Not while I’m Head of School, anyway.

Moreover, even researchers whose “outputs” are not selected may still contribute to the “Environment” and/or “Impact” sections so they still, in a very real sense, do participate in their department’s REF submission.

My opinion? All this silliness could easily have been avoided by requiring all staff in all units of assessment to be submitted by all departments. You know, like would have happened if the system were actually designed to identify and reward research excellence. Instead, it’s yet another example of a bureaucratic machine that’s become entirely self-serving. It exists simply because it exists.  Research would be much better off without it.