REF 2021 Results and Ranking

The results of the UK’s Research Excellence Framework (REF) 2021 have now been published. You can find them all here at the REF’s own website because they are presented there in a much more informative way than the half-baked “rankings” favoured by, e.g., The Times Higher.

To give some background: the overall REF score for a Unit of Assessment (UoA; usually a Department or School) is obtained by adding three different components: outputs (quality of research papers); impact (referring to the impact beyond academia); and environment (which measures such things as grant income, numbers of PhD students and general infrastructure). Scores are assigned to these categories, e.g. for submitted outputs on a scale of 4* (world-leading), 3* (internationally excellent), 2* (internationally recognized), 1* (nationally recognized) and unclassified. Similar star ratings are applied to the impact and environment. These are weighted at 60%, 25% and 15% respectively in the current incarnation of REF.

You can find further discussion of the REF submission rules, especially concerning changes with respect to 2014, here.

The way the star ratings are often reported is via a Grade Point Average reflecting the percentage of in each band. A hypothetical UoA that scored 100% in the top category would have a GPA of 4.0, for example. One that had 50% 4* and 50* 3* would be 3.5, and so on.

In the 2014 REF institutions were allowed to be selective in the number of staff submitted so the GPA wasn’t really a very appropriate measure: some institutions chose to submit only their very best research in order to get a high GPA. The funding allocated as a result of REF turned out to be highly weighted towards 4* so this was a sensible strategy for them, but it made the simple GPA-based rankings even more meaningless than usual. That didn’t stop e.g. The Times Higher making such rankings though.

This time the rules on selection are stricter so the GPA is arguably more relevant, though many institutions have achieved selectivity anyway by moving certain staff onto teaching-only contracts. Staff on such contracts do not have to be submitted. I note that the main REF website does not use the GPA at all but instead gives profiles like this:

I show the example of Sussex because of my bad memories of the last REF (the 2014 exercise). I had moved to Sussex in 2013 at which point preparations were not well advanced and although everyone concerned worked very hard to put together the best submission for Physics & Astronomy we had to face the problem that our staff numbers had grown significantly in 2013 response to an increase in student numbers. While new staff could bring publications with them they couldn’t bring impact or environment, and while the outputs scored well the latter two categories didn’t so the Department of Physics & Astronomy did poorly in the ensuing rankings.

It must be said however that the primary purpose of the REF (allegedly) is to allocate blocks of funding- the so-called QR funding – to support research in the UoA concerned and while the GPA at Sussex was disappointing the fact that the money depends on the number of staff submitted meant that we got a substantial increase in QR dosh. Note further that the formula for allocation of funds to 4*, 3*, etc is not even specified in advance of the exercise: it is likely to be highly concentrated on research graded 4* and that the funding formula will probably different in England, Wales, Scotland and Northern Ireland. A ranking in terms of money earned is likely to look rather different from one based on GPA.

Another, even more fundamental, problem with the GPA is that the scores are so close together that the differences are of doubtful significance. In the Physics UoA, for example, the gap between top GPA (Sheffield) and 5th place (Bristol) is just 0.05 (3.65 versus 3.60) respectively. I see also that Cardiff is ranked equal 18th (with Imperial College) on a GPA of 3.45.

I say these things just to illustrate how much more subtle the criteria for success are compared to a simple GPA. It’s even hard to tell on an objective basis who to congratulate and who to commiserate.

Anyway, back to Sussex I see that Physics & Astronomy has done far better on environment and impact than last time round and the outputs (95% of which are either 4* or 3*) are comparable to last time (96%) so by those measures they have done well although this might not be reflected in a GPA-based ranking. Sussex is 26th in the rankings with a GPA of 3.35, if you’re interested, which is better than last time, though they will probably be disappointed at the presence of 2* elements in their profile.

Indeed, looking through the Physics list I can’t see any UoA that has a lower GPA this time than last time. The pot of money to be allocated for QR funding is fixed so if every UoA does better that doesn’t mean every UoA gets more money; some institutions will no doubt find that their improved GPA is accompanied by a cut in QR funding.

I’ll end by re-iterating that, having moved to Ireland in 2017, I’m very glad to be out of the path of the bureaucratic juggernaut that is the REF. In its first incarnations (as the Research Assessment Exercise) it did fulfil a useful purpose and did, I believe, improve the quality of UK research. Since then, however, it has become an industry that is largely self-serving. I quote from an article in the Times Higher itself:

The allocation of QR funding could be done in a much simpler and fairer way but the REF is now such a huge edifice it will resist being replaced by something smaller. No doubt before long the staff who spent so much time preparing for REF 2021 will start work on the next exercise. And so it goes on.

The changes in ranking that now occur from exercise to exercise are generally small in magnitude and in number. In other words, huge effort and cost are being invested to discover less and less information.

P.S. For completeness I should say that I am glad we don’t have an equivalent of the REF here in Ireland, we don’t have an equivalent of the QR funding either. This latter is a serious problem for the sustainability of research in third-level institutions, and it is not addressed at all in the recent proposals for reform.

Follow @telescoper

The REF goes on

A few communications with former colleagues from the United Kingdom last week reminded me that, despite the Covid-19 pandemic, the deadline for submissions to the 2021 Research Excellence Framework is next week. It seems very strange to me to push ahead with this despite the Coronavirus disruption, but it’s yet another sign that academics have to serve the bureaucrats rather than the other way round.

I know quite a few people at quite a few institutions that are completely exhausted by the workload required to deal with the enormous exercise in paperwork that is intended to assess the quality and impact of research at UK universities.

With apologies for adding to the stack of memes based on recent events in the Suez Canal, it made me think of this:

One of the major plusses of being in Ireland is that there is no REF, so I’m able to avoid the enormous workload and stress generated by this exercise in bean-counting. That’s good because there are more than enough things on my plate right now, and more are being added every day.

My memories of the last REF in 2014 when I was Head of School at Sussex are quite painful, as it went badly for us then. I hope that the long-term investments we made then will pay off, though, and I hope things turn out better for Sussex this time especially for the Department of Physics & Astronomy for which the impact and environment components of the assessment dragged the overall score down.

Not being involved personally in the REF this time round I haven’t really paid much attention to the changes that have been adopted since 2014. One I knew about is that the rules make it harder for institutions to leave staff out of their REF return. Some universities played the system in 2014 by being very selective about whom they put in. Only staff with papers considered likely to be rated top-notch were submitted.

Having a quick glance at the documents I see two other significant differences.

One is that in 2014, with very few exceptions, all staff had to submit four research outputs (i.e. papers) to be graded. in 2021 the system is more flexible: the total number of outputs must equal 2.5 times the summed FTE (full-time equivalent) of the unit’s submitted staff, with no individual submitting more than 5 and none fewer than 1 (except in special cases related to Covid-19). Overall, then there will be fewer outputs than before, the multiplier of FTE being 2.5 (2021) instead of 4 (2014). There will still be a lot of papers, of course, not least because many Departments have grown since 2014, so the panels will have a great deal of reading to do. If that’s what they do with the papers. They’ll probably just look up citations…

The other difference relates to staff who have left an institution during the census period. In 2014 the institution to which a researcher moved got all the credit for the publications, while the institution they left got nothing. In 2021, institutions “may return the outputs of staff previously employed as eligible where the output was first made publicly available during the period of eligible employment, within the set number of outputs required.” I suppose this is to prevent the departure of a staff member causing too much damage to the institution they left and also to credit the institution where the work was done rather than specifically the individual who did it.

Thinking about the REF an amusing thought occurred to me about Research Assessment. My idea was to set up a sort of anti-REF (perhaps the Research Inferiority Framework) based not on the best outputs produced by an institutions researchers, but on the worst. The institutions producing the highest number of inferior papers could receive financial penalties and get relegated in the league tables for encouraging staff to write too many papers that nobody ever reads or are just plain wrong. My guess is that papers published in Nature might figure even more prominently in this…

Anyway, let me just take this opportunity to wish former colleagues at Cardiff and Sussex all the best for their REF submission on Wednesday 31st March. I hope it turns out well

Follow @telescoper

Three Years in Maynooth

It’s 1st December 2020 which means that it’s now been three whole years since I started my job as Professor of Theoretical Physics in the Department of Theoretical Physics at Maynooth University, in County Kildare in the Irish Republic.

Until the Summer of 2018 I was working part-time at Cardiff and part-time in Maynooth, which required a lot of flying to and from between Wales and Ireland.  That situation would have been impossible to sustain during the pandemic for reasons of quarantine and also because FlyBe went bust. The timing of my move to Maynooth was providential in many ways apart from that.

I didn’t think it would take me the best part of three years to buy a house in Ireland, but owing to a combination of circumstances it took until the end of this summer to do that. Still, all’s well that ends well and I’m very happy with my home.

When I first arrived in Maynooth I stayed in St Patrick’s House (above), part of the Roman Catholic seminary on Maynooth University’s South Campus. I took this picture of the corridor I was on the night I arrived because it reminded me of  The Shining:

The arrival of the Covid-19 Pandemic earlier this year has been another completely unexpected development. I wonder what fate has next in store?

Follow @telescoper

Out of the REF

I was talking over Zoom with some former colleagues from the United Kingdom last week, and was surprised to learn that, despite the Covid-19 pandemic, the 2021 Research Excellence Framework is ploughing ahead next year, only slightly delayed. There’s no stopping bureaucratic juggernauts once they get going…

One of the major plusses of being in Ireland is that, outside the UK academic system, there is no REF. One can avoid the enormous workload and stress generated by this exercise in bean-counting My memories of the last REF in 2014 when I was Head of School at Sussex are quite painful, as it went badly for us then. I hope that the long-term investments we made then will pay off, though, and I hope things turn out better for Sussex this time especially for the Department of Physics & Astronomy for which the impact and environment components of the assessment dragged the overall score down.

The census period for the new REF is 1st August 2013 to 31st July 2020. Not being involved personally in the REF this time round I haven’t really paid much attention to the changes that have been adopted since 2014. One I knew about is that the rules make it harder for institutions to leave staff out of their REF return. Some universities played the system in 2014 by being very selective about whom they put in. Only staff with papers considered likely to be rated top-notch were submitted.

Having a quick glance at the documents I see two other significant differences.

One is that in 2014, with very few exceptions, all staff had to submit four research outputs (i.e. papers) to be graded. in 2021 the system is more flexible: the total number of outputs must equal 2.5 times the summed FTE (full-time equivalent) of the unit’s submitted staff, with no individual submitting more than 5 and none fewer than 1 (except in special cases related to Covid-19). Overall, then there will be fewer outputs than before, the multiplier of FTE being 2.5 (2021) instead of 4 (2014). There will still be a lot, of course, so the panels will have a great deal of reading to do. If that’s what they do with the papers. They’ll probably just look up citations…

The other difference relates to staff who have left an institution during the census period. In 2014 the institution to which a researcher moved got all the credit for the publications, while the institution they left got nothing. In 2021, institutions “may return the outputs of staff previously employed as eligible where the output was first made publicly available during the period of eligible employment, within the set number of outputs required.” I suppose this is to prevent the departure of a staff member causing too much damage to the institution they left.

I was wondering about this last point when chatting with friends the other day. I moved institutions twice during the relevant census period, from Sussex to Cardiff and then from Cardiff to Maynooth. In principle, therefore, both former employees could submit my outputs I published while I was there to the 2021 REF. I only published a dozen or so papers while I was at Sussex – the impact of being Head of School on my research productivity was considerable – and none of them are particularly highly cited so I don’t think that Sussex will want to submit any of them, but they could if they wanted to. They don’t have to ask my permission!

I doubt if Cardiff will be worried about my papers. Among other things they have a stack of gravitational wave papers that should all be 4*.

Anyway, thinking about the REF an amusing thought occurred to me about Research Assessment. My idea was to set up a sort of anti-REF (perhaps the Research Inferiority Framework) based not on the best outputs produced by an institutions researchers but on the worst. The institutions producing the highest number of inferior papers could receive financial penalties and get relegated in the league tables for encouraging staff to write too many papers that nobody ever reads or are just plain wrong. My guess is that papers published in Nature might figure even more prominently in this

Diversity in Physics – LGBTQ+ STEMDay

The nice people involved with Physics at Cardiff University, Prism Exeter and the GW4 group generally have organized a (virtual) event to celebrate Diversity and Inclusion for LGBTQ+ STEM Day 2020 which is to take place on November 18th (that’s next Wednesday). I’m very honoured to have been invited to give a keynote talk at this event, the poster for which is below, and am looking forward to it.

There’s a blog post here that gives more information about the event, including how to register in order to receive the Zoom connection.

I won’t be able to stay for the whole event as I am teaching later that day. I’d have been particularly interested in the session on Open Science…

Follow @telescoper

Moving On..

After attending my second Repeat Examination Board of the week (this one in the Department of Engineering) it’s now time to begin the task of moving the contents of my office into the new one I’ll be in as Head of Department. Roughly simultaneously, the current Head of Department, Jonivar Skullerud, will be moving his clobber from the Head of Department’s office into my current office. Some coordination may be necessary to avoid collisions and/or other confusion, but I’m confident of a successful outcome…

While I’m on the subject of moving to a new job, though in my case remaining at the same institution, this very afternoon my wonderful former colleague from the School of Mathematical and Physical Sciences at the University of Sussex, Dorothy Lamb, is having a leaving do. She will soon be moving to a position at the University of Birmingham (in the Midlands). I’m very sad that I couldn’t be there for her farewell party, but the least I can do is wish Dorothy (aka Miss Lemon) all the best in her new job, and hope that her move from Brighton to Birmingham, after (I think) 25 years, goes as smoothly and as free from stress as possible.

UPDATE: You can read Dorothy’s farewell edition of the MPS Newslettter here.

Dorothy isn’t the only former colleague to be moving on to pastures new. I heard this morning that Ian Harvey and Unai Lopez from the Data Innovation Research Institute at Cardiff University are leaving soon. Unai is taking up a Lectureship at the University of the Basque Country in Bilbao so Bon Voyage Unai!

Follow @telescoper

 

Should students be (financially) compensated for strike action by lecturers?

Regular readers of this blog will know that last year I was still employed part of the time at Cardiff University and during that period I was participating in strike action called by the Universities and Colleges Union (UCU) over pensions. As a result of that action students on my module on Physics of the Early Universe missed quite a lot of lectures (and I was docked a large fraction of my pay).

I refused to do extra lectures after the strike was over to make up for those lost, as it seemed to me that defeated the point of the strike action, but I did make notes available for the students (which I do anyway). Students were also given access to recordings of the previous year’s lectures on the same module. I know some lecturers also adjusted their examinations and/or other assessments to exclude material that had not been covered.

It seems practice for dealing with this (admittedly difficult) situation has varied from institution to institution, and some students who feel that they missed out as a result of the strike have apparently asked to be compensated by their University. Institutions could of course pay compensation to students out of the money saved by not paying lecturers, but that wouldn’t go very far because only a small part of the £9000+ students pay in fees goes to the salaries of teaching staff. Another issue is that I recall one or two students didn’t come to lectures even before the strike started. Should they be compensated too?

Anyway I thought this might be an interesting topic for a poll, so here goes:

As always views are welcome through the comments box too!

Follow @telescoper

The New IOP Physics Technician Award

Picture Credit: Cardiff University School of Physics & Astronomy

I remember a few years ago one of my colleagues when I worked in the School of Physics & Astronomy at Cardiff University, Steven Baker, won an award for being the best STEM Technician in the category of Physical Sciences in the whole country! At the time this was a new award set up by the Higher Education Academy, so Steven was the inaugural winner of it.

Now there’s another new award, this time from the Institute of Physics and dedicated to Physics technicians (not necessarily in universities). I quote:

The IOP Technician Award enables the community to recognise and celebrate the skills and experience of technicians and their contribution to physics.

You can find full details of how to nominate an awardee here. The deadline is 14th June 2019. The prize is worth £1000, but more importantly it serves to encourage Physics departments to reflect on the vital role played by technicians. I feel very strongly that the contribution made by support staff in university departments is drastically undervalued.  No Physics department can run without a dedicated technical support team who apply their skills and expertise in both teaching and research laboratories. Even a department like mine dedicated purely to Theoretical Physics needs computing support, and there are many more people – including clerical staff, library staff, etc – without whom many of our activities would grind to a halt. None of these support staff gets the recognition they deserve; they are often poorly paid and lack an appropriate career structure that reflects the importance of the work they do.

As well as being a nice award this is an opportunity to remind us academics that we couldn’t do what we do without others doing all the difficult stuff!

So please get your nominations in!

Follow @telescoper

Surrey, another unsustainable UK University

A couple of weeks ago I posted an item about planned job losses at Cardiff University during which I remarked that it seemed that a number of other universities are suffering financial problems.

I quote:

There are cold winds blowing through the sector. Many institutions (including Cardiff) have committed to ambitious building programs funded by a combination of borrowing and on optimistic assumptions about growth in student numbers and consequent increases in fee income. Although I no longer work in the UK Higher Education system, I do worry greatly about its sustainability. Even from across the Irish Sea the situation looks extremely precarious: the recent boom could easily end in some institutions going bust. I don’t think that will include Cardiff, by the way. I don’t think the Welsh Government would ever allow that to happen. But I think the English Government wouldn’t act if an English university went bankrupt.

Now the University of Surrey is ringing alarm bells, making £15M cuts and opening up a redundancy scheme `to all staff’. I wonder if that includes the Vice Chancellor, Max Lu, who is quoted on the BBC website:

Mr Lu said: “Some of the main financial challenges include reduced income due to Brexit and an ever more competitive student recruitment environment, significantly increasing pension costs and a national review of tuition fee levels.”

…

Mr Lu added: “Our university also faces the not inconsiderable impact of a fall in our national league table positions.”

The latter shows the ruinously real effect meaningless league tables can have on an institution and, more importantly, the livelihoods of the staff that work there. The whole higher education sector is suffering because of the ideologically-driven attempt to turn it into some sort of market. A crash seems inevitable unless the dangers are recognized and dealt with.

One thing I want to know, though, is: if the University of Surrey’s redundancy scheme is open `to all staff’, what happens if they all take it?

Follow @telescoper

Follow @telescoper

Cardiff Blues: Sustainability and UK Universities

Just before I left on my travels last week I saw a rather depressing news item about Cardiff University. It seems that, after posting a deficit of £22.8 M last year, the University is planning to cut about 380 staff positions. According to the news item:

“The university plans to reduce current staff levels by 7%, or 380 full-time equivalent over five years,” said vice chancellor Colin Riordan in an email to staff.

Since I left Cardiff University in the summer I didn’t get the email from which this is quoted and I don’t know the wider picture. (If anyone would like to forward the V-C’s email to me I’d be very interested.)

The news item also says

Its aim is to get back into surplus by 2019-20 and it wants to cut staffing costs from 59.6% of total income to no more than 56% of income by 2022-23.

Between you and me I was quite surprised that a University can be spending less than 60% of its income on staff, since staff are by far its most valuable resource. Bear in mind also that academic staff will be responsible for only a fraction of this expenditure. In some universities this fraction is only about half. Cutting this still further seems a very retrograde step to me, as it means that student-staff ratios will inevitably rise, making the institution less attractive to prospective students, as well as increasing the workload on existing staff to intolerable levels.

I sincerely hope none of my former colleagues in the School of Physics & Astronomy is affected by the deterioration of the University’s finances. At least the news item I referred to does mention new investments in Data Science, so that is presumably a positive development for the Data Innovation Research Institute with which I was formerly associated.

Incidentally, best wishes to anyone at Cardiff who is reading this, and good luck against England in the Six Nations on Saturday!

I’ve mentioned Cardiff here just because I noticed a specific news item (and I used to work there) but it seems a number of other universities are suffering financial problems. There are cold winds blowing through the sector. Many institutions (including Cardiff) have committed to ambitious building programs funded by a combination of borrowing and on optimistic assumptions about growth in student numbers and consequent increases in fee income. Although I no longer work in the UK Higher Education system, I do worry greatly about its sustainability. Even from across the Irish Sea the situation looks extremely precarious: the recent boom could easily end in some institutions going bust. I don’t think that will include Cardiff, by the way. I don’t think the Welsh Government would ever allow that to happen. But I think the English Government wouldn’t act if an English university went bankrupt.

Follow @telescoper