Archive for Physics & Astronomy

REF 2021 Results and Ranking

Posted in Biographical, Cardiff with tags , , , on May 12, 2022 by telescoper

The results of the UK’s Research Excellence Framework (REF) 2021 have now been published. You can find them all here at the REF’s own website because they are presented there in a much more informative way than the half-baked “rankings” favoured by, e.g., The Times Higher.

To give some background: the overall REF score for a Unit of Assessment (UoA; usually a Department or School) is obtained by adding three different components: outputs (quality of research papers); impact (referring to the impact beyond academia); and environment (which measures such things as grant income, numbers of PhD students and general infrastructure). Scores are assigned to these categories, e.g. for submitted outputs on a scale of 4* (world-leading), 3* (internationally excellent), 2* (internationally recognized), 1* (nationally recognized) and unclassified. Similar star ratings are applied to the impact and environment. These are weighted at 60%, 25% and 15% respectively in the current incarnation of REF.

You can find further discussion of the REF submission rules, especially concerning changes with respect to 2014, here.

The way the star ratings are often reported is via a Grade Point Average reflecting the percentage of in each band. A hypothetical UoA that scored 100% in the top category would have a GPA of 4.0, for example. One that had 50% 4* and 50* 3* would be 3.5, and so on.

In the 2014 REF institutions were allowed to be selective in the number of staff submitted so the GPA wasn’t really a very appropriate measure: some institutions chose to submit only their very best research in order to get a high GPA. The funding allocated as a result of REF turned out to be highly weighted towards 4* so this was a sensible strategy for them, but it made the simple GPA-based rankings even more meaningless than usual. That didn’t stop e.g. The Times Higher making such rankings though.

This time the rules on selection are stricter so the GPA is arguably more relevant, though many institutions have achieved selectivity anyway by moving certain staff onto teaching-only contracts. Staff on such contracts do not have to be submitted. I note that the main REF website does not use the GPA at all but instead gives profiles like this:

I show the example of Sussex because of my bad memories of the last REF (the 2014 exercise). I had moved to Sussex in 2013 at which point preparations were not well advanced and although everyone concerned worked very hard to put together the best submission for Physics & Astronomy we had to face the problem that our staff numbers had grown significantly in 2013 response to an increase in student numbers. While new staff could bring publications with them they couldn’t bring impact or environment, and while the outputs scored well the latter two categories didn’t so the Department of Physics & Astronomy did poorly in the ensuing rankings.

It must be said however that the primary purpose of the REF (allegedly) is to allocate blocks of funding- the so-called QR funding – to support research in the UoA concerned and while the GPA at Sussex was disappointing the fact that the money depends on the number of staff submitted meant that we got a substantial increase in QR dosh. Note further that the formula for allocation of funds to 4*, 3*, etc is not even specified in advance of the exercise: it is likely to be highly concentrated on research graded 4* and that the funding formula will probably different in England, Wales, Scotland and Northern Ireland. A ranking in terms of money earned is likely to look rather different from one based on GPA.

Another, even more fundamental, problem with the GPA is that the scores are so close together that the differences are of doubtful significance. In the Physics UoA, for example, the gap between top GPA (Sheffield) and 5th place (Bristol) is just 0.05 (3.65 versus 3.60) respectively. I see also that Cardiff is ranked equal 18th (with Imperial College) on a GPA of 3.45.

I say these things just to illustrate how much more subtle the criteria for success are compared to a simple GPA. It’s even hard to tell on an objective basis who to congratulate and who to commiserate.

Anyway, back to Sussex I see that Physics & Astronomy has done far better on environment and impact than last time round and the outputs (95% of which are either 4* or 3*) are comparable to last time (96%) so by those measures they have done well although this might not be reflected in a GPA-based ranking. Sussex is 26th in the rankings with a GPA of 3.35, if you’re interested, which is better than last time, though they will probably be disappointed at the presence of 2* elements in their profile.

Indeed, looking through the Physics list I can’t see any UoA that has a lower GPA this time than last time. The pot of money to be allocated for QR funding is fixed so if every UoA does better that doesn’t mean every UoA gets more money; some institutions will no doubt find that their improved GPA is accompanied by a cut in QR funding.

I’ll end by re-iterating that, having moved to Ireland in 2017, I’m very glad to be out of the path of the bureaucratic juggernaut that is the REF. In its first incarnations (as the Research Assessment Exercise) it did fulfil a useful purpose and did, I believe, improve the quality of UK research. Since then, however, it has become an industry that is largely self-serving. I quote from an article in the Times Higher itself:

The allocation of QR funding could be done in a much simpler and fairer way but the REF is now such a huge edifice it will resist being replaced by something smaller. No doubt before long the staff who spent so much time preparing for REF 2021 will start work on the next exercise. And so it goes on.

The changes in ranking that now occur from exercise to exercise are generally small in magnitude and in number. In other words, huge effort and cost are being invested to discover less and less information.

P.S. For completeness I should say that I am glad we don’t have an equivalent of the REF here in Ireland, we don’t have an equivalent of the QR funding either. This latter is a serious problem for the sustainability of research in third-level institutions, and it is not addressed at all in the recent proposals for reform.

The Rising Stars of Sussex Physics

Posted in Bad Statistics, Biographical, Education with tags , , , , on July 28, 2016 by telescoper

This is my penultimate day in the office in the School of Mathematical and Physical Sciences at the University of Sussex, and a bit of news has arrived that seems a nice way to round off my stint as Head of School.

It seems that Physics & Astronomy research at the University of Sussex has been ranked as 13th in western Europe and 7th in the UK by leading academic publishers, Nature Research, and has been profiled as one of its top-25 “rising stars” worldwide.

I was tempted to describe this rise as ‘meteoric’ but in my experience meteors generally fall down rather than rise up.

Anyway, as regular readers of this blog will know, I’m generally very sceptical of the value of league tables and there’s no reason to treat this one as qualitatively any different. Here is an explanation of the (rather curious) methodology from the University of Sussex news item:

The Nature Index 2016 Rising Stars supplement identifies the countries and institutions showing the most significant growth in high-quality research publications, using the Nature Index, which tracks the research of more than 8,000 global institutions – described as “players to watch”.

The top 100 most improved institutions in the index between 2012 and 2015 are ranked by the increase in their contribution to 68 high-quality journals. From this top 100, the supplement profiles 25 rising stars – one of which is Sussex – that are already making their mark, and have the potential to shine in coming decades.

The institutions and countries examined have increased their contribution to a selection of top natural science journals — a metric known as weighted fractional count (WFC) — from 2012 to 2015.

Mainly thanks to a quadrupling of its physical sciences score, Sussex reached 351 in the Global 500 in 2015. That represents an 83.9% rise in its contribution to index papers since 2012 — the biggest jump of any UK research organisation in the top 100 most improved institutions.

It’s certainly a strange choice of metric, as it only involves publications in “high quality” journals, presumably selected by Journal Impact Factor or some other arbitrary statistical abominatio,  then taking the difference in this measure between 2012 and 2015  and expressing the change as a percentage. I noticed one institution in the list has improved by over 4600%, which makes Sussex’s change of 83.9% seem rather insignificant…

But at least this table provides some sort of evidence that the investment made in Physics & Astronomy over the last few years has made a significant (and positive) difference. The number of research faculty in Physics & Astronomy has increased by more than 60%  since 2012 so one would have been surprised not to have seen an increase in publication output over the same period. On the other hand, it seems likely that many of the high-impact papers published since 2012 were written by researchers who arrived well before then because Physics research is often a slow burner. The full impact of the most recent investments has probably not yet been felt. I’m therefore confident that Physics at Sussex has a very exciting future in store as its rising stars look set to rise still further! It’s nice to be going out on a high note!

 

 

League Table Positions

Posted in Education, Football with tags , , , , , , , on May 26, 2015 by telescoper

Among the things I didn’t have time to blog about over a very busy Bank Holiday Weekend was the finish of the English Premiership season. I haven’t posted much about my own team, Newcastle United, this season because I haven’t been able to think of anything particularly positive to say. Since Alan Pardew quit in January to join Crystal Palace, Newcastle slumped to such an alarming extent that they went into their last game of the season (against West Ham) just two points above the drop zone. Had they lost their game, which did not seem unlikely on the basis of their recent form, and had Hull won against Manchester United, which did not seem unlikely on the grounds that Man Utd wwould finish in 4th place whatever happened in that game, then Newcastle would be relegated to the Championship. In the event, however, Newcastle won 2-0 which made them safe while Hull could only draw 0-0 which meant that Newcastle would have survived even if they had lost against West Ham. Moreover, Sunderland also lost their last game, which meant that the final Premier League Table looked like this:

Premiership_League

(courtesy of the BBC Website). The important places are 15 and 16, obviously. The natural order of things has been restored….

Another League Table came out over the Bank Holiday. This was the annual Guardian University Guide. I’m deeply sceptical of the value of these league tables, but there’s no question that they’re very important to potential students so we have to take them seriously. This year was pretty good for Sussex as far as the Guardian Table is concerned: the University of Sussex rose to 19th place overall and the two departments of the School of Mathematical and Physical Sciences both improved: Physics & Astronomy is back in the top 10 (at number 9, up from 11th place last year) and Mathematics rose 22 places to take 21st place. Gratifyingly, both finished well above Sunderland.

While these results are good news in themselves, at least around my neck of the woods, as they will probably lead to increased applications to Sussex from students next year, it is important to look behind the simplistic narrative of “improvements”. Since last year there have been several substantial changes to the Guardian’s methodology. The weighting given to “spend-per-student” has been reduced from 15% to 10% of the overall score and the method of calculating “value added” has excluded specific predictions based on “non-tariff” students (i.e. those without UK entry qualifications, especially A-levels). What the Guardian consistently fails to do is explain the relative size of the effect of arbitrary methodological changes on its tables compared to actual changes in, e.g., cash spent per student.

Imagine the outrage there would be if football teams were not told until the end of a Premier League season how many points would be awarded for a win….

Combining Research and Teaching in Physics & Astronomy

Posted in Education, The Universe and Stuff with tags , , , , , on August 7, 2014 by telescoper

Among the distinctive things we do here in the School of Mathematical and Physical Sciences at the University of Sussex are our degree programmes that involve a Research Placement (RP). Students on these courses take the normal lectures, laboratory classes and workshops during the academic year, but they spend the summer vacation doing (paid) work with research groups in the School to get an experience of what the world of research is really like. Various combinations of Physics and Astronomy with a Research Placement have been around for some time. These courses have been so popular and successful that we’ve extended the idea to Mathematics for 2015 entry. We have also started extending the RP scheme to include placements in laboratories elsewhere, either in industry or in a university abroad; we even have two students currently doing their placements in China.

Here are a couple of videos we’ve made featuring two RP students who have been working in the Department of Physics & Astronomy this summer.

This is Ross Callaghan:

And this is Nathaniel Wiesendanger Shaw:

Both these students are in between their 2nd and 3rd years of a 4-year MPhys programme. As it happens, both survived the experience of being in my Theoretical Physics class last term too!

It’s an ongoing frustration of mine that so many influential people think that teaching and research are separate functions of a university and should not be mixed. I believe that the two go hand-in-hand and that you can’t really claim to be getting a real university education if it’s not informed by the latest developments in research. Moreover, some also imply that research-led teaching only happens in the Russell Group, which is not the case at all. In fact, I think we provide a much better environment for this in Sussex than either of the Russell Group universities in which I’ve previously worked.

Many Departments talk about how important it is that their teaching is based on state-of-the-art research, but here at Sussex we don’t just talk about research to undergraduates – we let them do it!

The Busyness of Examination Time

Posted in Biographical, Education with tags , , , , , on June 12, 2014 by telescoper

Just time this evening for the briefest of brief posts. This is probably the busiest week of the year at the University of Sussex, and it’s not over yet. The main reason for this busyness is the business of examinations, assessment and degree classification.

This morning we had our meeting of the School Progression and Award Board for Years 3 and 4 at which, among other things, we sorted out the classification for honours of our graduating students. This involves distilling the marks gained over several years of assessments down to a final “Grand Mean”. It’s not a trivial process but I’m glad to say it went off very smoothly.

The pass lists have now gone to be officially signed off by the University administration. They will be posted tomorrow at noon, at which time we’ll have a celebratory drink or several ready for those getting their results.

One of my duties as Head of School is to chair this meeting, but I don’t take credit for the successful running of the meeting because all the hard work of preparation was done by our excellent office staff, especially Oonagh and Chrystelle.

That done there was time for a quick sandwich lunch before heading off to Stanmer House for a teaching “away afternoon” for the Department of Physics & Astronomy, at which we discussed ideas for improvements to the way we teach and assess students.

Stanmer

I’m actually in the group sitting under the parasol in the left foreground.

Stanmer House is set in beautiful parkland just a short walk from Sussex University. I took the more strenuous route over the hill, but am glad I did so because the view was so nice in the glorious sunshine and it made be realise I don’t make as much of the opportunity for walking around the campus as I should.

Tomorrow is going to be another busy day but, if you’ll excuse me, I’m now going to have a glass of chilled white wine and a bite to eat.

A Summer of Undergraduate Physics Research

Posted in Biographical, Education, The Universe and Stuff with tags , , , , on January 14, 2014 by telescoper

I was compiling a bibliography for a new paper yesterday and noticed that a paper published in December 2013 cited one I wrote in 2005 with Andrew Stannard while I was working at the University of Nottingham; the rarity of anyone actually referring to any of my papers caught my attention. I include the abstract of the Stannard-Coles paper here for reference:

We investigate the properties of the (complex) coefficients obtained in a spherical harmonic representation of temperature maps of the cosmic microwave background (CMB). We study the effect of the coefficient phase only, as well as the combined effects of phase and amplitude. The method used to check for anomalies is to construct a `random walk’ trajectory in the complex plane where the step length and direction are given by the amplitude and phase (respectively) of the harmonic coefficient. If the fluctuations comprise a homogeneous and isotropic Gaussian random field on the sky, the path so obtained should be a classical `Rayleigh flight’ with very well known statistical properties. We illustrate the use of this random-walk representation by using the net walk length as a test statistic, and apply the method to the coefficients obtained from a Wilkinson Microwave Anisotropy Probe (WMAP) preliminary sky temperature map.

This seems like ancient history now, particularly with regard to the use of “preliminary” WMAP data, but it was only just over 8 years ago. I never imagined during the time that we were working on this paper that I’d be moving twice! By contrast, Andrew Stannard remained in Nottingham, doing a PhD there and is now employed as a Research Fellow, although he switched fields after the project and moved into nanoscience.

Anyway, I wasn’t posting this so I could take a trip down memory lane. I thought I’d post it in order to point out that this paper actually came about as an undergraduate research project that Andrew did under my supervision during the summer of 2005, funded by a Mary Cannell Summer Studentship. Mary Cannell was the author of a book about the life of George Green (the famous mathematician who came from Nottingham); she passed away in 2000 leaving some money to the School of Physics & Astronomy to fund summer research placements for undergraduates. If I recall correctly, we completed the analysis and wrote the paper during the summer of 2005, submitting it in August and having it accepted in September. If only it were always so straightforward! Since publication it has garnered 15 citations according to the ADS website; not exactly earth-shattering, but respectable enough, especially given the background. I think it may get a few more in the next few years because the quality of data from Planck may now be good enough to actually detect the features we were looking for all those years ago!

It’s worth mentioning that in the School of Physics and Astronomy at the University of Sussex we have a degree programme in which students receive a stipend to cover living expenses during a summer vacation placement with one of the research groups each and every year of their studies. This is in addition to the usual lectures and laboratory work of the standard  course. This involves many more students than was the case back in Nottingham in 2005, but since I’ve only been at the University of Sussex for  a year I don’t know how many such placements have led to actual publications.

Does anyone know of any really important papers out there that came from undergraduate research projects? If so, please let me know through the comments box..

Meet the Panel …

Posted in Education, Science Politics with tags , , , , , , , on February 21, 2011 by telescoper

Just a quick post to pass on the news that the Higher Education Funding Council for England (HEFCE) has announced the list of panel members for the forthcoming Research Excellence Framework (REF), a massive exercise in bean counting which will drag on until 2014.

Much as I enjoy ploughing through HEFCE’s fascinating documents, in this case I went straight to the Physics (& Astronomy) sub-panel, which is:

All estimable folk and a good selection of different expertise. There’s also a good geographical spread with members from the English regions, Scotland, Northern Ireland, and of course Wales. Oh, wait a minute. Not Wales. Apparently Wales doesn’t merit any representation on the Physics REF panel. Nor did it last time. Why am I thinking to myself “here we go again”?

To be perfectly honest, I don’t really understand why Welsh universities are being forced to take part in the REF anyway. Or those from Scotland and Northern Ireland for that matter. The REF is driven by an English agenda which is certainly at variance with Welsh priorities. Whereas in England, HEFCE is allocating funding using a formula involving an exceedingly steep weighting towards “internationally leading” research, here in Wales the equivalent body HEFCW is resisting the urge to concentrate research cash so heavily according to such a doubtful measure of research quality.

And don’t get me started on the so-called “impact” measures. All I can say about them is that Kafka would have been proud.

The Welsh Assembly Government has recently taken steps to protect Welsh students against the effects of Higher Education cuts imposed by Westminster. However, there will be substantial cuts in resource to Welsh universities in order to pay for this. At the same time as making “efficiency savings”, as is appropriate for the age of austerity, we’re also being forced to participate in a monstrously wasteful bureaucratic exercise of little relevance to the needs or aspirations of Welsh universities.

I think there’s a strong case for HEFCW to show a bit of real independence and withdraw from the REF altogether.


Share/Bookmark