Archive for Universities

More Worthless University Rankings

Posted in Bad Statistics, Education with tags , , , on September 6, 2017 by telescoper

The Times Higher World University Rankings, which were released this week. The main table can be found here and the methodology used to concoct them here.

Here I wish to reiterate the objection I made last year and the year before that to the way these tables are manipulated year on year to create an artificial “churn” that renders them unreliable and impossible to interpret in any objective way. In other words, they’re worthless. This year the narrative text includes:

This year’s list of the best universities in the world is led by two UK universities for the first time. The University of Oxford has held on to the number one spot for the second year in a row, while the University of Cambridge has jumped from fourth to second place.

Overall, European institutions occupy half of the top 200 places, with the Netherlands and Germany joining the UK as the most-represented countries. Italy, Spain and the Netherlands each have new number ones.

Another notable trend is the continued rise of China. The Asian giant is now home to two universities in the top 30: Peking and Tsinghua. The Beijing duo now outrank several prestigious institutions in Europe and the US. Meanwhile, almost all Chinese universities have improved, signalling that the country’s commitments to investment has bolstered results year-on-year.

In contrast, two-fifths of the US institutions in the top 200 (29 out of 62) have dropped places. In total, 77 countries feature in the table.

These comments are all predicated on the assumption that any changes since the last tables represent changes in data (which in turn are assumed to be relevant to how good a university is) rather than changes in the methodology used to analyse that data. Unfortunately, every single year the Times Higher changes its methodology. This time we are told:

This year, we have made a slight improvement to how we handle our papers per academic staff calculation, and expanded the number of broad subject areas that we use.

What has been the effect of these changes? We are not told. The question that must be asked is how can we be sure that any change in league table position for an institution from year to year represents a change in “performance”,rather than a change in the way metrics are constructed and/or combined? Would you trust the outcome of a medical trial in which the response of two groups of patients (e.g. one given medication and the other placebo) were assessed with two different measurement techniques?

There is an obvious and easy way to test for the size of this effect, which is to construct a parallel set of league tables, with this year’s input data but last year’s methodology, which would make it easy to isolate changes in methodology from changes in the performance indicators. The Times Higher – along with other purveyors of similar statistical twaddle – refuses to do this. No scientifically literate person would accept the result of this kind of study unless the systematic effects can be shown to be under control. There is a very easy way for the Times Higher to address this question: all they need to do is publish a set of league tables using, say, the 2016/17 methodology and the 2017/18 data, for comparison with those constructed using this year’s methodology on the 2017/18 data. Any differences between these two tables will give a clear indication of the reliability (or otherwise) of the rankings.

I challenged the Times Higher to do this last year, and they refused. You can draw your own conclusions about why.

P.S. For the record, Cardiff University is 162nd in this year’s table, a rise of 20 places on last year. My former institution, the University of Sussex, is up two places to joint 147th. Whether these changes are anything other than artifacts of the data analysis I very much doubt.

Advertisements

The Worthless University Rankings

Posted in Bad Statistics, Education with tags , , , on September 23, 2016 by telescoper

The Times Higher World University Rankings, which were released this weekk. The main table can be found here and the methodology used to concoct them here.

Here I wish to reiterate the objection I made last year to the way these tables are manipulated year on year to create an artificial “churn” that renders them unreliable and impossible to interpret in an objective way. In other words, they’re worthless. This year, editor Phil Baty has written an article entitled Standing still is not an option in which he makes a statement that “the overall rankings methodology is the same as last year”. Actually it isn’t. In the page on methodology you will find this:

In 2015-16, we excluded papers with more than 1,000 authors because they were having a disproportionate impact on the citation scores of a small number of universities. This year, we have designed a method for reincorporating these papers. Working with Elsevier, we have developed a new fractional counting approach that ensures that all universities where academics are authors of these papers will receive at least 5 per cent of the value of the paper, and where those that provide the most contributors to the paper receive a proportionately larger contribution.

So the methodology just isn’t “the same as last year”. In fact every year that I’ve seen these rankings there’s been some change in methodology. The change above at least attempts to improve on the absurd decision taken last year to eliminate from the citation count any papers arising from large collaborations. In my view, membership of large world-wide collaborations is in itself an indicator of international research excellence, and such papers should if anything be given greater not lesser weight. But whether you agree with the motivation for the change or not is beside the point.

The real question is how can we be sure that any change in league table position for an institution from year to year are is caused by methodological tweaks rather than changes in “performance”, i.e. not by changes in the metrics but by changes in the way they are combined? Would you trust the outcome of a medical trial in which the response of two groups of patients (e.g. one given medication and the other placebo) were assessed with two different measurement techniques?

There is an obvious and easy way to test for the size of this effect, which is to construct a parallel set of league tables, with this year’s input data but last year’s methodology, which would make it easy to isolate changes in methodology from changes in the performance indicators. The Times Higher – along with other purveyors of similar statistical twaddle – refuses to do this. No scientifically literate person would accept the result of this kind of study unless the systematic effects can be shown to be under control. There is a very easy way for the Times Higher to address this question: all they need to do is publish a set of league tables using, say, the 2015/16 methodology and the 2016/17 data, for comparison with those constructed using this year’s methodology on the 2016/17 data. Any differences between these two tables will give a clear indication of the reliability (or otherwise) of the rankings.

I challenged the Times Higher to do this last year, and they refused. You can draw your own conclusions about why.

Staff Whereabouts

Posted in Education with tags , , , on April 19, 2016 by telescoper

There’s been a predictably strong reaction from academic colleagues to an announcement by the University of Edinburgh that it is introducing a new staff monitoring policy that will require employees to tell management if they leave their “normal place of work” for half a day or longer.

Some have argued that this is measure is simply unenforceable and that the University concerned will have to employ extra people if all academics have to notify a management person every time they travel somewhere off campus. Perhaps the plan is to have all staff fitted with microchips like we do with pets so we can find out where they are if they go wandering off, or get temporarily adopted by friendly neighbours.

I did some time ago draft an April Fool email in which I claimed  my current employer was going to extend the attendance monitoring we perform with undergraduate students (which is partly to assess usage of teaching spaces and thus improve timetabling efficiency) to include academic staff, so we could assess usage of office space on a similar basis. I never sent the email because I thought too many would think it was real and get very angry. Although being at least slightly credible is an essential part of an April Fool, causing a riot is not.

Here at the University of Sussex academic staff are obliged to inform the University (via an official form) if they are travelling elsewhere in the course of their duties. In practice this form comes to the Head of School, which is me in in the case of Mathematical and Physical Sciences. This bit of bureaucracy is primarily for insurance purposes, but also means we have a record of where to contact people in case of emergency.

Most staff comply with this procedure  if they are travelling abroad, but they don’t always do so when they’re travelling in the UK for a day or so, e.g. for doing a PhD examination or something like that. Staff also often fail to let us know if they are working from home, which some (especially theorists and mathematicians) do a lot in order to get on with their research without interruption. Although this doesn’t often cause problems, I think it is reasonable that we should be able to get in touch with staff when they’re doing that (in case, e.g., one of their academic advisees has a problem) but it seems to me excessive that they should have to inform someone at an official level every time they work off campus for whatever reason. Leaving a contact phone number for use during working hours is quite adequate.

It seems to me that behind this move by the University of Edinburgh there’s the managerialist suspicion that everyone must be a shirker at heart. In fact one of the problems I have as a manager is not persuading staff to work longer hours, but to  stop working  excessively long hours.  I don’t think I’ve succeeded, largely because I haven’t found a way of doing my job at the same time as achieving  a sensible work-life balance.

Anyway, the point is that academic contracts do not usually specify where staff should work. There is a good reason for this, which is that the job is very diverse and replies flexible work arrangements. Academic contracts do not usually specify fixed hours of work per week, either, for the same reason. Some don’t even give a specific holiday entitlement. Staff in technical and professional service areas generally have contracts that specify both. I floated an idea at a staff that academics should file an official log of official leave. It wasn’t a popular suggestion because academic staff thought there was an implication that they were skiving by taking excessively long holidays. In fact my motivation was quite the opposite: to try to ensure that they take all the leave to which they are entitled.

 

 

Research Funding – A Modest Proposal

Posted in Education, Science Politics with tags , , , , , on September 9, 2015 by telescoper

This morning, the Minister for Universities, Jo Johnson, made a speech in which, among other things, he called for research funding to be made simpler. Under the current “dual funding” system, university researchers receive money through two main routes: one is the Research Excellence Framework (REF) which leads to so-called “QR” funding allocations made via the Higher Education Funding Council for England (HEFCE); and the other is through research grants which have to be applied for competitively from various sources, including the Seven Research Councils.

Part of the argument why this system needs to be simplified is the enormous expense and administrative burden of the Research Excellence Framework.  Many people have commented to me that although they hate the REF and accept that it’s ridiculously expensive and time-consuming, they didn’t see any alternative. I’ve been thinking about it and thought I’d make a suggestion. Feel free to shoot it down in flames through the box at the end, but I’ll begin with a short introduction.

Those of you old enough to remember will know that before 1992 (when the old `polytechnics’ were given the go-ahead to call themselves `universities’) the University Funding Council – the forerunner of HEFCE – allocated research funding to universities by a simple formula related to the number of undergraduate students. When the number of universities suddenly increased this was no longer sustainable, so the funding agency began a series of Research Assessment Exercises to assign research funds (now called QR funding) based on the outcome. This prevented research money going to departments that weren’t active in research, most (but not all) of which were in the ex-Polytechnics. Over the years the apparatus of research assessment has become larger, more burdensome, and incomprehensibly obsessed with short-term impact of the research. Like most bureaucracies it has lost sight of its original purpose and has now become something that exists purely for its own sake.

It is especially indefensible at this time of deep cuts to university core funding that we are being forced to waste an increasingly large fraction of our decreasing budgets on staff-time that accomplishes nothing useful except pandering to the bean counters.

My proposal is to abandon the latest manifestation of research assessment mania, i.e. the REF, and return to a simple formula, much like the pre-1992 system,  except that QR funding should be based on research student (i.e. PhD student) rather than undergraduate numbers. There’s an obvious risk of game-playing, and this idea would only stand a chance of working at all if the formula involved the number of successfully completed research degrees over a given period .

I can also see an argument  that four-year undergraduate students (e.g. MPhys or MSci students) also be included in the formula, as most of these involve a project that requires a strong research environment.

Among the advantages of this scheme are that it’s simple, easy to administer, would not spread QR funding in non-research departments, and would not waste hundreds of millions of pounds on bureaucracy that would be better spent actually doing research. It would also maintain the current “dual support” system for research, if that’s  a benefit.

I’m sure you’ll point out disadvantages through the comments box!


Why Research Loans Should Replace Grants For Commercially-Driven Research

Posted in Education, Politics, Science Politics with tags , , , on August 4, 2015 by telescoper

Two recent items in the Times Higher about UK Higher education – concerning the abolition of maintenance grants for less well-off students and whether business should contribute more to the cost of research reminded me of a post I wrote almost exactly five years ago. Was it really so long ago? Anyway, I am old so I am allowed to repeat myself even if people aren’t listening, so here’s the gist of the argument I made way back then….

Universities essentially do two things, teaching and research. However, when you think about it, there’s a fundamental inconsistency in the way these are funded. It seems to me that correcting this anomaly could significantly improve  both the main benefits  universities contribute to the UK economy.

First, research. If  research is going to pay off in the short term it should be funded by private investors, interested businesses or venture capitalists of some sort. Dragon’s Den, even. When the public purse is so heavily constrained, it should only be asked to fund those things that can’t in practice be funded any other way. This is pretty much the opposite of what the Treasury thinks. It wants to concentrate public funds in projects that  can demonstrate immediate commercial potential. Taxpayer’s money used in this way either ends up in the pockets of entrepreneurs if the research succeeds or, if it doesn’t,  the grant has effectively been wasted. It’s yet another example of the taxpayer bearing the risk that an investment might fail, but not sharing in the benefits if it succeeds. This is analogous to the way  the taxpayer bailed out the banking sector in the aftermath of the Credit Crunch in 2008, only to see the profits subsequently transferred back into private hands.  This is happening to an increasing extent elsewhere in the United Kingdom, as public services built up through state investment are being transferred to for-profit organizations. Even our beloved National Health Service seems to be on an irreversible path to privatization.

My proposal for research funding would involve phasing out research grants for groups that want to concentrate on commercially-motivated research and replace them with research loans. If the claims the researchers  make to secure the advance are justified, they should have no problem repaying it  from the profits they make from patent income, commercial sales,  or other forms of exploitation. If not, then they will have to pay back the loan from their own funds (as well as being exposed for having made over-optimistic claims). In the current economic situation the loans could be made at very low interest rates and still save a huge amount of the current research budget for higher education. Indeed after a few years – I suggest the loans should be repayable in 3-5 years –  it would be self-financing. I think a large fraction of research in the Applied Sciences and Engineering should be funded in this way.

The money saved by replacing grants  to commercially driven research with loans could be re-invested in those areas where public investment is really needed, such as pure science and medicine. Here grants are needed because the motivation for the research is different. Much of it does, in fact, lead to commercial spin-offs, but that is accidental and likely to appear only in the very long term. The real motivation of doing this kind of research is to enrich the knowledge base of the UK and the world in general. In other words, it’s for the public good. Remember that?

Most of you probably think that this is a crazy idea, but if you do I’ll ask you to now think about how the government funds teaching in universities and ask yourself why research is handled in such a  different way.

Way back in the mists of time when I was a student, I didn’t have to pay fees and even got a maintenance grant from the government that was more-or-less sufficient to live on. That system changed so that students don’t get grants any more, but may qualify for loans. They also have to pay fees. The government only pays an amount directly to the university on their behalf if they are studying an “expensive” subject, i.e. a laboratory-based science, and that amount is very small (and decreasing with time). This change of policy happened because the (then) Labour government wanted to boost the rate of participation in universities, but didn’t think the taxpayer should pay the whole cost. The logic goes that the students benefit from their education, e.g. in terms of increased earnings over their working lifetime, so they should pay a contribution to it. The policy has changed since then into one in which many students bear the full cost of their tuition.

I don’t come from a wealthy family background so it’s not clear whether I would have been able to go to University under the current system. I would have been prepared to borrow to fund tuition fees, but without a maintenance grant for day-to-day living I don’t think I could have afforded it as my parents could not have supported me financially. In my opinion the removal of maintenance grants is far more likely to deter students from poorer backgrounds from going to University than the introduction of fees.

Anyway, the problem with all these changes is that they have led to a huge increase in enrolment on degree courses in “vocational” areas such as Leisure & Tourism, Media Studies, and Business while traditional courses, such as those in STEM disciplines, providing the sort of rigorous intellectual training that is essential for many sectors of the economy, have struggled to keep up. This is partly because subjects like Mathematics and Physics are difficult, partly because they are expensive, and partly because the UK school system has ceased to provide adequate preparation for such courses. I’m by no means against universities supplying training in vocational subjects, but because these are the areas where the primary beneficiary is indeed the student, I don’t think the government should subsidise them as much as the more rigorous courses that we really need to encourage the brightest students to take up. Universities are not just for training. They have a much deeper purpose.

If it’s fair to ask students to contribute to their teaching, it’s fair to ask commercial companies to pay for the research that they exploit. Just as student grants should be re-introduced for certain disciplines, so should research loans be introduced for others. You know it makes sense.

However, if you want to tell me why it doesn’t, via the comments box, please feel free!

The Renewed Threat to STEM

Posted in Education, Finance, Science Politics with tags , , , , , , on July 26, 2015 by telescoper

A couple of years ago, soon after taking over as Head of the School of Mathematical and Physical Sciences (MPS) at the University of Sussex, I wrote a blog post called The Threat to STEM from HEFCE’s Funding Policies about how the funding policies of the Higher Education Funding Council for England (HEFCE) were extremely biased against STEM disciplines. The main complaint I raised then was that the income per student for science subjects does not adequately reflect the huge expense of teaching these subjects compared to disciplines in the arts and humanities. The point is that universities now charge the same tuition fee for all subjects (usually £9K per annum) while the cost varies hugely across disciplines: science disciplines can cost as much as £16K per annum per student whereas arts subjects can cost as little as £6K. HEFCE makes a small gesture towards addressing this imbalance by providing an additional grant for “high cost” subjects, but that is only just over £1K per annum per student, not enough to make such courses financially viable on their own. And even that paltry contribution has been steadily dwindling.  In effect, fees paid by arts students are heavily subsidising the sciences across the Higher Education sector.

The situation was bad enough before last week’s announcement of an immediate £150M cut in HEFCE’s budget. Once again the axe has fallen hardest on STEM disciplines. Worst of all, a large part of the savings will be made retrospectively, i.e. by clawing back money that had already been allocated and which institutions had assumed in order to plan their budgets. To be fair, HEFCE had warned institutions that cuts were coming in 2015/16:

This means that any subsequent changes to the funding available to us from Government for 2015-16, or that we have assumed for 2016-17, are likely to affect the funding we are able to distribute to institutions in the 2015-16 academic year. This may include revising allocations after they have already been announced. Accordingly, institutions should plan their budgets prudently.

However, this warning does not mention the possibility of cuts to the current year (i.e. 2014-15). No amount of prudent planning of budgets will help when funding is taken away retrospectively, as it is now to the case. I should perhaps explain that funding allocations are made by HEFCE in a lagged fashion, based on actual student numbers, so that income for the academic year 2014-15 is received by institutions during 15/16. In fact my institution, in common with most others, operates a financial year that runs from August 1st to July 31st and I’ve just been through a lengthy process of setting the budget from August 1st 2015 onward; budgets are what I do most of the time these days, if I’m honest. I thought I had finished that job for the time being, but look:

In October 2015, we will notify institutions of changes to the adjusted 2014-15 teaching grants we announced in March 20158. These revised grant tables will incorporate the pro rata reduction of 2.4 per cent. This reduction, and any other changes for individual institutions to 2014-15 grant, will be implemented through our grant payments from November 2015. We do not intend to reissue 2014-15 grant tables to institutions before October 2015, but institutions will need to reflect any changes relating to 2014-15 in their accounts for that year (i.e. the current academic year). Any cash repayments due will be confirmed as part of the October announcements.

On top of this, any extra students recruited as as  result of the government scrapping student number controls won’t attract any support at all from HEFCE, so we wll only get the tuition fee.And the government says it wants the number of STEM students to increase? Someone tell me how that makes sense.

What a mess! It’s going to be back to the drawing board for me and my budget. And if a 2.4 per cent cut doesn’t sound much to you then you need to understand it in terms of how University budgets work. It is my job – as the budget holder for MPS – to ensure that the funding that comes in to my School is spent as efficiently and effectively on what the School is meant to do, i.e. teaching and research. To that end I have to match income and expenditure as closely as possible. It is emphatically not the job of the School to make a profit: the target I am given is to return a small surplus (actually 4 per cent of our turnover) to contribute to longer-term investments. I’ve set a budget that does this, but now I’ll have to wait until October to find out how much I have to find in terms of savings to absorb the grant cut. It’s exasperating when people keep moving the goalposts like this. One would almost think the government doesn’t care about the consequences of its decisions, as long as it satisfies its fixation with cuts.

And it’s not only teaching that is going to suffer. Another big slice of savings (£52M) is coming from scrapping the so-called “transitional relief” for STEM departments who lost out as a result of the last Research Excellence Framework. This again is a policy that singles out STEM disciplines for cuts. You can find the previous allocations of transitional relief in an excel spreadsheet here. The cash cuts are largest in large universities with big activities in STEM disciplines – e.g. Imperial College will lose £10.9M previous allocated, UCL about £4.3M, and Cambridge about £4M. These are quite wealthy institutions of course, and they will no doubt cope, but that doesn’t make it any more acceptable for HEFCE to break a promise.

This cut in fact won’t alter my School’s budget either. Although we were disappointed with the REF outcome in terms of league table position, we actually increased our QR income. As an institution the University of Sussex only attracted £237,174 in transitional relief so this cut is small potatoes for us, but that doesn’t make this clawback any more palatable from the point of view of the general state of health of STEM disciplines in the United Kingdom.

These cuts are also directly contrary to the claim that the UK research budget is “ring-fenced”. It clearly isn’t, and with a Comprehensive Spending Review coming up many of us are nervous that these cuts are just a foretaste of much worse things to come. Research Councils are being asked to come up with plans based on a 40% cut in cash.

Be afraid. Be very afraid.

Yes, science produces too many PhDs

Posted in Science Politics with tags , , , , , on February 19, 2015 by telescoper

I came across a blog post this morning entitled Does Science Produce Too Many PhDs? I think the answer is an obvious “yes” but I’ll use the question as an excuse to rehash an argument I have presented before, which is that most analyses of the problems facing yearly career researchers in science are looking at the issue from the wrong end. I think the crisis is essentially caused by the overproduction of PhDs in this field. To understand the magnitude of the problem, consider the following.

Assume that the number of permanent academic positions in a given field (e.g. astronomy) remains constant over time. If that is the case, each retirement (or other form of departure) from a permanent position will be replaced by one, presumably junior, scientist.

This means that over an academic career, on average, each academic will produce just one PhD who will get a permanent job in academia. This of course doesn’t count students coming in from abroad, or those getting faculty positions abroad, but in the case of the UK these are probably relatively small corrections.

Under the present supply of PhD studentships an academic can expect to get a PhD student at least once every three years or so. At a minimum, therefore, over a 30 year career one can expect to have ten PhD students. A great many supervisors have more PhD students than this, but this just makes the odds worse. The expectation is that only one of these will get a permanent job in the UK. The others (nine out of ten, according to my conservative estimate) above must either leave the field or the country to find permanent employment.

The arithmetic of this situation is a simple fact of life, but I’m not sure how many prospective PhD students are aware of it. There is still a reasonable chance of getting a first postdoctoral position, but thereafter the odds are stacked against them.

The upshot of this is we have a field of understandably disgruntled young people with PhDs but no realistic prospect of ever earning a settled living working in the field they have prepared for. This problem has worsened considerably in recent  years as the number of postdoctoral positions has almost halved since 2006. New PhDs have to battle it out with existing postdoctoral researchers for the meagre supply of suitable jobs. It’s a terrible situation.

Now the powers that be – in this case the Science and Technology Facilities Council – have consistently argued that the excess PhDs go out into the wider world and contribute to the economy with the skills they have learned. That may be true in a few cases. However, my argument is that the PhD is not the right way to do this because it is ridiculously inefficient.

What we should have is a system wherein we produce more and better trained Masters level students  and fewer PhDs. This is the system that exists throughout most of Europe, in fact, and the UK is actually committed to adopt it through the Bologna process.  Not that this commitment seems to mean anything, as precisely nothing has been done to harmonize UK higher education with the 3+2+3 Bachelors+Masters+Doctorate system Bologna advocates.

The training provided in a proper two-year Masters programme will improve the skills pool for the world outside academia, and also better prepare the minority of students who go on to take a PhD. The quality of the  PhD will also improve, as only the very best and most highly motivated researchers will take that path. This used to be what happened, of course, but I don’t think it is any longer the case.

The main problem with this suggestion is that it requires big changes to the way both research and teaching are funded. The research councils turned away from funding Masters training many years ago, so I doubt if they can be persuaded to to a U-turn now. Moreover, the Research Excellence Framework provides a strong incentive for departments to produce as many PhDs as they possibly can, as these are included in an algorithmic way as part of the score for “Research Environment”. The more PhDs a department produces, the higher it will climb in the league tables. One of my targets in my current position is to double the number of PhDs produced by my School over the period 2013-18. What happens to the people concerned seems not to be a matter worthy of consideration. They’re only “outputs”…