A Modest Proposal

Last week I posted a short item about the looming Kafka-esque nightmare that is the Research Excellence Framework. A few people commented to me in private that although they hate the REF and accept that it’s ridiculously expensive and time-consuming, they didn’t see any alternative. I’ve been thinking about it and thought I’d make a suggestion. Feel free to shoot it down in flames through the box at the end, but I’ll begin with a short introduction.

Those of you old enough to remember will know that before 1992 (when the old `polytechnics’ were given the go-ahead to call themselves `universities’) the University Funding Council – the forerunner of HEFCE – allocated research funding to universities by a simple formula related to the number of undergraduate students. When the number of universities suddenly increased this was no longer sustainable, so the funding agency began a series of Research Assessment Exercises to assign research funds (now called QR funding) based on the outcome. This prevented research money going to departments that weren’t active in research, most (but not all) of which were in the ex-Polys. Over the years the apparatus of research assessment has become larger, more burdensome, and incomprehensibly obsessed with “research concentration”. Like most bureaucracies it has lost sight of its original purpose and has now become something that exists purely for its own sake.

It’s especially indefensible at this time of deep cuts to university budgets that we are being forced to waste an increasingly large fraction of our decreasing budgets on staff-time that accomplishes nothing useful except pandering to the bean counters.

My proposal is to abandon the latest manifestation of research assessment mania, i.e. the REF, and return to a simple formula, much like the pre-1992 system,  except that QR funding should be based on research student rather than undergraduate numbers.

There’s an obvious risk of game-playing, and this idea would only stand a chance of working at all if the formula involved the number of successfully completed research degrees over a given period .

I can also see an argument  that four-year undergraduate students (e.g. MPhys or MSci students) also be included in the formula, as most of these involve a project that requires a strong research environment.

Among the advantages of this scheme are that it’s simple, easy to administer, would not spread QR funding in non-research departments, and would not waste hundreds of millions of pounds on bureaucracy that would be better spent on research. It would also maintain the current “dual support” system for research.

I’m sure you’ll point out disadvantages through the comments box!


Share/Bookmark

About these ads

20 Responses to “A Modest Proposal”

  1. Rob Fender Says:

    Peter – completed PhDs is actually (likely to be) one of the REF metrics. Not the only one of course..

  2. Matt Burleigh Says:

    Immediate worry:
    - STFC PhD allocation mechanism is not transparent and leaves a typical academic with 1 STFC funded student per ~4 years. In absence of any other funding source, that academic has little power to positively influence PhD student numbers. Linking research funding to PhD student allocation assumes individual academic is currently being rewarded for performance with studentships. It is not at all clear that this is the case, both at STFC level and then within individual departments and groups…

    But I agree with you on the REF. I object to being judged on nebulous concepts such as “impact” and “esteem” (which by definition almost certainly bias against younger academics) as well as the ludicrous weight given by our whole industry to simplistic paper and citation counting.

    • Andrew Liddle Says:

      Dear Matt,

      From my biased position as STFC Education, Training and Careers (ETCC) chair, I have to dispute that the STFC studentships allocation is not transparent. It is entirely algorithm based and the algorithm is public, depending only on the number of supervisors and STFC postdocs within a department. The number of supervisors and postdocs are verified by departments during the process. See for instance
      http://www.stfc.ac.uk/Funding+and+Grants/14798.aspx

      Once departments have their allocation they have freedom to distribute them as they wish, and I can imagine that this step is not always done transparently.

      The number of studentships per year is roughly one fifth of the number of faculty (ie somewhat less than you say), and I expect the real complaint is that that is not larger. It has decreased by approximately 15% in the last two years, while faculty numbers have risen slightly. The studentships level is one of the things set by the 2009 programmatic review and CSR2010 delivery plan, ETCC doing rather badly in the first of these and quite well in the second.

      A desire to increase student numbers should of course be set against the ability to provide those students with meaningful career opportunities afterwards.

      Incidentally, there is no `esteem’ in the REF, it is what `impact’ has replaced. Out of the frying plan, into the fire …

      all the best,

      Andrew

    • andrew:

      can i suggest that the definition of a supervisor is tightened up by ET&C?

      at the moment i believe it simply refers to any PI/co-I on an STFC grant, irrespective of their FEC level – ie their research activity in the STFC area (or indeed whether they are eligible to be a supervisor). thus having a large number of co-I’s on a grant who each are spending just 5% of their time on STFC science will be rewarded with a larger student allocation than a smaller number who are spending an equal amount of time (in total) working on STFC-research.

      similarly, it is disappointing that the awards no longer reflect the “quality” of the training being given to the students as this removes a motivation from departments to improve the quality and content of their PG courses.

      ian

  3. telescoper Says:

    I’d be interested to know, in fact, how many departments have significant numbers of astro-related students other than those supported by STFC. There’ll be overseas students of course, but many universities also have their own studentship schemes and some are funded by departments themselves.

  4. Andrew Liddle Says:

    Dear Ian,

    To qualify as a supervisor the minimum FEC on STFC grants is 10%. The executive were keen for a higher number but I am glad they were persuaded otherwise as a significant fraction of the community might otherwise have faced exclusion given the diminished faculty FEC returns in recent rounds (particularly in nuclear physics so far; AGP seems to have done comparatively well in maintaining faculty FEC).

    I agree it is a shame that training is not still accounted for (too hard to quantify algorithmically), but as of next year completion rates will begin to factor, with penalties if the four-year completion rate falls below 70%. This statistic is chosen because that is what BIS use to assess RCUK’s performance.

    The old quality assessment, used up to 2007, favoured large departments as an outcome of peer review and perceived training quality was a part of that. Interestingly however every objective statistic, including completion rates, we looked at suggested that it is small departments who look after their students better (though samples too small to really draw firm conclusions). Not the message our political masters would like to hear, I expect.

    cheers,

    Andrew

    • telescoper Says:

      Andrew,

      I thought completion rates had already been factored in for this year’s quota. Your comment suggests not.

      Peter

  5. Andrew Liddle Says:

    Dear Peter,

    Completion rates were not included in the algorithm this year, though they are always collected and the committee has a look at them. Their effect first comes in for the next allocation round, though the planned penalty is fairly mild except for the (hopefully rare) cases where four-year completion rates falls below 50%.

    There will also be some protection for the smallest departments, which might otherwise be disadvantaged by small number statistics.

    Andrew

    • yes – its very disappointing that quality of training (no matter how hard it is to quantify) is no longer used to assess the number of studentships awarded. i thought that this particular metric had been useful to try to get groups to improve the quality of their training… so its removal is a big loss. but as Chair of ET&C i assume you could initiate a discussion within the committee to change this if you could wanted to do so?

      more importantly i don’t believe that “completion times” are at all related to the issue of quality of training – so that isn’t a like-for-like replacement in the new scheme. completion tiems are simply a measure of how good a department is at churning PhDs out – it bears no relation to the training the student receives (for anyone’s needs – the student’s, the academic or industrial employers). equally, many in the community may argue that while completion times is the stick which BIS is beating RCUK with – that doesn’t mean that STFC has to use the same weapon on its community.

      anyway – one last specific question – is there any plan for dealing with instrumentation PhDs in the completion time metric – these frequently suffer from delays due to instrument development issues which are potentially more severe than in other areas (or at least are less controlable by supervisors if they have due diligance). can the departments request waivers for such events to ensure they are not unduely penalised? if not then there will of course be pressure to pass any penalties onto to the relevant sections which might adversely effect the “impact” of STFC-funded science.

    • telescoper Says:

      There’s another issue. Increasingly, Research Councils require all sorts of training (especially transferable skills), but these are not part of the assessment of a thesis, the successful completion and defence of which is the basis of the award of a PhD degree. If students complete the PhD then they have by definition received the training required for that. Any other skills they have picked up along the way are simply irrelevant.

      If this training is to be a formal part of the PhD programme then shouldn’t it be assessed? And if so, how? Should we change the criteria for the award of PhD programmes to include it?

  6. Bryn Jones Says:

    The original A Modest Proposal – by Jonathan Swift – was a satire. I’ll assume that the spirit of this one isn’t (it certainly doesn’t appear that way).

    Peter’s proposal is interesting and might work reasonably well. However, there might be some problems, although details of the scheme could lessen some of these.

    A majority of British PhD studentships are funded by research councils. Different research councils have different allocation methods and fund different numbers of studentships per academic. Caution would have to be taken to avoid the studentship policies of individual research councils determining the level of research funding of those disciplines. The arts and humanities tend to have significantly fewer PhD studentships than the sciences, for example.

    One potential problem with Peter’s proposal in STFC areas is that there would be a multiplier effect of success in getting grant funding. Random factors that could determine success of a grant application (e.g. who is chosen to referee the application, the composition of the grant panel) would affect the HE research funding stream, as well as the winning of the grant itself. For all its faults, one very good thing about the RAE was that its quality assessment criteria were largely orthogonal to other funding streams.

    The policy might encourage teaching-only universities to introduce MSci-type courses just to win research funding, regardless of whether they are really up to it.

    I’m not sure how masters postgraduate students (on MSc, MA, MBA etc. courses) would fit in the picture. Some academic disciplines have large numbers of such students (such as business departments) and would benefit from large HE funding.

    Funding according to the numbers of PhD and masters students would create pressures to increase the numbers of such students. I don’t see any undersupply of PhD graduates for the needs of academia or general society at present.

    Of course, the great advantages of Peter’s proposal would be simplicity, efficiency and absence of bureaucracy.

    • telescoper Says:

      Bryn

      I promise to do a satirical post in due course, in which I propose that PhD students be used as a source of food.

      Peter

    • Bryn Jones Says:

      I can’t comment on the relative nutritional merits of PhD students and Irish babies.

  7. Andrew Liddle Says:

    Regarding PhD training issues above, ETCC does discuss this regularly and no doubt will again at the next meeting. However we are are required by the Executive to use only quantitative measures only in the algorithm, so it is not possible to incorporate training until someone comes up with a way of measuring it other than peer review. Incidentally, at the last time it was used, in 2007, it was felt that there was very little between the different departments, regardless of size, as there had been so much guidance that everyone had converged on number of courses offered, generic skills, second supervisor and review practices, etc.

    On four-year completion rates, they are understood to be a limited indicator and as such the penalty for falling below 70% is very weak, at 10% of allocation. While they give a very limited view, I’m sure no one (certainly not the students concerned) is going to argue that having a poor completion rate is a good thing. It does correlate with other negative indicators such as student satisfaction surveys. By the way, Durham in particular has an exceptionally strong completion rate record.

    cheers,

    Andrew

  8. Andrew Liddle Says:

    Concerning your actual proposal, the trouble is that, like NHS targets, focussing on a small part of the activity distorts the way people act. We are sadly now all obsessed with gaming the system. In your proposal, an extra PhD student probably would generate more income than the student costs. So every department `invests’ massively in PhD students. Unfortunately, everyone does it and there’s no extra money, so no one gains.

    cheers,

    Andrew

    • telescoper Says:

      Andrew,

      Obviously the scheme would fail if each PhD student were to generate more income than the cost, so it would be have to be ensured that was the case. In fact, I doubt would be. I’d have to look at the numbers, but the scenario you’re talking about also happened with previous RAEs – departments taking on staff at unsustainable levels to make the multiplier work for them. Moreover, PhD students don’t grow on trees.

      It might make it more economically viable to take on EU students not funded by STFC, but I don’t think that is a bad thing.

      Peter

  9. Anton Garrett Says:

    A point that arises often in these discussions is instability – research projects typically last longer than grant durations, so you have to cross your fingers and hope that THEY won’t pull the plug halfway on you and your postdocs and research students. The impression might be gained that Britain is uniquely bad in this regard. However in the 1980s I asked a leading US academic in my field if I could work with him as a postdoc and, to my delight, he said Yes. But it never came to pass because something called the Gramm-Rudman Act had the effect of temporarily blocking all US government funding, and during the stalemate my would-be boss had to retire through ill health. Gramm and Rudman were making a wider political point into which I do not wish to divert, but the idea that short-term uncertainty in academic science is uniquely bad in Britain is not true.
    Anton

    • telescoper Says:

      The UK is undoubtedly getting worse as regards stability of research funding, but I agree that there other countries where scientists have an even harder time of it.

    • The first rule of discourse is that two things must be avoided: 1) not complaining since someone, somewhere is worse off and 2) not recognising that there is someone, somewhere who is worse off.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 3,280 other followers

%d bloggers like this: