Stern Response

The results of the Stern Review of the process for assessing university research and allocating public funding has been published today. This is intended to inform the way the next Research Excellence Framework (REF) will be run, probably in 2020, so it’s important for all researchers in UK universities.

Here are the main recommendations, together with brief comments from me (in italics):

  1. All research active staff should be returned in the REF. Good in principle, but what is to stop institutions moving large numbers of staff onto teaching-only contracts (which is what happened in New Zealand when such a move was made)?
  2. Outputs should be submitted at Unit of Assessment level with a set average number per FTE but with flexibility for some faculty members to submit more and others less than the average.Outputs are countable and therefore “fewer” rather than “less”. Other than that, having some flexibility seems fair to me as long as it’s not easy to game the system. Looking it more detail at the report it suggests that some could submit up to six and others potentially none, with an average of perhaps two across the UoA. I’m not sure precise  numbers make sense, but the idea seems reasonable.
  3. Outputs should not be portable. Presumably this doesn’t mean that only huge books can be submitted, but that outputs do not transfer when staff transfer. I don’t think this is workable, but that what should happen is that credit for research should be shared between institutions when a researcher moves from one to another.
  4. Panels should continue to assess on the basis of peer review. However, metrics should be provided to support panel members in their assessment, and panels should be transparent about their use. Good. Metrics only tell part of the story.
  5. Institutions should be given more flexibility to showcase their interdisciplinary and collaborative impacts by submitting ‘institutional’ level impact case studies, part of a new institutional level assessment. It’s a good idea to promote interdisciplinarity, but it’s not easy to make it happen…
  6. Impact should be based on research of demonstrable quality. However, case studies could be linked to a research activity and a body of work as well as to a broad range of research outputs. This would be a good move. The existing rules for Impact seem unnecessarily muddled.
  7. Guidance on the REF should make it clear that impact case studies should not be narrowly interpreted, need not solely focus on socio-economic impacts but should also include impact on government policy, on public engagement and understanding, on cultural life, on academic impacts outside the field, and impacts on teaching. Also good.
  8. A new, institutional level Environment assessment should include an account of the institution’s future research environment strategy, a statement of how it supports high quality research and research-related activities, including its support for interdisciplinary and cross-institutional initiatives and impact. It should form part of the institutional assessment and should be assessed by a specialist, cross-disciplinary panel. Seems like a reasonable idea, but a “specialisr cross-disciplinary” panel might be hard to assemble…
  9. That individual Unit of Assessment environment statements are condensed, made complementary to the institutional level environment statement and include those key metrics on research intensity specific to the Unit of Assessment. Seems like a reasonable idea.
  10. Where possible, REF data and metrics should be open, standardised and combinable with other research funders’ data collection processes in order to streamline data collection requirements and reduce the cost of compiling and submitting information. Reasonable, but a bit vague.
  11. That Government, and UKRI, could make more strategic and imaginative use of REF, to better understand the health of the UK research base, our research resources and areas of high potential for future development, and to build the case for strong investment in research in the UK. This sounds like it means more political interference in the allocation of research funding…
  12. Government should ensure that there is no increased administrative burden to Higher Education Institutions from interactions between the TEF and REF, and that they together strengthen the vital relationship between teaching and research in HEIs. I believe that when I see it.

Any further responses (stern or otherwise) are welcome through the comments box!

 

5 Responses to “Stern Response”

  1. Philip Moriarty Says:

    Reblogged this on Symptoms Of The Universe and commented:
    Peter Coles (telescoper) on the Stern Review, published today. I’d certainly not quibble with any of his comments on the main recommendations of the report.

  2. Bryn Jones Says:

    Some points about the Stern report and Peter’s comments.

    The recommendation that all research-active staff are included in the REF is interesting. The report does not state it that it means all research-active academic staff. Does it therefore suggest including research officers? What about research fellows and postdocs on fixed-term contracts? Or is this lack of clarity in the report accidental?

    Yes, as Peter pointed out, people could be excluded from future REFs by classifying them as teaching only. In some sense, such a system might not differ greatly from REF (and before it RAE) practice in the past, which may not be what is intended.

    Placing a limit on the number of submitted outputs (papers etc.) of two times the number of FTE staff in a unit of assessment within an institution is interesting (replacing the old limit of four from each FTE staff member). The report assumes that this will lead to a similar total number of outputs (papers) next time as in 2014, on the assumption there will be more staff. I wonder how this would affect the politics within departments: how will departments choose who submits more than two outputs, and who enters fewer than two?

    It’s not clear why the report claims costs of £56 million for institutions in selecting outputs for the 2014 REF. I was entered in the 2008 RAE and my choice of four papers for inclusion was fairly straightforward. There was an internal institution mock RAE then, but that happened after the four outputs were selected, so did not add costs to the selection in itself.

    The suggested prohibition on moving outputs when researchers move institution seems reasonable when the person moves from one permanent contract to another. However, it would be very unreasonable for researchers who have previously been on fixed term contracts or are currently on fixed-term contracts. People on fixed-term contracts need the possibility of taking REF potential outputs with them as an incentive for institutions to offer them permanent posts.

    (Indeed, when I was a fixed-term lecturer I would have moved to any university that offered me a new lecturing contract of two years or longer, taking my RAE submission with me. Unfortunately no offers came up. I do feel the continued HEFCE research funding of my old institution that came from having included me in the 2008 RAE was not justified after my contract had come to an end and I had left, especially because I was not given access to research facilities to continue my research.)

    I don’t follow the philosophy behind restricting the number of impact case studies. Surely what is expensive is the volume of the description, not the number? Long descriptions of one or two case studies would not necessarily require less effort than succinct descriptions of a large number of case studies.

    Mentioning public engagement explicitly in relation to impact is very good. Keeping impact at 20% of the overall assessment means maintaining it at a rather high level.

    Reducing the administrative burden of the REF would be excellent. How this would be achieved is unclear.

    Recommending the use ORCID identifiers is very good.

    • “Recommending the use ORCID identifiers is very good.”

      On the train I was reading an editorial in the members’ magazine of the German Physical Society, where a similar recommendation was made (though primarily for other reasons). There is only one Peter Coles in cosmology, as far as I know, but there are more than 1200 people named J. Wang publishing on nanoscience. (There are at least 2 Simon D. Whites in astronomy, perhaps why the Simon White is S. D. M. White.)

      I wonder how many of the J. Wangs have permanent jobs because the selection committee was overwhelmed by their productivity. 😐

  3. You agree with more of Lord Stern’s recommendations than is usually the case for such documents. Apparently Lord Stern didn’t miss the boat and is playing with a full deck. Perhaps he should stand up and take a bow. Maybe he was advised by a group of astronomers, the so-called starboard. In any case, celebrate with a glass of port, perhaps in Keele or Hull.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: