Archive for Research Excellence Framework

Come off it, REF!

Posted in Science Politics with tags , , , , , , on October 27, 2011 by telescoper

Yesterday we all trooped off to the Millennium Stadium in Cardiff for a Staff Away Day. We didn’t actually get to play on the pitch of course, which wasn’t even there, as it had been removed to reveal a vast expanse of soil. Instead we were installed in the “Dragon Suite” for a discussion about our preparation for the forthcoming Research Excellence Framework.

Obviously I can’t post anything about our internal deliberations, but I’m sure departments up and down the United Kingdom are doing similar things so I thought I’d mention a few things which are already in the public domain and my personal reactions to them. I should also say that the opinions I express below are my own and not necessarily those of anyone else at Cardiff.

The first thing is the scale of the task facing members of the panel undertaking this assessment. Each research active member of staff is requested to submit four research publications (“outputs”) to the panel, and we are told that each of these will be read by at least two panel members. The panel comprises 20 members.

As a rough guess I’d say that the UK has about 40 Physics departments, and the average number of research-active staff in each is probably about 40. That gives about 1600 individuals for the REF. Actually the number of category A staff submitted to the 2008 RAE was 1,685.57 FTE (Full-Time Equivalent), pretty  close to this figure. At 4 outputs per person that gives 6400 papers to be read. We’re told that each will be read by at least two members of the panel, so that gives an overall job size of 12800 paper-readings. There are 20 members of the panel, so that means that between 29th November 2013 (the deadline for submissions) and the announcement of the results in December 2014 each member of the panel will have to have read 640 research papers. That’s an average of about two a day…

Incidentally, as I’ve mentioned before, the Physics REF panel includes representatives from institutions in England, Scotland and Northern Ireland, but not Wales. The decision to exclude representation from Welsh physics departments was a disgrace, in my view.

Now we are told the panel will use their expert judgment to decide which outputs belong to the following categories:

  • 4*  World Leading
  • 3* Internationally Excellent
  • 2* Internationally Recognized
  • 1* Nationally Recognized
  • U   Unclassified

There is an expectation that the so-called QR  funding allocated as a result of the 2013 REF will be heavily weighted towards 4*, with perhaps a small allocation to 3* and probably nothing at all for lower grades. “Internationally recognized” research is probably worthless in the view of HEFCE, in other words. Will the papers belonging to the category “Not really understood by the panel member” suffer the same fate?

The panel members will apparently know enough about every single one of the papers they are going to read in order to place them  into one of the above categories, especially the crucial ones “world-leading” or “internationally excellent”, both of which are obviously defined in a completely transparent and objective manner. Not.

We are told that after forming this judgement based on their expertise the panel members will “check” the citation information for the papers. This will be done using the SCOPUS service provided (no doubt at considerable cost) by   Elsevier, which by sheer coincidence also happens to be a purveyor of ridiculously overpriced academic journals. I’ve just checked the citation information for some of my papers on SCOPUS, and found an alarming number of errors. No doubt Elsevier are  on a nice little earner peddling meaningless data for the HECFE bean-counters, but I haven’t any confidence that it will add much value to the assessment process.

There have been high-profile statements to the effect that the REF will take no account of where the relevant “outputs”  are published, including a recent pronouncement by David Willetts. On the face of it, that would suggest that a paper published in the spirit of Open Access in a free archive would not be disadvantaged. However, I very much doubt that will be the case.

I think if you look at the volume of work facing the REF panel members it’s pretty clear that citation statistics will be much more important for the Physics panel than we’ve been led to believe. The panel simply won’t have the time or the breadth of understanding to do an in-depth assessment of every paper, so will inevitably in many cases be led by bibliometric information. The fact that SCOPUS doesn’t cover the arXiv means that citation information will be entirely missing from papers just published there.

The involvement of  a company like Elsevier in this system just demonstrates the extent to which the machinery of research assessment is driven by the academic publishing industry. The REF is now pretty much the only reason why we have to use traditional journals. It would be better for research, better for public accountability and better economically if we all published our research free of charge in open archives. It wouldn’t be good for academic publishing houses, however, so they’re naturally very keen to keep things just the way they are. The saddest thing is that we’re all so cowed by the system that we see no alternative but to participate in this scam.

Incidentally we were told before the 2008 Research Assessment Exercise that citation data would emphatically not be used;  we were also told afterwards that citation data had been used by the Physics panel. That’s just one of the reasons why I’m very sceptical about the veracity of some of the pronouncements coming out from the REF establishment. Who knows what they actually do behind closed doors?  All the documentation is shredded after the results are published. Who can trust such a system?

To put it bluntly, the apparatus of research assessment has done what most bureaucracies eventually do; it has become  entirely self-serving. It is imposing increasingly  ridiculous administrative burdens on researchers, inventing increasingly  arbitrary assessment criteria and wasting increasing amounts of money on red tape which should actually be going to fund research.

And that’s all just about “outputs”. I haven’t even started on “impact”….

Commodification, the Academic Journal Racket and the Digital Commons (via The Disorder Of Things)

Posted in Open Access, Uncategorized with tags , , , , , , , , , , , , , , , , , , , , , on September 15, 2011 by telescoper

Here’s another reasoned rant regarding the rapacity of the research racketeers. I think it makes some really good points.

The video clip is worth watching too, it being very funny.

Commodification, the Academic Journal Racket and the Digital Commons David, my erstwhile ‘parasitic overlord’ from when I was co-editing Millennium, points me to some posts by Kent Anderson of the Society for Scholarly Publishing, who defends the industry on a number of grounds from Monbiot’s polemic against the journal racket. The comments threads on both pieces are populated by academics who agree with Monbiot and by publishing industry colleagues who agree with Anderson (and who alternate between dismissing and … Read More

via The Disorder Of Things

Science Publishing: What is to be done?

Posted in Science Politics with tags , , , on September 10, 2011 by telescoper

The argument about academic publishing has been bubbling away nicely in the mainstream media and elsewhere in the blogosphere; see my recent post for links to some of the discussion elsewhere.

I’m not going to pretend that there’s a consensus amongst all scientists about this, but everything I’ve read has confirmed my rather hardline view, which is that in my field, astrophysics, academic journals are both unnecessary and unhealthy. I can certainly accept that in days gone by, perhaps up to around 1990, scientific journals provided the only means of disseminating research to the wider world. With the rise of the internet, that is no longer the case. Year after year we have been told that digital technologies would make scientific publishing cheaper. That has not happened. Journal subscriptions have risen faster than inflation for over a decade. Why is this happening? The answer is that we’re being ripped off. What began by providing a useful service has now become simply a parasite and, like most parasites, it is endangering the health of its subject.

The scale of the racket is revealed in an article I came across in Research Fortnight. Before I give you the figures let me explain that the UK Higher Education funding councils, such as HEFCE in England and HEFCW in Wales, award funding in a manner determined by the the quality of research going on in each department as judged by various research assessment exercises; this funding is called QR funding. Now listen to this. It is estimated that around 10 per cent of all QR funding in the UK goes into journal subscriptions. There is little enough money in science research these days for us to be paying a tithe of such proportions. This has to stop.

You might ask why such an obviously unsustainable situation carries on. I think there are two answers to this. One is the rise of the machinery of research assessment, which plays into the hands of the publishing industry. For submitted work to count in the Research Assessment Exercise (or its new incarnation, the Research Excellence Framework) it must be published in a refereed journal. Scientists who want to break the mould by publishing their papers some other way will be stamped on by our lords and masters who hold the purse strings. The whole system is invidious.

The second answer is even more discomforting. It is that many scientists actually like the current system. Each paper in a “prestigious” journal is another feather in your cap, another source of pride. It doesn’t matter if nobody reads any of them, ones published output is a measure of status. For far too many researchers gathering esteem by publishing in academic journals has become an end in itself. The system corrupts and has become corrupted. You can find similar comments in a piece in last week’s Guardian.

So what can be done? Well, I think that physics and astronomy can show the way forward. There is already a rudimentary yet highly effective prototype in place, called the arXiv. In many fields, including astronomy, all new papers are put on the arXiv, and these can be downloaded by anyone for free. Particle physics led the way towards the World Wide Web, an invention that has revolutionised so many things. It’s no coincidence that physicists are also ahead of the game on academic publishing too.

Of course it takes money to run the arXiv and that money is at the moment paid by contributions from universities that use it extensively. You might then argue that means the arXiv is just another journal, just one where the subscription cost is less obvious.

Perhaps that’s true, but then just take a look at the figures. The total running costs of the arXiv amount to just $400,000 per annum. That’s not just for astronomy but for a whole range of other branches of physics too, and not only new papers but a back catalogue going back at least 15 years.

There are about 40 UK universities doing physics research. If UK Physics had to sustain the costs of the arXiv on its own the cost would be an average of just $10,000 per department per annum. Spread the cost around the rest of the world, especially the USA, and the cost would be peanuts. Even $10,000 is less than most single physics journal subscriptions; indeed it’s not even 10 per cent of my departments annual budget for physics journals!

Whenever I’ve mentioned the arXiv to publishers they’ve generally dismissed it, arguing that it doesn’t have a “sustainable business plan”. Maybe not. But it is not the job of scientific researchers to support pointless commercial enterprises. We do the research. We write the papers. We assess their quality. Now we can publish them ourselves. Our research is funded by the taxpayer, so it should not be used to line the pockets of third parties.

I’m not saying the arXiv is perfect but, unlike traditional journals, it is, in my field anyway, indispensable. A little more investment, adding a comment facilities or a rating system along the lines of, e.g. reddit, and it would be better than anything we get academic publishers at a fraction of the cost. Reddit, in case you don’t know the site, allows readers to vote articles up or down according to their reaction to it. Restrict voting to registered users only and you have the core of a peer review system that involves en entire community rather than relying on the whim of one or two referees. Citations provide another measure in the longer term. Nowadays astronomical papers attract citations on the arXiv even before they appear in journals, but it still takes time for new research to incorporate older ideas.

Apparently, Research Libraries UK, a network of libraries of the Russell Group universities and national libraries, has already warned journal publishers Wiley and Elsevier that they will not renew subscriptions at current prices. If it were up to me I wouldn’t bother with a warning…

 

Stellar Research?

Posted in Education, Science Politics with tags , , , , on August 24, 2011 by telescoper

I heard today that  Chief Scientific Advisor to the Welsh Government, John Harries, has called for Welsh universities to be more “predatory” in attracting “star researchers” to Wales. At first sight I thought that sounded like good news for astronomy, but reading the article more closely I realise that’s not what he meant!

The point is that, according to the BBC article,  Welsh universities currently only attract about 3% of the UK’s research funding whereas the famous Barnett formula allocates Wales about 5% of the total in other areas of expenditure.  Nobody involved in research  would argue for funds to be allocated on any other basis than through quality, so there’s no clamour for having research funding allocated formulaically a là Barnett; the only way to improve the success rate is to improve the quality of applications. John Harries suggests that means poaching groups from elsewhere who’ve already got a big portfolio of research grants…

The problem with that strategy is that it’s not very easy to persuade such people to leave their current institutions, especially if they’ve already spend years acquiring the funding needed to equip their laboratories. It’s not just a question of moving people, which is relatively easy, but can involve trying to replace lots of expensive and delicate equipment. The  financial inducements needed to fund the relocation of a major research group and fight off counter-offers from its present host are likely to be so expensive that the benefit gained from doing this takes years to accrue, even they are successful.

I agree with Prof. Harries that Welsh universities need to raise their game in research, but I don’t think this “transfer market” approach is likely to provide a solution on its own. I think Wales needs a radical restructuring of research, especially in science, across the whole sector, which I think is unacceptably complacent about the challenges ahead.

For a start, much more needs to be done to identify and nurture  younger researchers, i.e. future research stars  rather than present ones.  Most football clubs nowadays have an “academy” dedicated to the development of promising youngsters, so why can’t we do a similar thing for research? Research groups in different Welsh universities also need to develop closer collaborations, and perhaps even full mergers, in order to compete with larger English institutions.

More controversially I’d say that the problem is not being helped by Welsh universities continuing to be burdened by the monstrous bureaucracy and bizarre practices of the Research Excellent Framework, which allocates “QR” research funds according to priorities set by HEFCE in a way that reflects the thinking of the Westminster parliament. The distribution of QR funding in Wales, which is meant to supplement competitive grant income from UK  funding bodies, should be decided by HEFCW in line with Welsh strategic priorities. Wales would be far better off withdrawing from the REF and doing its own thing under the auspices of the Welsh Assembly Government.

What I’m saying is that I’ve got nothing against Welsh universities trying to entice prominent research leaders here;  we’ve recently tried (unsuccessfully) to do it here in the School of Physics & Astronomy at Cardiff University, in fact. But in the current funding climate it’s not easy to persuade their current institutions to let them go. In any case,  I don’t think parachuting in a few high-profile individuals will in itself solve the deep-rooted problems of the Welsh university system. A longer term strategy needs to be found.

Scotland already punches above its weight in terms of research income for its universities and there’s no reason why, in the long run, Wales can’t do likewise.

The Graveyard of Ambition?

Posted in Education, Finance, Politics, Science Politics with tags , , , , , on May 23, 2010 by telescoper

The news today is full of speculation about the nature and extent of impending public spending cuts expected to be announced in the Queen’s Speech next Tuesday. Among the more specific figures being bandied about is a £700 million cut to the budget Department of Business, Innovation and Skills (BIS) which encompasses both scientific research and the university sector. It’s impossibly to say precisely where the axe will fall, but it’s very likely that university-based science groups in England will face a double-whammy, losing income both from HEFCE and from the Research Councils. The prospect looks particular dire for Physics & Astronomy, which rely for their research grants on the Science and Technology Facilities Council (STFC) which savagely cut back science research even before the credit crunch arrived. If STFC gets cut any further  then the result will be even worse carnage un universities than we’ve experienced over the last year or two, especially since it looks like there will be no changes in its Executive.

Here in Wales the situation is even more complicated, as is explained in a long article in this week’s Times Higher. Cuts to the Research Councils will, of course, affect university research groups in the Principality as their remit covers the whole of the United Kingdom. However, responsibility for Higher Education in Wales is devolved to the Welsh Assembly Government. This means that any cuts to the University budget announced next week will not apply here (nor indeed in Scotland or Northern Ireland).

However, as I’ve blogged about before, it’s not obvious that this is good news for fundamental science in Wales. The Welsh Assembly Government’s blueprint for the shape of Higher Education in Wales, For Our Future, signals what could be dramatic changes in the way university funding is allocated here. There’s a lot of nervousness about how things will pan out.

Currently, most university funding in Wales comes through HEFCW in the form of recurrent grants. However the WAG has recently set up a Strategic Implementation Fund which in future supply 80% of all university funding. The new(ish) Minister responsible for Higher Education, Leighton Andrews (who will be giving a public lecture in Cardiff about the changes next week) seems to be determined to take control of the sector. It’s good to have a Minister who shows some interest in Higher Education, but I’m wary of politicians with Big Ideas.

We’ll have to wait and see what happens over the next year or so, but I think there’s an opportunity for Wales to do something truly radical and break away from systems that simply copy those in place in England with a much lower level of resource. Given that HEFCW has already been told how 80% of its funding should be administered, why bother with HEFCW at all? Scrapping this quango will remove a buffer between the universities and the WAG, which might be a dangerous thing to do, but will also save money that could be spent on higher education rather than bureaucracy. And while we’re at it, why doesn’t the WAG take Welsh universities out of the Research Excellence Framework? In the new era why should Welsh universities be judged according to English priorities?

On the teaching side, the WAG wants to see more flexible study options, more part-time degrees (including PhDs), more lifelong learning, and so on. I think that’s a reasonable thing to aim for given the particular socio-economic circumstances that pertain in Wales, but I can’t really see scope for significant numbers of part-time degrees in physics, especially at the doctoral level.

A crucial issue that has to be addressed is the proliferation of small universities in Wales. England has a population of 49.1 million, and has  91 universities (a number that many consider to be way too high in any case). The population of Wales is just 2.98 million but has 12 universities which is about twice as many per capita as in England. I for one think this situation is unsustainable, but I’m not sure to what extent mergers would be politically acceptable.

The WAG also wants to focus funding on “priority areas” that it perceives to be important to future development of industry in Wales, including health and biosciences, the digital economy, low-carbon technologies, and advanced engineering and manufacturing.  Fair enough, I say, as long as “focus on” doesn’t mean “scrap everything else but”.  The big worry for me is that research doesn’t feature very strongly at all in the WAG’s document, and it isn’t in good shape in any case. According to the last Research Assessment Exercise (RAE), only around 14% of Welsh research is of world-leading quality and most of that (90%)  is concentrated in just four institutions (Aberystwyth, Bangor, Cardiff and Swansea).

Physics in Wales did particularly poorly in the RAE and in any case only involves three universities, Bangor having closed its Physics department many years ago. Indeed the RAE panel went out of its way to make unfavourable comments about the lack of coordination in Welsh  physics – comments, I might add, that went entirely beyond the panel’s remit and should have attracted censure. Physics is also an expensive subject so if we are to do better in future we need additional investment. Cardiff University is doing its best bring this about, but I think we should explore closer ties with Swansea and explicit encouragement from the WAG.

STEM areas are woefully under-represented in Wales. Some think the WAG should seize the chance to boost this area of activity, but others think it’s already too late. According to the Times Higher,

Julie Lydon, vice-chancellor of the University of Glamorgan and the first female head of a university in Wales, says expertise in STEM will have to be developed in “distinct areas”. Given its small size, Wales must be careful to set itself realistic aims, she says.

The country faces a complex challenge, Lydon adds. “We don’t have anywhere near the range and extent of research (that we should) for our size. We’ve got to move it up a gear, and we’ve got to raise aspirations. We’ll do that in niche areas, and we’ll do that by partnership, not on our own.

“We haven’t the scope and scale; Wales isn’t a large enough sector to be able to do that across the board, but it’s an agenda that is slightly wider than the narrow view of STEM.”

A focus on STEM would neglect some areas in which Wales is strong, Lydon says. Thanks to investment from major employers such the BBC, disciplines such as media are growth areas and critical to the economy, but they are not strictly defined as STEM subjects.

No, media studies isn’t a STEM subject. Nor do I think Wales can continue to rely on its economy being propped up by public bodies such as the BBC. The expected round of wider public spending cuts I mentioned at the start of this piece will effectively scupper that argument and I’m sure privatisation of the BBC is on the new government’s agenda anyway. The future requires more ambition than this kind of thinking exemplifies. Sadly, however, ambition doesn’t seem to be something that the Welsh are particularly good at.  Dylan Thomas’s phrase “The Graveyard of Ambition” was specifically aimed at his home town of Swansea, but it does sum up an attitude you can find throughout the country: a  resolute determination to be mediocre.

 Wales is indeed a small country. So is Scotland (population about 5 million), but the Scots have for a long time placed a much higher premium on science and university education generally than the Welsh (and even the English) and they have a thriving university sector that’s the envy of other nations (including England). I think it’s time for a change of mentality.

Author Credits

Posted in Science Politics, The Universe and Stuff with tags , , , , , , on December 10, 2009 by telescoper

I’ve posted before about the difficulties and dangers of using citation statistics as measure of research output as planned by the forthcoming Research Excellence Framework (REF). The citation numbers are supposed to help quantify the importance of research as judged by peers. Note that, in the context of the REF, this is a completely different thing to impact which counts a smaller fraction of the assessment and which is supposed measure the influence of research beyond its own discipline. Even the former is difficult to measure, and the latter is well nigh impossible.

One of the problems of using citations as a metric for research quality is to do with how one assigns credit to large teams of researchers who work in collaboration. This is a particularly significant, and rapidly growing, problem in astronomy where large consortia are becoming the exception rather than the rule. The main questions are: (i) if paper A is cited 100 times and has 100 authors should each author get the same credit? and (ii) if paper B is also cited 100 times but only has one author, should this author get the same credit as each of the authors of paper A?

An interesting suggestion over on the e-astronomer addresses the first question by suggesting that authors be assigned weights depending on their position in the author list. If there are N authors the lead author gets weight N, the next N-1, and so on to the last author who gets a weight 1. If there are 4 authors, the lead gets 4 times as much weight as the last one.

This proposal has some merit but it does not take account of the possibility that the author list is merely alphabetical which I understand will be the case in forthcoming Planck publications, for example. Still, it’s less draconian than another suggestion I have heard which is that the first author gets all the credit and the rest get nothing. At the other extreme there’s the suggestion of using normalized citations, i.e. just dividing the citations equally among the authors and giving them a fraction 1/N each.

I think I prefer this last one, in fact, as it seems more democratic and also more rational. I don’t have many publications with large numbers of authors so it doesn’t make that much difference to me which you measure happen to pick. I come out as mediocre on all of them.

No suggestion is ever going to be perfect, however, because the attempt to compress all information about the different contributions and roles within a large collaboration into a single number, which clearly can’t be done algorithmically. For example, the way things work in astronomy is that instrument builders – essential to all observational work and all work based on analysing observations – usually get appended onto the author lists even if they play no role in analysing the final data. This is one of the reasons the resulting papers have such long author lists and why the bibliometric issues are so complex in the first place.

Having dozens of authors who didn’t write a single word of the paper seems absurd, but it’s the only way our current system can acknowledge the contributions made by instrumentalists, technical assistants and all the rest. Without doing this, what can such people have on their CV that shows the value of the work they have done?

What is really needed is a system of credits more like that used in the television or film. Writer credits are assigned quite separately from those given to the “director” (of the project, who may or may not have written the final papers), as are those to the people who got the funding together and helped with the logistics (production credits). Sundry smaller but still vital technical roles could also be credited, such as special effects (i.e. simulations) or lighting (photometic calibration). There might even be a best boy. Many theoretical papers would be classified as “shorts” so they would often be written and directed by one person and with no technical credits.

The point I’m trying to make is that we seem to want to use citations to measure everything all at once but often we want different things. If you want to use citations to judge the suitability of an applicant for a position as a research leader you want someone with lots of directorial credits. If you want a good postdoc you want someone with a proven track-record of technical credits. But I don’t think it makes sense to appoint a research leader on the grounds that they reduced the data for umpteen large surveys. Imagine what would happen if you made someone director of a Hollywood blockbuster on the grounds that they had made the crew’s tea for over a hundred other films.

Another question I’d like to raise is one that has been bothering me for some time. When did it happen that everyone participating in an observational programme expected to be an author? It certainly hasn’t always been like that.

For example, go back about 90 years to one of the most famous astronomical studies of all time, Eddington‘s measurement of the bending of light by the gravitational field of the Sun. The paper that came out from this was this one

A Determination of the Deflection of Light by the Sun’s Gravitational Field, from Observations made at the Total Eclipse of May 29, 1919.

Sir F.W. Dyson, F.R.S, Astronomer Royal, Prof. A.S. Eddington, F.R.S., and Mr C. Davidson.

Philosophical Transactions of the Royal Society of London, Series A., Volume 220, pp. 291-333, 1920.

This particular result didn’t involve a collaboration on the same scale as many of today’s but it did entail two expeditions (one to Sobral, in Brazil, and another to the Island of Principe, off the West African coast). Over a dozen people took part in the planning,  in the preparation of of calibration plates, taking the eclipse measurements themselves, and so on.  And that’s not counting all the people who helped locally in Sobral and Principe.

But notice that the final paper – one of the most important scientific papers of all time – has only 3 authors: Dyson did a great deal of background work getting the funds and organizing the show, but didn’t go on either expedition; Eddington led the Principe expedition and was central to much of the analysis;  Davidson was one of the observers at Sobral. Andrew Crommelin, something of an eclipse expert who played a big part in the Sobral measurements received no credit and neither did Eddington’s main assistant at Principe.

I don’t know if there was a lot of conflict behind the scenes at arriving at this authorship policy but, as far as I know, it was normal policy at the time to do things this way. It’s an interesting socio-historical question why and when it changed.

Negative Impact

Posted in Science Politics with tags , , , , , on December 2, 2009 by telescoper

After spending the best part of the last couple of days being prodded and poked and subjected to all manner of indignity in the name of medical science, I think it’s appropriate to return to the blogosphere with another rant. Before I start, however, I’d seriously like to thank everyone at the University Hospital of Wales at Heath Park  for making my visit there as brief and painless as possible. Everyone was very kind and very efficient. I’m not going to blog about the details, as Columbo doesn’t like reading about other peoples’ ailments.

Over the past few weeks there has been a lot of discussion about the UK government’s agenda for research, particularly science research, that includes something called “impact”. The Research Excellence Framework (REF; successor to the Research Assessment Exercise, RAE) will include such a thing:

Significant additional recognition will be given where researchers build on excellent research to deliver demonstrable benefits to the economy, society, public policy, culture and quality of life

Apparently, however, they don’t really know how to do this so they have set up a number of pilot studies to try to find out. I’d feel a little more comfortable if the bureaucrats had thought about what they were going to do before announcing that our future research funds were going to depend on it. Meanwhile, applicants for grants from any of the research councils must  include a statement of the “economic or social” impact their research will have.

Understandably, those of us working in “blue skies” research are very nervous about this new regime. There is more than a suspicion that the new emphasis on impact is intended to divert funds away from “pure” curiosity driven research and into areas where it can have an immediately identifiable short-term economic benefit. This has led to a petition, with over 13000 signatures, by the University and College Union calling for the impact statements to be abandoned.

I don’t know who is going to assess these impact statements, but unless they have a flawless ability to predict future technology I don’t think fundamental physics is going to score very well at all. To see my point, consider the case of  J. J. Thomson, who is generally credited with having discovered the electron and who was awarded the Nobel Prize for Physics in 1906. Thomson made extensive use of cathode ray tubes in his studies; these later found their way into sitting rooms across the world as essential components of the classic television set. But that took decades. I doubt if an impact panel looking at Thomson’s work – even if they were physicists rather than grey-suited bureaucrats – would have found any of it likely to lead to immediate economic benefit. The point is that when he discovered the electron it wasn’t because he was actually trying to invent the television set.

I think there are basically two possible interpretations of this impact business. One is that it is a deliberate plan to wind down fundamental research and use the money saved to subsidise UK industry. The other is that it’s another exercise in pointless box-ticking. I am in two minds. On the one hand, it is clear that the recent behaviour of the Science and Technology Facilities Council shows strong evidence of the former. Fundamental research is being slashed, yet projects involving space technology have been funded on the nod without scientific  peer review. On the other hand, the RCUK Impact “Champion”, a person by the name of David Delpy, has written in the Times Higher to defend the new agenda. Consider the following paragraph

Recently I have read that some believe it is impossible to predict the economic impact of blue-skies research. To be clear, we are not asking for accurate predictions – simply a consideration of potential. Basic research underpins all disciplines and builds pathways to new technologies with economic and social applications. It may build on an existing body of knowledge, connect to other research around the world or attract new industries to the UK. There are many routes to impact. I believe that I could write a statement indicating potential impact for any proposal I have seen, and to hear that bright academics say they can’t do it sounds a little disingenuous.

Champion Delpy thus suggests he could write a statement for any proposal he has seen, which sounds to me like an admission that what is called for is just a load of flannel. In fact, if he’s paid to be the Impact Champion perhaps he should write all the bullshit and save us scientists the need to jump through these silly hoops? Or perhaps we could get one of those little Microsoft Office Assistant things:

Hello. Looks like you’re writing an Impact Assessment. Would you like me to pad it out with meaningless but impressive-looking socio-economic buzzwords for you?

If it’s just another exercise in vacuous bureaucracy then it’s bad enough, but if it is the other possibility then of course it’s even worse. It could be the end for disciplines like astronomy and particle physics as well as the end of Britain’s history of excellence in those areas. I’ve already blogged about my view of short-termism in research funding. Essentially, my point is that government money should be used to fund precisely those things that don’t have immediate economic benefit. Those that do should be funded by the beneficiaries, i.e. commercial companies.

Politicians probably think that all this complaining about impact means that scientists  are arrogantly assuming that the taxpayer should fund them regardless of the cost or the benefit. I can only speak for myself, but I think that’s very unfair. I’m very conscious that my research is funded by Joe Public; that’s one of the reasons I think I should spend time giving public talks and doing other outreach activities. But I think the public funds me and others like me to do “useless” things because, in the end, useless things are more important than money.

The government is probably right to say that the UK economy doesn’t benefit as much from our scientific expertise as is the case with other countries. The reason for that, however, lies not with our universities and research laboratories but with our private industrial and commercial sectors which are, for the most part, managed with a very low level of competence. British universities are demonstrably excellent; our industry is demonstrably feeble. The persistent failure of the private sector to invest in research and development shows that it is in drastic need of a good shake up. British companies, not the taxpayer, should be paying for research that leads to profit for them and for that to happen they will have to learn to engage better with the University sector rather than expecting inventions to be served up on a plate funded by the taxpayer. Universities and research labs should continue do what they’re good at,  maintaining a culture within which curiosity and learning are promoted for their own sake not just as part of the dreary materialistic cycle of production and consumption that is all we seem to be able to think about these days.

So at the end I’ve come to the conclusion that, perhaps, insofar as it can be demonstrated, economic impact should be included in the forthcoming Research Excellence Framework. Research which leads directly to the economic gain of the private sector is  precisely the type of research that the taxpayer should not be paying for. If it can be proven that a given department has engaged in such activity, its state funding should therefore be cut and it should be told to recover the funds it has misused from the company that has benefitted from it. Economic impact should be included with a negative weight.

And if you think that’s a silly point of view, consider what happens with the other major part of a university’s activity, teaching. Students, we are told, are the primary beneficiaries of their education so they should have to pay fees. In the current regime, however, they only do so when their earnings reach a certain level. If commercial companies are to be the primary beneficiaries of state-funded research, why should they not likewise be asked to pay for it?

Reffing

Posted in Science Politics with tags , , , on September 28, 2009 by telescoper

No sooner has the dust settled on the  2008 Research Assessment Exercise (RAE) when the Higher Education Funding Council for England (HEFCE) has tabled its proposals for a new system called the Research Excellence Framework (REF) in a 56-page consultation document that you can download and peruse at your leisure.

I won’t try to give a complete account of the new system except to say that apart from the change of acronym there won’t be much different. Many of us hoped that the new framework would involve a lighter touch than the RAE, so we could actually get on with research instead of filling in forms all our lives. Fat chance. You can call me cynical if you like, but I think it’s obvious that once you set up a monstrous bureaucratical nightmare like the RAE it is almost impossible to kill it off. Things like this gather their own momentum and become completely self-serving. The apparatus of research assessment no longer exists to fulfil a particular purpose. It exists because it exists.

It might be useful however to summarise the main changes:

  1. The number of Units of Assessment and sub-panels is to be reduced from 67 to 30 and the number of main assessment panels from 15 to 4. This move is bound to prove controversial as it will clearly reduce the number of specialists involved in the quality appraisal side of things. However, the last RAE produced clear anomalies in the assessment carried out by different panels: physics overall did very poorly compared to other disciplines, for example. Having fewer panels might make it easier to calibrate different subjects. Might.
  2. In REF the overall assessments are going to be based on three elements: research output (60%); impact (25%); and environment (15%). In the last RAE each panel was free to vary the relative contribution of different components to the overall score. Although the “research output” category is similar to the last RAE, it is now proposed to include citation measures in the overall assessment. Officially, that is. It’s an open secret that panel members did look at citations last time anyway.  Citation impact will however be used only for certain science and engineering subjects.  “Impact” is a new element and its introduction is  in line with the government’s agenda to pump research funds into things which will generate wealth, so this measure will probably shaft fundamental physics. “Environment” includes things like postgraduate numbers, research funding and the like; this is also similar to the RAE.
  3. A roughly similar number of experts will be involved as in RAE 2008 – so it will be similarly expensive to run.
  4. The consultation document asks whether the number of outputs submitted per person should be reduced from four to three, and also whether “substantive outputs” (whatever they are) should be “double-weighted”.
  5. The results will be presented in terms of “profiles” as in 2008, with the percentage of activity at each level being given.
  6. The consultation also suggests honing the description of “world-leading” (4*) and “internationally excellent” (3*) to achieve greater discrimination at the top end of the scale. This is deeply worrying, as well as completely absurd. The last RAE applied a steeply rising funding formula to the scores so that 4*:3*:2*:1* was weighted 7:3:1:0. However the fraction of  work in each category is subject to considerable uncertainty, amplified by the strong weighting.  If the categories are divided further then I can see an even steeper weighting emerging, with the likely outcome that small variations in the (subjective) assessment will lead to drastic variations in funding. Among the inevitable consequences of this will be that  some excellent research will lose out.

No doubt university administrators across the United Kingdom will already be plotting how best to play the new system. I think we need to remember, though, that deep cuts in public spending have been promised by both major political parties and there is a general election due next year. I can see the overall  budget for university research being slashed so we’ll be fighting for shares of a shrinking pot. Killing off the bureaucracy would save money, but somehow I doubt that will be on the agenda.

Returning to Lognormality

Posted in Biographical, Science Politics, The Universe and Stuff with tags , , , on June 7, 2009 by telescoper

I’m off later today for a short trip to Copenhagen, a place I always enjoy visiting. I particularly remember a very nice time I had there back in 1990 when I was invited by Bernard Jones, who used to work at the Niels Bohr Institute.  I stayed there several weeks over the May/June period which is the best time of year  for Denmark; it’s sufficiently far North that the summer days are very long, and when it’s light until almost midnight it’s very tempting to spend a lot of time out late at night.

As well as being great fun, that little visit also produced my most-cited paper. I’ve never been very good at grabbing citations – I’m more likely to fall off bandwagons rather than jump onto them – but this little paper seems to keep getting citations. It hasn’t got that many by the standards of some papers, but it’s carried on being referred to for almost twenty years, which I’m quite proud of; you can see the citations per year statistics are fairly flat. The model we proposed turned out to be extremely useful in a range of situations, hence the long half-life.

nph-ref_history

I don’t think this is my best paper, but it’s definitely the one I had most fun working on. I remember we had the idea of doing something with lognormal distributions over coffee one day,  and just a few weeks later the paper was  finished. In some ways it’s the most simple-minded paper I’ve ever written – and that’s up against some pretty stiff competition – but there you go.

Picture1

The lognormal seemed an interesting idea to explore because it applies to non-linear processes in much the same way as the normal distribution does to linear ones. What I mean is that if you have a quantity Y which is the sum of n independent effects, Y=X1+X2+…+Xn, then the distribution of Y tends to be normal by virtue of the Central Limit Theorem regardless of what the distribution of the Xi is  If, however, the process is multiplicative so  Y=X1×X2×…×Xn then since log Y = log X1 + log X2 + …+log Xn then the Central Limit Theorem tends to make log Y normal, which is what the lognormal distribution means.

The lognormal is a good distribution for things produced by multiplicative processes, such as hierarchical fragmentation or coagulation processes: the distribution of sizes of the pebbles on Brighton beach  is quite a good example. It also crops up quite often in the theory of turbulence.

I;ll mention one other thing  about this distribution, just because it’s fun. The lognormal distribution is an example of a distribution that’s not completely determined by knowledge of its moments. Most people assume that if you know all the moments of a distribution then that has to specify the distribution uniquely, but it ain’t necessarily so.

If you’re wondering why I mentioned citations, it’s because it looks like they’re going to play a big part in the Research Excellence Framework, yet another new bureaucratical exercise to attempt to measure the quality of research done in UK universities. Unfortunately, using citations isn’t straightforward. Different disciplines have hugely different citation rates, for one thing. Should one count self-citations?. Also how do you aportion citations to multi-author papers? Suppose a paper with a thousand citations has 25 authors. Does each of them get the thousand citations, or should each get 1000/25? Or, put it another way, how does a single-author paper with 100 citations compare to a 50 author paper with 101?

Or perhaps the REF should use the logarithm of the number of citations instead?

Follow

Get every new post delivered to your Inbox.

Join 3,271 other followers