The Impact of Impact

Interesting analysis of the 2014 REF results by my colleague Seb Oliver. Among other things, it shows that Physics was the subject in which “Impact had the greatest impact”..

Seb Boyd

I wrote the following article to explore how Impact in the Research Excellence Framework 2014 (REF2014) affected the average scores of departments (and hence rankings). This produced a “league table” of how strongly impact affected different subjects. Some of the information in this article was used in a THE article by Paul Jump due to come out 00:00 on 19th Feb 2015.  I’ve now also produced ranking tables for each UoA using the standardised weighting I advocate below (see Standardised Rankings).

UoA	Unit of Assessment	Effective Weight of GPA ranking in each sub-profile as %
Outputs	Impact	Envir.
9	Physics	37.9	38.6	23.5
23	Sociology	34.1	38.6	27.3
10	Mathematical Sciences	37.6	37.5	24.9
24	Anthropology and Development Studies	40.2	35.0	24.8
6	Agriculture, Veterinary and Food Science	42.0	33.0	25.0
31	Classics	43.3	32.6	24.0
16	Architecture, Built Environment and Planning	48.6	31.1	20.3

View original post 1,558 more words

Counting for the REF

It’s a lovely day in Brighton and I’m once again on campus for an Admissions Event at Sussex University, this time for the Mathematics Department in the School of Mathematical and Physical Sciences.  After all the terrible weather we’ve had since I arrived in February, it’s a delight and a relief to see the campus at its best for today’s crowds. Anyway, now that I’ve finished my talk and the subsequent chats with prospective students and their guests I thought I’d do a quick blogette before heading back home and preparing for this evenings Physics & Astronomy Ball. It’s all go around here.

What I want to do first of all is to draw attention to a very nice blog post by a certain Professor Moriarty who, in case you did not realise it, dragged himself away from his hiding place beneath the Reichenbach Falls and started a new life as Professor of Physics at Nottingham University.  Phil Moriarty’s piece basically argues that the only way to really judge the quality of a scientific publication is not by looking at where it is published, but by peer review (i.e. by getting knowledgeable people to read it). This isn’t a controversial point of view, but it does run counter to the current mania for dubious bibliometric indicators, such as journal impact factors and citation counts.

The forthcoming Research Excellence Framework involves an assessment of the research that has been carried out in UK universities over the past five years or so, and a major part of the REF will be the assessment of up to four “outputs” submitted by research-active members of staff over the relevant period (from 2008 to 2013). reading Phil’s piece might persuade you to be happy that the assessment of the research outputs involved in the REF will be primarily based on peer review. If you are then I suggest you read on because, as I have blogged about before, although peer review is fine in principle, the way that it will be implemented as part of the REF has me deeply worried.

The first problem arises from the scale of the task facing members of the panel undertaking this assessment. Each research active member of staff is requested to submit four research publications (“outputs”) to the panel, and we are told that each of these will be read by at least two panel members. The panel comprises 20 members.

As a rough guess let’s assume that the UK has about 40 Physics departments, and the average number of research-active staff in each is probably about 40. That gives about 1600 individuals for the REF. Actually the number of category A staff submitted to the 2008 RAE was 1,685.57 FTE (Full-Time Equivalent), pretty  close to this figure. At 4 outputs per person that gives 6400 papers to be read. We’re told that each will be read by at least two members of the panel, so that gives an overall job size of 12800 paper-readings. There is some uncertainty in these figures because (a) there is plenty of evidence that departments are going to be more selective in who is entered than was the case in 2008 and (b) some departments have increased their staff numbers significantly since 2008. These two factors work in opposite directions so not knowing the size of either it seems sensible to go with the numbers from the previous round for the purposes of my argument.

There are 20 members of the panel so 6400 papers submitted means that, between 29th November 2013 (the deadline for submissions) and the announcement of the results in December 2014 each member of the panel will have to have read 640 research papers. That’s an average of about two a day…

It is therefore blindingly obvious that whatever the panel does do will not be a thorough peer review of each paper, equivalent to refereeing it for publication in a journal. The panel members simply won’t have the time to do what the REF administrators claim they will do. We will be lucky if they manage a quick skim of each paper before moving on. In other words, it’s a sham.

Now we are also told the panel will use their expert judgment to decide which outputs belong to the following categories:

• 4*  World Leading
• 3* Internationally Excellent
• 2* Internationally Recognized
• 1* Nationally Recognized
• U   Unclassified

There is an expectation that the so-called QR  funding allocated as a result of the 2013 REF will be heavily weighted towards 4*, with perhaps a small allocation to 3* and probably nothing at all for lower grades. The word on the street is that the weighting for 4* will be 9 and that for 3* only 1. “Internationally recognized”  will be regarded as worthless in the view of HEFCE. Will the papers belonging to the category “Not really understood by the panel member” suffer the same fate?

The panel members will apparently know enough about every single one of the papers they are going to read in order to place them  into one of the above categories, especially the crucial ones “world-leading” or “internationally excellent”, both of which are obviously defined in a completely transparent and objective manner. Not. The steep increase in weighting between 3* and 4* means that this judgment could mean a drop of funding that could spell closure for a department.

We are told that after forming this judgement based on their expertise the panel members will “check” the citation information for the papers. This will be done using the SCOPUS service provided (no doubt at considerable cost) by   Elsevier, which by sheer coincidence also happens to be a purveyor of ridiculously overpriced academic journals.  No doubt Elsevier are  on a nice little earner peddling meaningless data for the HECFE bean-counters, but I have no confidence that they will add any value to the assessment process.

There have been high-profile statements to the effect that the REF will take no account of where the relevant “outputs”  are published, including a pronouncement by David Willetts. On the face of it, that would suggest that a paper published in the spirit of Open Access in a free archive would not be disadvantaged. However, I very much doubt that will be the case.

I think if you look at the volume of work facing the REF panel members it’s pretty clear that citation statistics will be much more important for the Physics panel than we’ve been led to believe. The panel simply won’t have the time or the breadth of understanding to do an in-depth assessment of every paper, so will inevitably in many cases be led by bibliometric information. The fact that SCOPUS doesn’t cover the arXiv means that citation information will be entirely missing from papers just published there.

The involvement of  a company like Elsevier in this system just demonstrates the extent to which the machinery of research assessment is driven by the academic publishing industry. The REF is now pretty much the only reason why we have to use traditional journals. It would be better for research, better for public accountability and better economically if we all published our research free of charge in open archives. It wouldn’t be good for academic publishing houses, however, so they’re naturally very keen to keep things just the way they are. The saddest thing is that we’re all so cowed by the system that we see no alternative but to participate in this scam.

Incidentally we were told before the 2008 Research Assessment Exercise that citation data would emphatically not be used;  we were also told afterwards that citation data had been used by the Physics panel. That’s just one of the reasons why I’m very sceptical about the veracity of some of the pronouncements coming out from the REF establishment. Who knows what they actually do behind closed doors?  All the documentation is shredded after the results are published. Who can trust such a system?

To put it bluntly, the apparatus of research assessment has done what most bureaucracies eventually do; it has become  entirely self-serving. It is imposing increasingly  ridiculous administrative burdens on researchers, inventing increasingly  arbitrary assessment criteria and wasting increasing amounts of money on red tape which should actually be going to fund research.

And that’s all just about “outputs”. I haven’t even started on “impact”….

Follow @telescoper

Reffing Madness

I’m motivated to make a quick post in order to direct you to a blog post by David Colquhoun that describes the horrendous behaviour of the management at Queen Mary, University of London in response to the Research Excellence Framework. It seems that wholesale sackings are in the pipeline there as a result of a management strategy to improve the institution’s standing in the league tables by “restructuring” some departments.

To call this strategy “flawed” would be the understatement of the year. Idiotic is a far better word.  The main problem being that the criteria being applied to retain or dismiss staff bear no obvious relation to those adopted by the REF panels. To make matters worse, Queen Mary has charged two of its own academics with “gross misconduct” for having the temerity to point out the stupidity of its management’s behaviour. Read on here for more details.

With the deadline for REF submissions fast approaching, it’s probably the case that many UK universities are going into panic mode, attempting to boost their REF score by shedding staff perceived to be insufficiently excellent in research and/or  luring  in research “stars” from elsewhere. Draconian though the QMUL approach may seem, I fear it will be repeated across the sector.  Clueless university managers are trying to guess what the REF panels will think of their submissions by staging mock assessments involving external experts. The problem is that nobody knows what the actual REF panels will do, except that if the last Research Assessment Exercise is anything to go by, what they do will be nothing like what they said they would do.

Nowhere is the situation more absurd than here in Wales. The purported aim of the REF is to allocated the so-called “QR” research funding to universities. However, it is an open secret that in Wales there simply isn’t going to be any QR money at all. Leighton Andrews has stripped the Higher Education budget bare in order to pay for his policy of encouraging Welsh students to study in England by paying their fees there.

So here we have to enter the game, do the mock assessments, write our meaningless “impact” cases, and jump through all manner of pointless hoops, with the inevitable result that even if we do well we’ll get absolutely no QR money at the end of it. The only strategy that makes sense for Welsh HEIs such as Cardiff University, where I work, is to submit only those researchers guaranteed to score highly. That way at least we’ll do better in the league tables. It won’t matter how many staff actually get submitted, as the multiplier is zero.

There’s no logical argument why Welsh universities should be in the REF at all, given that there’s no reward at the end. But we’re told we have to by the powers that be. Everyone’s playing games in which nobody knows the rules but in which the stakes are people’s careers. It’s madness.

I can’t put it better than this quote:

These managers worry me. Too many are modest achievers, retired from their own studies, intoxicated with jargon, delusional about corporate status and forever banging the metrics gong. Crucially, they don’t lead by example.

Any reader of this blog who works in a university will recognize the sentiments expressed there. But let’s not blame it all on the managers. They’re doing stupid things because the government has set up a stupid framework. There isn’t a single politician in either England or Wales with the courage to do the right thing, i.e. to admit the error and call the whole thing off.

Follow @telescoper

The Transparent Dishonesty of the Research Excellence Framework

Some of my colleagues in the School of Physics & Astronomy recently attended a briefing session about the  forthcoming Research Excellence Framework. This, together with the post I reblogged earlier this morning, suggested that I should re-hash an article I wrote some time ago about the arithmetic of the REF, and how it will clearly not do what it says on the tin.

The first thing is the scale of the task facing members of the panel undertaking the assessment. Every research active member of staff in every University in the UK is requested to submit four research publications (“outputs”) to the panel, and we are told that each of these will be read by at least two panel members. The Physics panel comprises 20 members.

As a rough guess I’d say that the UK has about 40 Physics departments, and the average number of research-active staff in each is probably about 40. That gives about 1600 individuals for the REF. Actually the number of category A staff submitted to the 2008 RAE was 1,685.57 FTE (Full-Time Equivalent), pretty close to this figure. At 4 outputs per person that gives 6400 papers to be read. We’re told that each will be read by at least two members of the panel, so that gives an overall job size of 12800 paper-readings. There are 20 members of the panel, so that means that between 29th November 2013 (the deadline for submissions) and the announcement of the results in December 2014 each member of the panel will have to have read 640 research papers. That’s an average of about two a day. Every day. Weekends included.

Now we are told the panel will use their expert judgment to decide which outputs belong to the following categories:

• 4*  World Leading
• 3* Internationally Excellent
• 2* Internationally Recognized
• 1* Nationally Recognized
• U   Unclassified

There is an expectation that the so-called QR  funding allocated as a result of the 2013 REF will be heavily weighted towards 4*, with perhaps a small allocation to 3* and probably nothing at all for lower grades. In other words “Internationally recognized” research will probably be deemed completely worthless by HEFCE. Will the papers belonging to the category “Not really understood by the panel member” suffer the same fate?

The panel members will apparently know enough about every single one of the papers they are going to read in order to place them  into one of the above categories, especially the crucial ones “world-leading” or “internationally excellent”, both of which are obviously defined in a completely transparent and objective manner. Not.

We are told that after forming this judgement based on their expertise the panel members will “check” the citation information for the papers. This will be done using the SCOPUS service provided (no doubt at considerable cost) by   Elsevier, which by sheer coincidence also happens to be a purveyor of ridiculously overpriced academic journals. No doubt Elsevier are  on a nice little earner peddling meaningless data for the HECFE bean-counters, but I haven’t any confidence that it will add much value to the assessment process.

There have been high-profile statements to the effect that the REF will take no account of where the relevant “outputs”  are published, including a recent pronouncement by David Willetts. On the face of it, that would suggest that a paper published in the spirit of Open Access in a free archive would not be disadvantaged. However, I very much doubt that will be the case.

I think if you look at the volume of work facing the REF panel members it’s pretty clear that citation statistics will be much more important for the Physics panel than we’ve been led to believe. The panel simply won’t have the time or the breadth of understanding to do an in-depth assessment of every paper, so will inevitably in many cases be led by bibliometric information. The fact that SCOPUS doesn’t cover the arXiv means that citation information will be entirely missing from papers just published there.

The involvement of  a company like Elsevier in this system just demonstrates the extent to which the machinery of research assessment is driven by the academic publishing industry. The REF is now pretty much the only reason why we have to use traditional journals. It would be better for research, better for public accountability and better economically if we all published our research free of charge in open archives. It wouldn’t be good for academic publishing houses, however, so they’re naturally very keen to keep things just the way they are. The saddest thing is that we’re all so cowed by the system that we see no alternative but to participate in this scam.

Incidentally we were told before the 2008 Research Assessment Exercise that citation data would emphatically not be used;  we were also told afterwards that citation data had been used by the Physics panel. That’s just one of the reasons why I’m very sceptical about the veracity of some of the pronouncements coming out from the REF establishment. Who knows what they actually do behind closed doors?  All the documentation is shredded after the results are published. Who can trust such a system?

To put it bluntly, the apparatus of research assessment has done what most bureaucracies eventually do; it has become  entirely self-serving. It is imposing increasingly  ridiculous administrative burdens on researchers, inventing increasingly  arbitrary assessment criteria and wasting increasing amounts of money on red tape which should actually be going to fund research.

Follow @telescoper

Come off it, REF!

Yesterday we all trooped off to the Millennium Stadium in Cardiff for a Staff Away Day. We didn’t actually get to play on the pitch of course, which wasn’t even there, as it had been removed to reveal a vast expanse of soil. Instead we were installed in the “Dragon Suite” for a discussion about our preparation for the forthcoming Research Excellence Framework.

Obviously I can’t post anything about our internal deliberations, but I’m sure departments up and down the United Kingdom are doing similar things so I thought I’d mention a few things which are already in the public domain and my personal reactions to them. I should also say that the opinions I express below are my own and not necessarily those of anyone else at Cardiff.

The first thing is the scale of the task facing members of the panel undertaking this assessment. Each research active member of staff is requested to submit four research publications (“outputs”) to the panel, and we are told that each of these will be read by at least two panel members. The panel comprises 20 members.

As a rough guess I’d say that the UK has about 40 Physics departments, and the average number of research-active staff in each is probably about 40. That gives about 1600 individuals for the REF. Actually the number of category A staff submitted to the 2008 RAE was 1,685.57 FTE (Full-Time Equivalent), pretty  close to this figure. At 4 outputs per person that gives 6400 papers to be read. We’re told that each will be read by at least two members of the panel, so that gives an overall job size of 12800 paper-readings. There are 20 members of the panel, so that means that between 29th November 2013 (the deadline for submissions) and the announcement of the results in December 2014 each member of the panel will have to have read 640 research papers. That’s an average of about two a day…

Incidentally, as I’ve mentioned before, the Physics REF panel includes representatives from institutions in England, Scotland and Northern Ireland, but not Wales. The decision to exclude representation from Welsh physics departments was a disgrace, in my view.

Now we are told the panel will use their expert judgment to decide which outputs belong to the following categories:

• 4*  World Leading
• 3* Internationally Excellent
• 2* Internationally Recognized
• 1* Nationally Recognized
• U   Unclassified

There is an expectation that the so-called QR  funding allocated as a result of the 2013 REF will be heavily weighted towards 4*, with perhaps a small allocation to 3* and probably nothing at all for lower grades. “Internationally recognized” research is probably worthless in the view of HEFCE, in other words. Will the papers belonging to the category “Not really understood by the panel member” suffer the same fate?

The panel members will apparently know enough about every single one of the papers they are going to read in order to place them  into one of the above categories, especially the crucial ones “world-leading” or “internationally excellent”, both of which are obviously defined in a completely transparent and objective manner. Not.

We are told that after forming this judgement based on their expertise the panel members will “check” the citation information for the papers. This will be done using the SCOPUS service provided (no doubt at considerable cost) by   Elsevier, which by sheer coincidence also happens to be a purveyor of ridiculously overpriced academic journals. I’ve just checked the citation information for some of my papers on SCOPUS, and found an alarming number of errors. No doubt Elsevier are  on a nice little earner peddling meaningless data for the HECFE bean-counters, but I haven’t any confidence that it will add much value to the assessment process.

There have been high-profile statements to the effect that the REF will take no account of where the relevant “outputs”  are published, including a recent pronouncement by David Willetts. On the face of it, that would suggest that a paper published in the spirit of Open Access in a free archive would not be disadvantaged. However, I very much doubt that will be the case.

I think if you look at the volume of work facing the REF panel members it’s pretty clear that citation statistics will be much more important for the Physics panel than we’ve been led to believe. The panel simply won’t have the time or the breadth of understanding to do an in-depth assessment of every paper, so will inevitably in many cases be led by bibliometric information. The fact that SCOPUS doesn’t cover the arXiv means that citation information will be entirely missing from papers just published there.

The involvement of  a company like Elsevier in this system just demonstrates the extent to which the machinery of research assessment is driven by the academic publishing industry. The REF is now pretty much the only reason why we have to use traditional journals. It would be better for research, better for public accountability and better economically if we all published our research free of charge in open archives. It wouldn’t be good for academic publishing houses, however, so they’re naturally very keen to keep things just the way they are. The saddest thing is that we’re all so cowed by the system that we see no alternative but to participate in this scam.

Incidentally we were told before the 2008 Research Assessment Exercise that citation data would emphatically not be used;  we were also told afterwards that citation data had been used by the Physics panel. That’s just one of the reasons why I’m very sceptical about the veracity of some of the pronouncements coming out from the REF establishment. Who knows what they actually do behind closed doors?  All the documentation is shredded after the results are published. Who can trust such a system?

To put it bluntly, the apparatus of research assessment has done what most bureaucracies eventually do; it has become  entirely self-serving. It is imposing increasingly  ridiculous administrative burdens on researchers, inventing increasingly  arbitrary assessment criteria and wasting increasing amounts of money on red tape which should actually be going to fund research.

And that’s all just about “outputs”. I haven’t even started on “impact”….

Follow @telescoper

An early draft of the UK Space Agency logo

Negative Impact

After spending the best part of the last couple of days being prodded and poked and subjected to all manner of indignity in the name of medical science, I think it’s appropriate to return to the blogosphere with another rant. Before I start, however, I’d seriously like to thank everyone at the University Hospital of Wales at Heath Park  for making my visit there as brief and painless as possible. Everyone was very kind and very efficient. I’m not going to blog about the details, as Columbo doesn’t like reading about other peoples’ ailments.

Over the past few weeks there has been a lot of discussion about the UK government’s agenda for research, particularly science research, that includes something called “impact”. The Research Excellence Framework (REF; successor to the Research Assessment Exercise, RAE) will include such a thing:

Significant additional recognition will be given where researchers build on excellent research to deliver demonstrable benefits to the economy, society, public policy, culture and quality of life

Apparently, however, they don’t really know how to do this so they have set up a number of pilot studies to try to find out. I’d feel a little more comfortable if the bureaucrats had thought about what they were going to do before announcing that our future research funds were going to depend on it. Meanwhile, applicants for grants from any of the research councils must  include a statement of the “economic or social” impact their research will have.

Understandably, those of us working in “blue skies” research are very nervous about this new regime. There is more than a suspicion that the new emphasis on impact is intended to divert funds away from “pure” curiosity driven research and into areas where it can have an immediately identifiable short-term economic benefit. This has led to a petition, with over 13000 signatures, by the University and College Union calling for the impact statements to be abandoned.

I don’t know who is going to assess these impact statements, but unless they have a flawless ability to predict future technology I don’t think fundamental physics is going to score very well at all. To see my point, consider the case of  J. J. Thomson, who is generally credited with having discovered the electron and who was awarded the Nobel Prize for Physics in 1906. Thomson made extensive use of cathode ray tubes in his studies; these later found their way into sitting rooms across the world as essential components of the classic television set. But that took decades. I doubt if an impact panel looking at Thomson’s work – even if they were physicists rather than grey-suited bureaucrats – would have found any of it likely to lead to immediate economic benefit. The point is that when he discovered the electron it wasn’t because he was actually trying to invent the television set.

I think there are basically two possible interpretations of this impact business. One is that it is a deliberate plan to wind down fundamental research and use the money saved to subsidise UK industry. The other is that it’s another exercise in pointless box-ticking. I am in two minds. On the one hand, it is clear that the recent behaviour of the Science and Technology Facilities Council shows strong evidence of the former. Fundamental research is being slashed, yet projects involving space technology have been funded on the nod without scientific  peer review. On the other hand, the RCUK Impact “Champion”, a person by the name of David Delpy, has written in the Times Higher to defend the new agenda. Consider the following paragraph

Recently I have read that some believe it is impossible to predict the economic impact of blue-skies research. To be clear, we are not asking for accurate predictions – simply a consideration of potential. Basic research underpins all disciplines and builds pathways to new technologies with economic and social applications. It may build on an existing body of knowledge, connect to other research around the world or attract new industries to the UK. There are many routes to impact. I believe that I could write a statement indicating potential impact for any proposal I have seen, and to hear that bright academics say they can’t do it sounds a little disingenuous.

Champion Delpy thus suggests he could write a statement for any proposal he has seen, which sounds to me like an admission that what is called for is just a load of flannel. In fact, if he’s paid to be the Impact Champion perhaps he should write all the bullshit and save us scientists the need to jump through these silly hoops? Or perhaps we could get one of those little Microsoft Office Assistant things:

Hello. Looks like you’re writing an Impact Assessment. Would you like me to pad it out with meaningless but impressive-looking socio-economic buzzwords for you?

If it’s just another exercise in vacuous bureaucracy then it’s bad enough, but if it is the other possibility then of course it’s even worse. It could be the end for disciplines like astronomy and particle physics as well as the end of Britain’s history of excellence in those areas. I’ve already blogged about my view of short-termism in research funding. Essentially, my point is that government money should be used to fund precisely those things that don’t have immediate economic benefit. Those that do should be funded by the beneficiaries, i.e. commercial companies.

Politicians probably think that all this complaining about impact means that scientists  are arrogantly assuming that the taxpayer should fund them regardless of the cost or the benefit. I can only speak for myself, but I think that’s very unfair. I’m very conscious that my research is funded by Joe Public; that’s one of the reasons I think I should spend time giving public talks and doing other outreach activities. But I think the public funds me and others like me to do “useless” things because, in the end, useless things are more important than money.

The government is probably right to say that the UK economy doesn’t benefit as much from our scientific expertise as is the case with other countries. The reason for that, however, lies not with our universities and research laboratories but with our private industrial and commercial sectors which are, for the most part, managed with a very low level of competence. British universities are demonstrably excellent; our industry is demonstrably feeble. The persistent failure of the private sector to invest in research and development shows that it is in drastic need of a good shake up. British companies, not the taxpayer, should be paying for research that leads to profit for them and for that to happen they will have to learn to engage better with the University sector rather than expecting inventions to be served up on a plate funded by the taxpayer. Universities and research labs should continue do what they’re good at,  maintaining a culture within which curiosity and learning are promoted for their own sake not just as part of the dreary materialistic cycle of production and consumption that is all we seem to be able to think about these days.

So at the end I’ve come to the conclusion that, perhaps, insofar as it can be demonstrated, economic impact should be included in the forthcoming Research Excellence Framework. Research which leads directly to the economic gain of the private sector is  precisely the type of research that the taxpayer should not be paying for. If it can be proven that a given department has engaged in such activity, its state funding should therefore be cut and it should be told to recover the funds it has misused from the company that has benefitted from it. Economic impact should be included with a negative weight.

And if you think that’s a silly point of view, consider what happens with the other major part of a university’s activity, teaching. Students, we are told, are the primary beneficiaries of their education so they should have to pay fees. In the current regime, however, they only do so when their earnings reach a certain level. If commercial companies are to be the primary beneficiaries of state-funded research, why should they not likewise be asked to pay for it?

Index Rerum

Following on from yesterday’s post about the forthcoming Research Excellence Framework that plans to use citations as a measure of research quality, I thought I would have a little rant on the subject of bibliometrics.

Recently one particular measure of scientific productivity has established itself as the norm for assessing job applications, grant proposals and for other related tasks. This is called the h-index, named after the physicist Jorge Hirsch, who introduced it in a paper in 2005. This is quite a simple index to define and to calculate (given an appropriately accurate bibliographic database). The definition  is that an individual has an h-index of  h if that individual has published h papers with at least h citations. If the author has published N papers in total then the other N-h must have no more than h citations. This is a bit like the Eddington number.  A citation, as if you didn’t know,  is basically an occurrence of that paper in the reference list of another paper.

To calculate it is easy. You just go to the appropriate database – such as the NASA ADS system – search for all papers with a given author and request the results to be returned sorted by decreasing citation count. You scan down the list until the number of citations falls below the position in the ordered list.

Incidentally, one of the issues here is whether to count only refereed journal publications or all articles (including books and conference proceedings). The argument in favour of the former is that the latter are often of lower quality. I think that is in illogical argument because good papers will get cited wherever they are published. Related to this is the fact that some people would like to count “high-impact” journals only, but if you’ve chosen citations as your measure of quality the choice of journal is irrelevant. Indeed a paper that is highly cited despite being in a lesser journal should if anything be given a higher weight than one with the same number of citations published  in, e.g., Nature. Of course it’s just a matter of time before the hideously overpriced academic journals run by the publishing mafia go out of business anyway so before long this question will simply vanish.

The h-index has some advantages over more obvious measures, such as the average number of citations, as it is not skewed by one or two publications with enormous numbers of hits. It also, at least to some extent, represents both quantity and quality in a single number. For whatever reasons in recent times h has undoubtedly become common currency (at least in physics and astronomy) as being a quick and easy measure of a person’s scientific oomph.

Incidentally, it has been claimed that this index can be fitted well by a formula h ~ sqrt(T)/2 where T is the total number of citations. This works in my case. If it works for everyone, doesn’t  it mean that h is actually of no more use than T in assessing research productivity?

Typical values of h vary enormously from field to field – even within each discipline – and vary a lot between observational and theoretical researchers. In extragalactic astronomy, for example, you might expect a good established observer to have an h-index around 40 or more whereas some other branches of astronomy have much lower citation rates. The top dogs in the field of cosmology are all theorists, though. People like Carlos Frenk, George Efstathiou, and Martin Rees all have very high h-indices.  At the extreme end of the scale, string theorist Ed Witten is in the citation stratosphere with an h-index well over a hundred.

I was tempted to put up examples of individuals’ h-numbers but decided instead just to illustrate things with my own. That way the only person to get embarrased is me. My own index value is modest – to say the least – at a meagre 27 (according to ADS).   Does that mean Ed Witten is four times the scientist I am? Of course not. He’s much better than that. So how exactly should one use h as an actual metric,  for allocating funds or prioritising job applications,  and what are the likely pitfalls? I don’t know the answer to the first one, but I have some suggestions for other metrics that avoid some of its shortcomings.

One of these addresses an obvious deficiency of h. Suppose we have an individual who writes one brilliant paper that gets 100 citations and another who is one author amongst 100 on another paper that has the same impact. In terms of total citations, both papers register the same value, but there’s no question in my mind that the first case deserves more credit. One remedy is to normalise the citations of each paper by the number of authors, essentially sharing citations equally between all those that contributed to the paper. This is quite easy to do on ADS also, and in my case it gives  a value of 19. Trying the same thing on various other astronomers, astrophysicists and cosmologists reveals that the h index of an observer is likely to reduce by a factor of 3-4 when calculated in this way – whereas theorists (who generally work in smaller groups) suffer less. I imagine Ed Witten’s index doesn’t change much when calculated on a normalized basis, although I haven’t calculated it myself.

Observers  complain that this normalized measure is unfair to them, but I’ve yet to hear a reasoned argument as to why this is so. I don’t see why 100 people should get the same credit for a single piece of work:  it seems  like obvious overcounting to me.

Another possibility – if you want to measure leadership too – is to calculate the h index using only those papers on which the individual concerned is the first author. This is  a bit more of a fiddle to do but mine comes out as 20 when done in this way.  This is considerably higher than most of my professorial colleagues even though my raw h value is smaller. Using first author papers only is also probably a good way of identifying lurkers: people who add themselves to any paper they can get their hands on but never take the lead. Mentioning no names of  course.  I propose using the ratio of  unnormalized to normalized h-indices as an appropriate lurker detector…

Finally in this list of bibliometrica is the so-called g-index. This is defined in a slightly more complicated way than h: given a set of articles ranked in decreasing order of citation numbers, g is defined to be the largest number such that the top g articles altogether received at least g² citations. This is a bit like h but takes extra account of the average citations of the top papers. My own g-index is about 47. Obviously I like this one because my number looks bigger, but I’m pretty confident others go up even more than mine!

Of course you can play with these things to your heart’s content, combining ideas from each definition: the normalized g-factor, for example. The message is, though, that although h definitely contains some information, any attempt to condense such complicated information into a single number is never going to be entirely successful.

Comments, particularly with suggestions of alternative metrics are welcome via the box. Even from lurkers.