The Citation Game

Last week I read an interesting bit of news in the Times Higher that the forthcoming Research Excellence Framework (REF) seems to be getting cold feet about using citation numbers as a metric for quantifying research quality. I shouldn’t be surprised about that, because I’ve always thought it was very difficult to apply such statistics in a meaningful way. Nevertheless, I am surprised – because meaningfulness has never seemed to me to be very high on the agenda for the Research Excellence Framework….

There are many issues with the use of citation counts, some of which I’ve blogged about before, but I was interested to read another article in the Times Higher, in this weeks issue, commenting on the fact that some papers have ridiculously large author lists. The example picked by the author, Gavin Fairbairn (Professor of Ethics and Language at Leeds Metropolitan University), turns out – not entirely surprisingly – to be from the field of astronomy. In fact it’s The Sloan Digital Sky Survey: Technical Summary which is published in the Astronomical Journal and has 144 authors. It’s by no means the longest author list I’ve ever seen, in fact, but it’s certainly very long by the standards of the humanities. Professor Fairbairn goes on to argue, correctly, that there’s no way every individual listed among the authors could have played a part in the writing of the paper. On the other hand, the Sloan Digital Sky Survey is a vast undertaking and there’s no doubt that it required a large number of people to make it work. How else to give them credit for participating in the science than by having them as authors on the paper?

Long author lists are increasingly common in astronomy these days, not because of unethical CV-boosting but because so many projects involve large, frequently international, collaborations. The main problem from my point of view, however, is not the number of authors, but how credit is assigned for the work in exercises like the REF.

The basic idea about using citations is fairly sound: a paper which is important (or “excellent”, in REF language) will attract more citations than less important ones because more people will refer to it when they write papers of their own. So far, so good. However the total number of citations for even a very important paper depends on the size and publication rate of the community working in the field. Astronomy is not a particularly large branch of the physical sciences but is very active and publication rates are high, especially when it comes to observational work.  In condensed matter physics citation rates are generally a lot lower, but that’s more to do with the experimental nature of the subject. It’s not easy, therefore, to compare from one field to another. Setting that issue to one side, however, we come to the really big issue, which is how to assign credit to authors.

You see, it’s not authors that get citations, it’s papers. Let’s accept that a piece of work might be excellent and that this excellence can be quantified by the number of citations N it attracts. Now consider a paper written by a single author that has excellence-measure N versus a paper with 100 authors that has the same number of citations. Don’t you agree that the individual author of the first paper must have generated more excellence than each of the authors of the second? It seems to me that it stands to reason that the correct way to apportion credit is to divide the number of citations by the number of authors (perhaps with some form of weighting to distinguish drastically unequal contributions). I contend that such a normalized citation count is the only way to quantify the excellence associated with an individual author.

Of course whenever I say this to observational astronomers they accuse me of pro-theory bias, because theorists tend to work in smaller groups than observers. However, that ignores the fact that not doing what I suggest leads to a monstrous overcounting of the total amout of excellence. The total amount of excellence spread around the community for the second paper in my example is not N but 100N. Hardly surprising, then, that observational astronomers tend to have such large h-indices – they’re all getting credit for each others contributions as well as their own! Most observational astronomers’ citation measures reduce by a factor of 3 or 4 when they’re counted properly.

I think of the citation game as being a bit like the National Lottery. Writing a paper is like buying a ticket. You can buy one yourself, or you can club together and buy one as part of a syndicate. If you win with your own ticket, you keep the whole jackpot. If a syndicate wins, though, you don’t expect each member to win the total amount – you have to share the pot between you.

Advertisements

17 Responses to “The Citation Game”

  1. “It seems to me to stand to reason that the correct way to apportion credit is to divide the number of citations by the number of authors (perhaps with some form of weighting to distinguish drastically unequal contributions). I contend that such a normalized citation count is the only way to quantify the excellence associated with an individual author.”

    First point: There is a typo (corrected in my quotation).

    Second point: What you say is so obvious that it is hard to believe that anyone can seriously question it.

    Third point: I think some additional weighting needs to be done, perhaps (as a very rough measure) by the number of pages. Let’s say 50 people right up a 50-page article describing a big survey or something. They each get 1/50 of a paper. On the other hand, a single author of a one-page paper gets a full paper. All else being equal, surely each author of the big paper has done a comparable amount of work to the single author of the one-page paper. (One might argue that this would encourage people to write more verbose papers. I don’t think this is a big problem, though, since a) editors could easily keep it in check and b) it is certainly no worse than the tendency to publish a number of small papers instead of one big paper in order to increase the paper count, each small paper being, to paraphrase Einstein, as short as possible but not shorter.)

    Fourth point: As you note, at best one can use bibliometry to compare authors WITHIN a field, not BETWEEN fields (though there might be a simple conversion factor from one field to another).

    Fifth point: Even with perfect bibliometry, there is the question of who is an author at all, in particular how much work is necessary to move from being mentioned in the acknowledgments to being a co-author. This varies greatly, even within a field, from country to country, institute to institute, collaboration to collaboration.

  2. “50 people write up”, of course, not “50 people right up”.

  3. I worry that the adoption of your approach will lead to a bias away from large projects like SDSS, and satellite missions like Planck & Herschel. The UK would then lose it’s prominent place in observational and instrumental astronomy. Great for you, maybe, but less so for those working for many years on large long term projects. And it would kill particle physics whose projects have more authors even than observational astronomy.

    • telescoper Says:

      I disagree. It would simply remove the present bias in favour of large projects which is killing off the more individual kinds of research that often lead to the greatest breakthroughs. Consortia like SDSS, 2dFGRS and, hopefully soon, Planck and Herschel still generate papers with large numbers of citations, even when normalised by the number of authors. People involved in these projects will still get healthy credit, in proportion to the value of such work. However, the current pro-consortium bias is leading to an unhealthy concentration of effort in mass-produced, incremental research which is not good for the development of the subject. The REF in particular needs to recognize the importance of individual contributions and use appropriate measures to quantify it.

  4. Woken Postdoc Says:

    I have misgivings about the basic premise that citations measure a paper’s merit (regardless of how the authors are weighted). Citation stats measure an aspect of popularity, which might or might not correspond to truth or worth. Fame breeds fame, and this bias needs to be normalised somehow. The preverse pursuit of citations may even (arguably) slow the advance of true science, by enhancing fads and rewarding groupthink.

    The citation network is vulnerable to intentional manipulation. Citation “black holes” are easily identifiable. Particularly mendacious collaborations deliberately under-cite the contributions of rivals and predecessors, and over-cite even the tiniest, repetitive letters from their own crew. The effect is that randomly chosen links in the web lead irreversibly inside one of several great incestuous knots.

    In order to penalise such misbehaviour, and to assess the centrality of a paper within the body of knowledge, I believe it’ll be necessary to apply graph theory somehow. The mean number of connections between one researcher and all others would reveal who influences the widest range of topics. OTOH, this might reward a few promiscuous, freeloading networkers who don’t perform much work (or thought) themselves.

    Maybe individual performance assessments should be based on first-author or single-author papers? Science is essentially about a contest of ideas (proposing testable theories, then evaluating evidence), and the first author is likely to be the main thinker. First-author statistics naturally demote freeloaders and camp-followers. Publishing a single-author paper is an even clearer sign of intellectual leadership (and IMHO ought to be a necessary minimum for a permanent job).

    One day, I would like to discover a fair way to penalise para-scientific papers — those that merely count polka-dots and butterflies (e.g. reiterations of huge surveys, and repetitive single-object papers). I am afraid that this type of activity is what STFC prefers. If you wander around any of the departments visited by the ongoing grant massacre, I believe you’ll notice that it’s the meek, facility-driven para-science that’s tending to survive. Many vibrantly creative and famous professors seem blind to the problem: they’re happily breeding herds of data-donkeys, instead of imaginative, worthy, potential successors.

  5. Woken Postdoc Says:

    Afterthought. The problem of these bibliometric distortions isn’t just a matter of theory vs observation. Simulation papers with vast author-lists raise the same difficulties.

    One suspects that some papers are written partly for the sake of showing off a big computer, just as some observational papers are written to “exploit” an existing instrument. The depressing common factor is that the scientific message or proposition often becomes less important than the tool or the medium.

  6. telescoper Says:

    There’s a lot of truth in those two comments. Let me just pick up on one point.

    I think citations do contain information about the impact of a paper but, as you say, bibliometrics can be distorted either deliberately or by its own inherent nonlinearity. One way of reducing the effect of self-promotion would be to remove auto-citations. By this I mean something stronger than what is usually suggested.

    Imagine scientist A is one of the authors of Paper I. This paper is then cited by paper II. Normally this is considered a self-citation if author A is also an author of paper II. However, I would define an auto-citation to be one in which any author of Paper II appears in the list of Paper I. This auto-citation should not be included in author A’s bibliometric regardless of whether that individual was involved in the referring paper.

  7. Anton Garrett Says:

    The variety – and ingenuity – of proposals suggests to me that this is an attempt to quantify the unquantifiable. What is and is not good science, or great science, is ultimately a matter of connoisseurship. The pressure for bibliometrics is due to a laudable desire for transparency, but is quantification possible? Might it be that the axioms of any desirable bibliometric system are mutually contradictory?

    I was told that a while ago US rowing eights shifted from selection by coach to selection by ergometer rating. They did so because excluded oarsmen with higher ergo ratings were threatening to sue. Since then they have not done anything like so well in world championships. Unquantifiables like how well somebody’s rowing style suits the rest of the boat get neglected, but they obviously count. I think the analogy is a good one. Theoretical physics is notorious for anomalies in which credit goes to the researcher who discovered something second.

    Anton

  8. telescoper Says:

    Anton,

    I think citation statistics are interesting, and they do contain some information about the impact of research. The problem is that the powers that be want to distil it all down into one number, which can’t be done.

    I think quantitative information is useful input into discussions of, e.g. grant funding and promotions but what is useful information depends strongly on the context.

    For example if you want to appoint a postdoc to support research on a large project then their total citation numbers even if they are related to multi-author papers might be relevant as it shows that they have collaborated successfully in teams, which is surely a requirement of such a job.

    However, suppose instead you are looking to decide whether to appoint someone to a chair (or promote them internally). For this, one should be looking for the ability to direct research and come up with original ideas. I strongly feel that normalised citation counts should be used intsead of raw ones for this.

    I know this is going to ruffle a few feathers, but I can think of many astronom,ers who have obtained senior permanent positions in academia owing to high citation counts from multi-author papers which have misinterpreted as evidence of research leadership when it just means they jumped on the right bandwagon. If this carries on, the result will be departments stuffed full of permanent academics that wouldn’t know an original thought if it hit them in the face.

    Peter

  9. Anton Garrett Says:

    Peter,

    Without necesssarily disagreeing, if I wanted to find out whether Dr X was a good team player, I would not see how many multi-author papers he appeared in. I would ring up several of his collaborators.

    Since it’s feather-ruffling time, I’d like to generalise the subject and suggest that economics is a discipline currently hobbled by its attempts to quantify the unquantifiable. Many people assume that, because economics involves certain quantitative variables (amounts of money, interest rates, exchange rates, gearing ratios, inflation rates) then it must be in analogy with physics, so that formulae should be sought that relate these variables, and evolution equations derived. The analogy fails because these quantitative variables are in interaction with other factors that cannot be quantified, such as investor confidence and political (or boardroom) factors and decisions. To use 18th century categories, economics (or oeconomics) is a ‘moral science’ rather than a ‘natural science’ like physics. This was well understood by its great pre-quantitative thinker Adam Smith, who repeatedly insisted on the caveat that his analysis of market forces worked only in a moral framework – where people kept their pledges to pay, and contract law was enforced. Economics *is* a genuine subject, but econometrics is largely bullshit.

    Anton

  10. telescoper Says:

    Anton,

    You may just have hit upon a fundamental law. Maybe everything that ends in -metrics is a load of bollocks!

    Econometrics, bibliometrics, psychometrics…can you think of any more?

    This could easily be the case because tacking on -metrics to the end of another word is probably just an attempt to dress something unquantifiable in quantitative terms. Subjects that are eseentially quantitative don’t need to apply this veneer…

    Peter

  11. Anton Garrett Says:

    Nice generalisation Peter. The deepest discussion of Adam Smith I know is not by any economist or philosopher, but the social anthropologist Alan Macfarlane, at

    http://www.alanmacfarlane.com/TEXTS/Smith_final.pdf

    Anton

  12. Yes – all these citation metrics are trying to quantify the unquantifiable. It’s part of the overall accountantization of many professions in the UK, from health to schools to, now, universities. The bottom line is that professionals are not trusted by government, at least not the current lot, and they’d rather measure something measurable, like citation statistics, than ask someone in the field what is a genuinely good paper. And any formal system can, as has been noted, be gamed to mess with the result.

    My comments onlarge collaborations were more along the lines that this is all unmeasurable by metrics anyway, though I remain confused why Peter seems to think only small groups or lone authors can be innovative.

  13. telescoper Says:

    Dave,

    You are indeed confused. I don’t think that “only small groups or lone authors can be innovative”. What is the case is that it is easier to identify where the ideas come from in a small group than in a large one.

    Peter

  14. Anton Garrett Says:

    Dave – I agree absolutely with the first paragraph of your 1:11pm contribution above – Anton

  15. Anton Garrett Says:

    Peter,

    You say above that “the powers that be want to distil it all down into one number, which can’t be done”. Let me *partly* recant my statements about quantifying the unquantifiable and say that I think it *is* possible in principle, because it is legitimate to order grant applications and there must therefore be an isomorphism with the positive reals. It’s just that it can’t be done in practice, any more than a jury can attach a number to the probability that somebody is guilty in a court case. Human factors call for human decision-makers.

    Anton

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: