What Counts as Productivity?

Apparently last year the United Kingdom Infra-Red Telescope (UKIRT) beat its own personal best for scientific productivity. In fact here’s a  graphic showing the number of publications resulting from UKIRT to make the point:

The plot also demonstrates that a large part of recent burst of productivity has been associated with UKIDSS (the UKIRT Infrared Deep Sky Survey) which a number of my colleagues are involved in. Excellent chaps. Great project. Lots of hard work done very well.  Take a bow, the UKIDSS team!

Now I hope I’ve made it clear that  I don’t in any way want to pour cold water on the achievements of UKIRT, and particularly not UKIDSS, but this does provide an example of how difficult it is to use bibliometric information in a meaningful way.

Take the UKIDSS papers used in the plot above. There are 226 of these listed by Steve Warren at Imperial College. But what is a “UKIDSS paper”? Steve states the criteria he adopted:

A paper is listed as a UKIDSS paper if it is already published in a journal (with one exception) and satisfies one of the following criteria:

1. It is one of the core papers describing the survey (e.g. calibration, archive, data releases). The DR2 paper is included, and is the only paper listed not published in a journal.
2. It includes science results that are derived in whole or in part from UKIDSS data directly accessed from the archive (analysis of data published in another paper does not count).
3. It contains science results from primary follow-up observations in a programme that is identifiable as a UKIDSS programme (e.g. The physical properties of four ~600K T dwarfs, presenting Spitzer spectra of cool brown dwarfs discovered with UKIDSS).
4. It includes a feasibility study of science that could be achieved using UKIDSS data (e.g. The possiblity of detection of ultracool dwarfs with the UKIRT Infrared Deep Sky Survey by Deacon and Hambly).

Papers are identified by a full-text search for the string ‘UKIDSS’, and then compared against the above criteria.

That all seems to me to by quite reasonable, and it’s certainly one way of defining what a UKIDSS paper is. According to that measure, UKIDSS scores 226.

The Warren measure does, however, include a number of papers that don’t directly use UKIDSS data, and many written by people who aren’t members of the UKIDSS consortium. Being picky you might say that such papers aren’t really original UKIDSS papers, but are more like second-generation spin-offs. So how could you count UKIDSS papers differently?

I just tried one alternative way, which is to use ADS to identify all refereed papers with “UKIDSS” in the title, assuming – possibly incorrectly – that all papers written by the UKIDSS consortium would have UKIDSS in the title. The number returned by this search was 38.

Now I’m not saying that this is more reasonable than the Warren measure. It’s just different, that’s all.  According to my criterion however UKIDSS measures 38 rather than 226. It sounds less impressive (if only because 38 is a smaller number than 226),  but what does it mean about UKIDSS productivity in absolute terms?

Not very much, I think is the answer.

Yet another way you might try to judge UKIDSS using bibliometric means is to look at its citation impact. After all, any fool can churn out dozens of papers that no-one ever reads. I know that for a fact. I am that fool.

But citation data also provide another way of doing what Steve Warren was trying to measure. Presumably the authors of any paper that uses UKIDSS data in any significant way would cite the main UKIDSS survey paper led by Andy Lawrence (Lawrence et al. 2007). According to ADS, the number of times this has been cited since publication is 359. That’s higher than the Warren measure (226), and much higher than the UKIDSS-in-the-title measure (38).

So there we are, three different measures, all in my opinion perfectly reasonable measures of, er,  something or other, but each giving a very different numerical value. I am not saying any  is misleading or that any is necessarily better than the others. My point is simply that it’s not easy to assign a numerical value to something that’s intrinsically difficult to define.

Unfortunately, it’s a point few people in government seem to be prepared to acknowledge.

Andy Lawrence is 57.


Share/Bookmark

Advertisements

14 Responses to “What Counts as Productivity?”

  1. Anton Garrett Says:

    It’s understandable that governments like numbers, because (real!) numbers are an easy way of ranking (pure math term: ordering) things, and government’s job is to see what projects would and would not survive at various levels of finance and then make the decision. Assigning numbers from any given algorithm is easy; the hard part is choosing a fair algorithm. It is hard because the real meaning of the numbers generated by any algorithm is hard to settle.

    What would happen if the algorithm “number of papers that could not have been written in the absence of this experiment” were adopted?

    • telescoper Says:

      How about the inverse criterion: How many papers resulting from this experiment really should never have been written?

    • “What would happen if the algorithm “number of papers that could not have been written in the absence of this experiment” were adopted?”

      isn’t this basically what steve warren did – and i agree i think its the most reasonable estimate of the “impact” of the survey as a whole

      (declaration of interest [something people in front of select committees should remember to do]: i am a member of UKIDSS)

    • Rob Ivison Says:

      they did occasionally state “i suppose that question was directed to me”, but it’s not quite the same thing. it was notable that the only mild squabble within the panel arose when one of their number stated a vested interest, only to be shouted down by those that hadn’t. [vested interest: he was talking about ROE…]

    • telescoper Says:

      Speaking for myself, I never wear a vest.

  2. What about the number of Grammys won by consortium members?

    http://www.stevewarrenthevoice.com/

  3. Just a quick note for clarification, Peter, since it isn’t clear in your excellent post, but a UKIDSS paper (using Steve’s definition) doesn’t necessarily mean it’s a UKIRT paper, we use a slightly different definition (http://www.jach.hawaii.edu/UKIRT/management/policies/publication_inclusion.html). This doesn’t change the point you’re making in any way of course, but I point this out just in case people don’t realise this.

  4. Bryn Jones Says:

    Productivity counted for nothing when it came to the issue of continuing British participation in the Anglo-Australian Observatory.

  5. Alastair Says:

    Peter,

    [Yes, I am a UKIDSS Survey insider but….]

    Can I just take issue with one sentence in your post?

    “…. and many written by people who aren’t members of the UKIDSS consortium.”

    The UKIDSS survey was designed to be a resource for the whole astronomical community. Yes, the UKIDSS consortium should be writing many papers (and we are!). However, if a survey is worthwhile and justifies the money UK taxpayers have put into it, then it is the number of papers published by others that you should be looking at.

    SDSS, 2MASS and WMAP are projects that have had an impact well beyond the papers written by the people that built the telescope and made the observations. I don’t claim that UKIDSS ranks with them (yet) but just ask publications that use UKIDSS data to be judged irrespective of who wrote them.

    Alastair

  6. telescoper Says:

    Of course that’s true, surveys produce catalogues and data products which are used by the wider community. All I’m saying is that one might argue that’s better measured by citations, i.e. it’s a reasonable argument that the “productivity” of UKIDSS is the number of papers produced directly by UKIDSS itself, whereas its “impact” is how much its results are used by others to do science. Moreover, as I said in the post, the number you get counting citations to the original UKIDSS paper is even higher than Steve Warren’s count, so UKIDSS does even better by that criterion!

    • Alastair Says:

      Can you clarify what you mean by “produced directly by UKIDSS itself”? Is it the UKIDSS data or the UKIDSS Consortium that you are talking about?

      Whichever it is, none of it would be possible with out the dedication and professionalism of the JAC staff and the grudging support that UKIRT gets from the STFC currently. I find it odd that when the Director of the JAC points out that the basic metric that every telescope on and off the planet uses hits a record we end up quibbling about the academic equivalent of the offside rule!

    • telescoper Says:

      Read the piece again and you’ll find the answer. I think you protest too much.

  7. […] 6. 一个英国天文学家自己Blog上关于UKIRT望远镜和科学产出的一般讨论 […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: