The Quality of Physics

Just time for a quick post this lunchtime,  in between meetings and exercise classes. My eye was drawn this morning to an article about a lengthy report from the Institute of Physics that gives an international comparison of citation impact in physics and related fields.

According to the IOP website..

Although the UK is ranked seventh in a list of key competitor countries for the quantity of its physics research output – measured by the number of papers published – the UK is second only to Canada, and now higher than the US, when ranked on the average quality of the UK’s physics research output – measured by the average number of times research papers are cited around world.

The piece also goes on to note that the UK’s share of the total number of research papers written has decreased

For the UK, however, its proportionate decrease in output – from 7.1% of the world’s physics research in 2001 to 6.4% in 2010 – has been accompanied by a celebratory increase in overall, average quality – with the average number of citations of UK research papers rising from 1.24 in 2001 to 1.72 in 2010.

This, of course, assumes that citations measure “quality” but I’ve got no time to argue that point today. What I will do is put up a couple of interesting figures from the report.  This one shows that Space Science in the UK (including Astronomy and Astrophysics) holds a much bigger share of the total world output of papers than other disciplines (by a factor of about three):

While this one shows that the “citation impact” for Physics and Space Science roughly track each other…

..apart from the downturn right at the end of the window for space sciences, which, one imagines, might be a result of decisions taken by the management of the Science and Technology Facilities Council  over that period.

Our political leaders will be tempted to portray the steady increase of citation impact across fields as a sign of improved quality arising from the various research assessment exercises.  But I don’t think it’s as simple as that. It seems that many developing countries – especially China – are producing more and more scientific papers. This inevitably drives the UK’s share of world productivity down, because our capacity is not increasing. If anything it’s going down, in fact, owing to recent funding cuts. However, the more papers there are, the more reference lists there are, and the more citations there will be. The increase in citation rates may therefore just be a form of inflation.

Anyway, you can download the entire report here (PDF). I’m sure there will be other reactions to it so, as usual, please feel free to comment via the box below…

21 Responses to “The Quality of Physics”

  1. “The increase in citation rates may therefore just be a form of inflation.”

    Sorry, I’m not sure I understand your point.

    Suppose there are 10,000 papers published in the world. Each of them includes 10 random references. If 1,000 of those 10,000 papers are UK-based (10%), in the average 10,000 of those 10,000 x 10 = 100,000 will refer to british papers, that means 10%.

    Now, suppose there are 100,000 papers published in the world. Each of them has still 10 random reference. In total there will be 100,000 x 10 = 1,000,000 citations. If UK-based papers are still 1,000, in the average there will be 1,000 / 100,000 = 1% citations of British papers, that means each paper is cited in the average the same number of times, but the normalized citation impact (normalized respect to other countries’ papers) has decreased a factor 10.

    There could rather be a “time” bias because new papers cites past papers, and in the past the number of UK papers were more in percentage than today.

    Am I wrong?

    • telescoper Says:

      I think what’s going on is that more papers = more crap papers citing good ones, so the good paper gets more citations than it would have done.

      I think the point is that references are not random…

  2. I believe the result of the STFC decisions will be a change in the overall derivative over a number of years and we have yet to see that appear in the graph…

    • better alternatives?

      • sorry – this reply was addressed to Ian smail

      • telescoper Says:

        Yes, I think there are better alternatives to Ian Smail.

      • i’m happy with citations (you can guess why) – but you need to normalise out a lot of factors to make any sense of them.

        it is peter who doesn’t seem to like them…

      • telescoper Says:

        I’m not opposed to citations themselves – they clearly contain information and information is good…

        I am, however, opposed to using them without thinking very hard about doing so sensibly. And I don’t think they can really be a substitute for peer review, since some very high quality papers are very slow burners taking a very long time to have the impact their quality merits.

      • “…slow burners”

        i’ve never really understood this argument – except for papers whose “time has yet to come” (e.g. a prediction for a field where the observational constraints are currently lacking). but such papers must be relatively rare.

        surely if a paper discusses something which is relevant for a research field *today* and nobody cites it… then there is probably a good reason for that. a moderate fraction of the researchers in most fields are not complete idiots – so they should be able to identify (and cite) a good paper when they see it. if they can’t – then how can you rely on anyone else to identify it (e.g. a less-qualified RAE assessor?)

        [do you think i needed more qualifiers in that final sentence?]

      • telescoper Says:

        It’s probably rarer now than it was in the past – because of the pace with which ideas can be disseminated these days – but there are definitely still papers that are ahead of the time, containing brilliant radical ideas that aren’t taken seriously when first published.

        I’ve often wondered what would happen to a brilliant paper that definitively demolishes a fashionable bandwagon. Who would cite it?

      • i think that every band-waggon probably has sufficient credible detractors in the field that they’d happily line up behind a sound alternative.

        cosmology is perhaps more rigid in this regard – but most of the galaxy formation band-waggons in the last decade have either morphed into something more palatable or gone complete off the rails (to mix my metaphors).

        in fact i’m happy for a contrarian paper nowadays – it makes writing a telescope proposal much easier if there is an extreme model out there to be shot down.

      • So it seems the faster-than-light neutrino story has gone away – an error in the cabling to do with GPS system – but their paper as 230 citations. What does that tell you about citations as a measure of quality?

      • …that:

        a) theoretical physicists like odd results – it makes them feel that they are relevant if they can come up with a model which “predicts” such behaviour;

        b) experimental particle physicists need to learn to be able to refer to a “result”, without actually having to cite it.

        regarding b – its usually better to simply cite the first paper which casts doubt on the original result – rather than the “discovery” paper itself… if you don’t believe it.

  3. Monica Grady Says:

    I don’t think I’d want to place too much significance in the space sciences downturn for 1 year. May be if the trend continues for another 2 or 3 years, then we should start to worry.

  4. So, I keep asking: which are the alternatives?

    If someone is given the task to decide if a paper is above or below a shit treshold, this will only generate endless debates. We don’t want “rating” agencies to evaluate papers and this is like democracy, plenty of problems but still the best.

    • Agreed. It’s the best way to rate papers. But that’s basically irrelevant. The real question is how to rate people. With 100+ authors per paper how do you work out who’s done what?

      • I suggest to use a font proportional to what is the author contribution, from 9 to 14.


        Jokes apart, we could introduce a voluntary system where the authors can add a “contribution index” to their paper. Something from 1 to 5, and forget about the order, that maybe alphabetical. We could also introduce a system to “rate” citations based on another index.

        Looks silly but actually the system is still like if we were using papers book grabbed in a real library: no interaction. Of course this wouldn’t be the perfect system, but would be better.

        People will probably laugh to this, but I expect that in 10, max 20 years, science papers will become really electronics, not just copying their paper-based counterpart, but introducing interaction or some (difficult to foresee) functionalities.

        And one more question, maybe if you want to work on that:

        is it really required that science papers are so impersonal and boring most of the time? Is serious eq. to boring?

        Ok, forget about this…

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: