Measuring the lack of impact of journal papers

I’ve been involved in a depressing discussion on the Astronomers facebook page, part of which was about the widespread use of Journal Impact factors by appointments panels, grant agencies, promotion committees, and so on. It is argued (by some) that younger researchers should be discouraged from publishing in, e.g., the Open Journal of Astrophysics, because it doesn’t have an impact factor and they would therefore be jeopardising their research career. In fact it takes two years for new journal to acquire an impact factor so if you take this advice seriously nobody should ever publish in any new journal.

For the record, I will state that no promotion committee, grant panel or appointment process I’ve ever been involved in has even mentioned impact factors. However, it appears that some do, despite the fact that they are demonstrably worse than useless at measuring the quality of publications. You can find comprehensive debunking of impact factors and exposure of their flaws all over the internet if you care to look: a good place to start is Stephen Curry’s article here.  I’d make an additional point here, which is that the impact factor uses citation information for the journal as a whole as a sort of proxy measure of the research quality of papers publish in it. But why on Earth should one do this when citation information for each paper is freely available? Why use a proxy when it’s trivial to measure the real thing?

The basic statistical flaw behind impact factors is that they are based on the arithmetic mean number of citations per paper. Since the distribution of citations in all journals is very skewed, this number is dragged upwards by a few papers with extremely large numbers of citations. In fact, most papers published have many few citations than the impact factor of a journal. It’s all very misleading, especially when used as a marketing tool by cynical academic publishers.

Thinking about this on the bus on my way into work this morning I decided to suggest a couple of bibliometric indices that should help put impact factors into context. I urge relevant people to calculate these for their favourite journals:

  • The Dead Paper Fraction (DPF). This is defined to be the fraction of papers published in the journal that receive no citations at all in the census period.  For journals with an impact factor of a few, this is probably a majority of the papers published.
  • The Unreliability of Impact Factor Factor (UIFF). This is defined to be the fraction of papers with fewer citations than the Impact Factor. For many journals this is most of their papers, and the larger this fraction is the more unreliable their Impact Factor is.

Another usefel measure for individual papers is

  • The Corrected Impact Factor. If a paper with a number N of actual citations is published in a journal with impact factor I then the corrected impact factor is C=N-I. For a deeply uninteresting paper published in a flashily hyped journal this will be large and negative, and should be viewed accordingly by relevant panels.

Other suggestions for citation metrics less stupid than the impact factor are welcome through the comments box…

 

19 Responses to “Measuring the lack of impact of journal papers”

  1. lordbubonicus Says:

    I read that discussion about impact factors with interest. In fact I read the whole thread with some interest, particularly the long explanations put up by the AAS representative.

    Do you think that the use of impact factors or not is a USA vs UK division? I’ve noticed previously that there’s a heavy American bias on the Astronomers Facebook group, and that a lot of the discussion on subjects like this is coloured by what’s commonplace in the USA.

  2. Ideally, people would not accept jobs where such bullshit criteria are used. Perhaps not something one could expect with temporary jobs, but people moving from one permanent job to another could decline the offer AND SHOUT ABOUT THE DECLINATION if impact factors are used at the potentially new institute.

    People who have more than one offer should take the offer without impact factor, and inform the impact-factor bullshitters and the public that they have done so.

  3. ” It is argue (by some) that younger researchers should be discouraged from publishing in, e.g., the Open Journal of Astrophysics, because it doesn’t have an impact factor and they would therefore be jeopardising their research career.”

    There are no papers to view.

    When can we expect to see the first OJA paper?

  4. […] “I’ve been involved in a depressing discussion on the Astronomers Facebook page, part of which was about the widespread use of Journal Impact factors by appointments panels, grant agencies, promotion committees, and so on …” (more) […]

  5. Sesh Nadathur Says:

    We had a discussion about open access and the Open Journal last week here at Portsmouth. A few points that came out of that discussion might be of interest:

    – there were a few mildly sceptical voices, mostly from permanent staff, worried about the potential impact on junior researchers’ careers from publishing in an “unknown” journal,
    – sceptics were primarily those who had been educated in the US,
    – a senior academic who was broadly supportive of the OJ suggested that if the editorial board were to succeed in negotiating the publication of all papers from some big collaboration (in the way that all Planck papers are submitted to A&A), this would go a long way to helping the journal “catch on”,
    – some people were also unhappy with the idea of putting a paper on the arXiv for all to see before it had undergone peer review – I think this was about professional embarrassment in case it required revision,
    – again, this opinion was more common among those who had been educated in the US (as well as among astronomers, as opposed to cosmologists).

  6. “– there were a few mildly sceptical voices, mostly from permanent staff, worried about the potential impact on junior researchers’ careers from publishing in an “unknown” journal,”

    The question then should be: Would you (i.e. the permanent staff) view papers less favourably if they are published in an “unknown” journal?

    I think the issue is not so much whether it is unknown as whether it represents quality. If the refereeing standards are high, then it should become known as a good journal relatively quickly.

    There are people who apply to jobs outside of the field in which they have worked, so the potential employer might not be familiar with the applicants’ papers, nor would he have the expertise to become familiar even if he had the time to read all papers of all applicants. So, the quality of the journal does matter. The best way to get this going is to have some people who have nothing to lose (i.e. permanent staff) publish some good papers in the journal.

    “– sceptics were primarily those who had been educated in the US,”

    More impact-factor bullshit there, perhaps.

    “– a senior academic who was broadly supportive of the OJ suggested that if the editorial board were to succeed in negotiating the publication of all papers from some big collaboration (in the way that all Planck papers are submitted to A&A), this would go a long way to helping the journal “catch on”,

    This is a good idea.

    “– some people were also unhappy with the idea of putting a paper on the arXiv for all to see before it had undergone peer review – I think this was about professional embarrassment in case it required revision,”

    That can be one reason (though minor revision shouldn’t cause embarrassment; major rewrite or rejection would be the problem). I have discussed the OJA with many people, both online and offline, both privately and publicly, and this is the most common objection I hear to the OJA. Again: (1) some people don’t want to waste other people’s time by posting what will probably turn out to be a preliminary version (yes, some merely submitted papers might get helpful comments, but most won’t); (2) there is the rare but potentially very troublesome issue of theft (i.e. preliminary version has good ideas and represents a lot of work, perhaps using data which in the meantime have become public, and some hotshot steals the idea and manages—because he doesn’t have a life otherwise—to get it accepted by another journal first: rare but it does happen); ( 3) some institutes have a policy against non-accepted stuff on the arXiv (this was the case at Jodrell Bank when I was there, when exceptions needed the approval of the Director; I don’t know if it still the case—ask
    Sarah (but even if there is no longer such a policy at Jodrell Bank, there might be elsewhere)); (4) there are people who cannot (yet) post to arXiv.

    The first three could be easily taken care of by an alternative submission mechanism. Instead of an arXiv URL, just allow any URL for submission. The manuscript could be retrieved and stored at the OJA website. (With disk space 10p or whatever per GB now, this is a negligible cost.) As for (4), it would be nice to negotiate an agreement with arXiv than acceptance by the OJA (a) constitutes automatic endorsement to the arXiv (if you don’t know about the necessity of endorsement, you are old; us old fogies who had put stuff on arXiv before the endorsement system was set up were grandfathered in) and (b) constitutes automatic acceptance for the paper in question. Quite frankly, if you can’t negotiate this with arXiv, it doesn’t look good for the journal, and if you don’t have an alternative submission mechanism, you exclude a fair number of people (some of them senior members of the astrophysical community) who would otherwise publish via the OJA. (It might be difficult to negotiate this with arXiv for political reasons. In that case, just link to your own copy of the paper rather than to arXiv.)

    “– again, this opinion was more common among those who had been educated in the US (as well as among astronomers, as opposed to cosmologists).”

    Astronomers as opposed to cosmologists?!?!?! Reminds me of the most scathing criticism David Schramm could think of when criticizing a student: “You are thinking like an astronomer and not like a physicist!” (Check out Dennis Overbye’s Lonely Hearts of the Cosmos for the full story.) Seriously, cosmology is a branch of astronomy, but astronomy is also a branch of cosmology. A similar relationship exists between cosmology and GR. (Among astronomers, I say I am a cosmologist, because otherwise someone might think I have experience in spectroscopy (That’s obviously magnesium II!); among cosmologists, I say I am an astronomer, because otherwise someone might think that I work on the early universe, inflation, and so on; among non-scientists, I say that I am an astrophysicists, since “astronomer” and “astrologer” are often confused (not to mention “cosmologist” and “cosmetologist”); when I wanted money from a bank, I used to say I was a physicist (now I say that I have a proper job and earn a lot more than they do).

  7. Reblogged this on Disturbing the Universe and commented:
    Why journal impact factor is a meaningless metric…

  8. The most relevant parameter is probably your ‘dead paper fraction’ although just like any citation index, deadness needs time to mature. I have never seen JIFs used by grant panels, promotion panels, etc, but I have often seen it mentioned in supporting documentation to proof to people in other fields that a journal called ‘A&A’ is high quality science and in no way related to the Daily Mail or indicating alcohol abuse. There ARE a lot of bogus journals around and people in the field kn ow which ones are real but people outside don’t. For young people it is more important to publish in recognized journals; established scientists can publish anywhere and Hawkins could publish on wordpress and still get cited. Reputation sadly is important, and if you don’t yet have a personal reputation, you need to borrow it from the journal. OJA should initially aim for papers from well known scientists.

    And I am definitely an astronomer – not a cosmologist. The word ‘cosmos’ seems to have become more distant since Sagan’s days.

    • “There ARE a lot of bogus journals around and people in the field kn ow which ones are real but people outside don’t.”

      Indeed. The impact factor might have some use demonstrating that a journal is at least well known and considered sensible. This is OK. What is wrong is rating someone higher because of a paper in a high-impact-factor journal. (If his paper is not one of the most popular papers in that journal, it might actually be cited less than average for his field.)

      What the bean counters might not realize is that if a paper is highly cited, it is probably worth citing, even if it didn’t appear in a high-impact-factor journal. In other words, one can easily publish in a shoddy journal, but such papers won’t be cited much, at least not by serious people. (OK, maybe again you need well known journals to determine who are the serious citers.)

      I am really surprised to what extent clueless people don’t even know what journals are reputable in their own field:

      https://scholarlyoa.com/

      Read it and weep.

  9. “Hawkins could publish on wordpress and still get cited”

    You probably mean Hawking. I’ll put it down to a typo and refrain from invoking the ghost of John Baez. 😀

  10. Distributions are more informative than indices:

    https://bernardrentier.wordpress.com/2015/12/31/denouncing-the-imposter-factor/

    “… using the impact factor of the journals where [one] publishes is like measuring someone’s qualities by the club where he/she is allowed to go dining. Stars are for restaurants, not for their customers…”

  11. It’s very strange that some astronomers seem to care so much about impact factors and prestige of journals. As far as I know, all of the major astronomy journals (ApJ, AJ, MNRAS, A&A) have very high acceptance rates (close to 90 %), so it’s not like getting a paper accepted necessarily means much.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: