Clusters, Splines and Peer Review

Time for a grumpy early morning post while I drink my tea.

There’s an interesting post on the New Scientist blog site by that young chap Andrew Pontzen who works at Oxford University (in the Midlands). It’s on a topic that’s very pertinent to the ongoing debate about Open Access. One of the points the academic publishing lobby always makes is that Peer Review is essential to assure the quality of research. The publishers also often try to claim that they actually do Peer Review, which they don’t. That’s usual done, for free, by academics.

But the point Andrew makes is that we should also think about whether the form of Peer Review that journals undertake is any good anyway.  Currently we submit our paper to a journal, the editors of which select one (or perhaps two or three) referees to decide whether it merits publication. We then wait – often many months – for a report and a decision by the Editorial Board.

But there’s also a free online repository called the arXiv which all astrophysics papers eventually appear on. Some researchers like to wait for the paper to be refereed and accepted before putting it on the arXiv, while others, myself included, just put it on the arXiv straight away when we submit it to the journal. In most cases one gets prompter and more helpful comments by email from people who read the paper on arXiv than from the referee(s).

Andrew questions why we trust the reviewing of a paper to one or two individuals chosen by the journal when the whole community could do the job quicker and better. I made essentially the same point in a post a few years ago:

I’m not saying the arXiv is perfect but, unlike traditional journals, it is, in my field anyway, indispensable. A little more investment, adding a comment facilities or a rating system along the lines of, e.g. reddit, and it would be better than anything we get academic publishers at a fraction of the cost. Reddit, in case you don’t know the site, allows readers to vote articles up or down according to their reaction to it. Restrict voting to registered users only and you have the core of a peer review system that involves en entire community rather than relying on the whim of one or two referees. Citations provide another measure in the longer term. Nowadays astronomical papers attract citations on the arXiv even before they appear in journals, but it still takes time for new research to incorporate older ideas.

In any case I don’t think the current system of Peer Review provides the Gold Standard that publishers claim it does. It’s probably a bit harsh to single out one example, but then I said I was feeling grumpy, so here’s something from a paper that we’ve been discussing recently in the cosmology group at Cardiff. The paper is by Gonzalez et al. and is called IDCS J1426.5+3508: Cosmological implications of a massive, strong lensing cluster at Z = 1.75. The abstract reads

The galaxy cluster IDCS J1426.5+3508 at z = 1.75 is the most massive galaxy cluster yet discovered at z > 1.4 and the first cluster at this epoch for which the Sunyaev-Zel’Dovich effect has been observed. In this paper we report on the discovery with HST imaging of a giant arc associated with this cluster. The curvature of the arc suggests that the lensing mass is nearly coincident with the brightest cluster galaxy, and the color is consistent with the arc being a star-forming galaxy. We compare the constraint on M200 based upon strong lensing with Sunyaev-Zel’Dovich results, finding that the two are consistent if the redshift of the arc is  z > 3. Finally, we explore the cosmological implications of this system, considering the likelihood of the existence of a strongly lensing galaxy cluster at this epoch in an LCDM universe. While the existence of the cluster itself can potentially be accomodated if one considers the entire volume covered at this redshift by all current high-redshift cluster surveys, the existence of this strongly lensed galaxy greatly exacerbates the long-standing giant arc problem. For standard LCDM structure formation and observed background field galaxy counts this lens system should not exist. Specifically, there should be no giant arcs in the entire sky as bright in F814W as the observed arc for clusters at  z \geq 1.75, and only \sim 0.3 as bright in F160W as the observed arc. If we relax the redshift constraint to consider all clusters at z \geq 1.5, the expected number of giant arcs rises to \sim 15 in F160W, but the number of giant arcs of this brightness in F814W remains zero. These arc statistic results are independent of the mass of IDCS J1426.5+3508. We consider possible explanations for this discrepancy.

Interesting stuff indeed. The paper has been accepted for publication by the Astrophysical Journal too.

Now look at the key result, Figure 3:

I’ll leave aside the fact that there aren’t any error bars on the points, and instead draw your attention to the phrase “The curves are spline interpolations between the data points”. For the red curve only two “data points” are shown; actually the points are from simulations, so aren’t strictly data, but that’s not the point. I would have expected an alert referee to ask for all the points needed to form the curve to be shown, and it takes more than two points to make a spline.  Without the other point(s) – hopefully there is at least one more! – the reader can’t reproduce the analysis, which is what the scientific method requires, especially when a paper makes such a strong claim as this.

I’m guessing that the third point is at zero (which is at – ∞ on the log scale shown in the graph), but surely that must have an error bar on it, deriving from the limited simulation size?

If this paper had been put on a system like the one I discussed above, I think this would have been raised…

About these ads

13 Responses to “Clusters, Splines and Peer Review”

  1. stringph Says:

    Hi Peter. No, ‘review by the community’ won’t work, because ‘the community’ will turn out to be the small subset of people with the time and energy and inclination and chutzpah to write stuff about other people’s papers in public.

    Voting isn’t review because it doesn’t have any content other than a single bit of information. It’s not even a one-dimensional criterion. And we already have citation as a proxy for voting.

    Proper review involves the potentially infinite number of dimensions in which one could change the paper towards being scientifically better. Again, only interfering jerks or professors would usually dare to write something off the cuff in public telling the world how someone else’s paper ought to be changed.

    The institution of anonymous review is absolutely necessary (for example, if a famous professor writes a paper that’s simply rubbish and no-one dares to say so in public…) however badly it may work in practice. The correct response to the current situation where it’s (often) not working very well is to propose some methods to incentivize better refereeing and more active editors – not abandon the whole thing, which would open the door to much worse abuses, i.e. domination of everything by the loudest voices.

    • telescoper Says:

      ..only interfering jerks or professors would usually dare to write something off the cuff in public telling the world how someone else’s paper ought to be changed

      As charming as ever, Dr Dent.

      But seriously, I couldn’t agree less with your comments. I’d argue that the problem with Peer Review is precisely that it is anonymous. That allows some to get away with lazy refereeing but others, more dangerously, to deliberately sabotage or delay the work of other people. And don’t think that doesn’t happen.

      And the point about Reddit is not the voting. It’s the fact that people can comment on posts, which I think people would do and I think that’s the future for scientific discussion and debate.

      The only way such a system could be abused would be if anonymity were allowed, just as is the case with your standard internet trolls…

      • stringph Says:

        Again, there is the assumption that the bad parts of anonymous review are unavoidable and irremediable and the good parts (i.e. detailed attention to a paper which may require it) are not worth preserving. That is a counsel of despair.

        The missing piece here is editors. They are publicly responsible figures and should arbitrate rather than deferring automatically to referees; referees should understand that they are not all-powerful. Editors should be ultimately responsible if refereeing is bad or corrupt – after all they choose the referees and take the final decisions: they should be taking a more active role in many cases.

        Don’t mistake the failures of particular journals and the reluctance of editors to do their jobs (understandable if they have to deal with order(10) papers a day…) for a failure of the whole institution. All it would take, in my opinion, is a slight change in behaviour by journals and their editorial boards to make a big difference to the general quality of refereeing.

        As for internet-based comment-leaving, there already is plenty of scientific discussion and debate on the web, but it’s very rarely on the same level of detail and intellect as one finds here…

        Incidentally, my singling out of tenured professors is just because they have an enviable position of being immune to the consequences of rubbishing other people’s work on the internet (or even in published Comments). Those of us on short-term contracts might be considerably less willing to stick necks above parapets.

        I should also disclose an interest as a Physical Review ‘Outstanding Referee’ for the last year… an award that rather shocked me, as I always aimed to be exactly average in my own refereeing.

  2. I agree with you sentiments, but I have my misgivings about your proposed reddit-like system. As I see it, there are two main problems. Firstly, clearly only some papers will get reviewed this way. Many perfectly correct papers will be ignored simply because they are not on currently fashionable topics. Although writing papers that are of current interest to your fellow scientists is of some importance, it shouldn’t be the sole criterion for judging merit!

    Secondly, simple votes up or down are not particularly useful to a reader: if I am to trust the judgement of the people voting on the quality of a paper I’d really need to read a justification from them of why they voted in a particular way. This would amount to a mini-review of the paper and its strong (or weak) points. Such reviews would need to be provided anonymously — by the registered users of the site — in order ensure that people felt secure enough to express honest opinions. Anonymity of the referee is one of the cornerstones of the review process. Would I provide a negative non-anonymous review of a paper by someone to whom I might in the future be applying for a job? Probably not.

    Non-anonymous review systems would I think inevitably dwindle into relative irrelevance like Cosmo Coffee has done, where no-one ventures any opinions at all any more. But completely anonymous systems would develop into flame wars in the classic tradition of the internet, as you said.

    There is of course the point that stringph mentioned as well, that some loud voices will end up dominating (even more than they do currently, that is).

    An idea I wrote about here, which is less drastic, is for journals to make available online the correspondence between the referee and the authors for every published paper. This would I think help to weed out sloppy negative reviews, though admittedly might not help with the problem of sloppy positive reviews you’re concerned with here.

  3. Hi Professor Coles,

    It seems that part of your complaint is related to infrastructure. Specifically, publishers claim to provide infrastructure but they either don’t at all or what they provide isn’t robust as they claim.

    Some colleagues from the University of Chicago and I have launched Scholastica (http://www.scholasticahq.com), a website that provides the infrastructure needed to publish academic journals.

    Since you mention the slowness of traditional review, we believe giving scholars the tools they want can speed the process up tremendously. Also, in the tool, new articles can be published whenever one likes, thus one doesn’t have to wait to publish things in quarterly issues. You can find an example of what a journal looks like on the system here: http://scholasticahq.com/the-scholastica-example-journal

    Finally, arXiv is integrated into Scholastica (http://vimeo.com/36720688) to allow scholars an easy way to get their paper from the arXiv submitted to a journal hosted on Scholastica.

    I hope you don’t take offense to me posting this on your blog, I do think it’s relevant to your post.

  4. I’m with you on the desirability of replacing the current system of peer review with one that is (a) more open, (b) more distributed (many reviewers rather than one), and (c) untethered from the publishing industry. I think the practical difficulties are greater than you seem to suggest. It’s not obvious to me how a system of the sort you’re describing would lead to clean, reliable results of the sort that university administrators would want to rely on in making, say, promotion and tenure decisions. Administrators won’t / can’t read comment threads on everyone’s papers, and a simple Reddit-like voting system seems prone to highly variable outcomes depending on which people happen to bother to read and vote on a given paper. Sesh’s comment that many papers will not be reviewed at all seems right to me. At the very least, lots of papers will be reviewed by very few people, leading to large random (or worse, systematic) variations.

    I’m not saying that a good system along these lines can’t be designed, nor do I deny the large imperfections in the current system. I suspect that it would be possible to design a better system built around the three principles I listed above, although I don’t know exactly how it would work.

  5. John Peacock Says:

    All very interesting, but none of the alternatives will deal with the fact that there are just too many papers, mostly with results that are relatively sound technically, but lacking a critical mass of really striking content. When there is already far more stuff than you can possibly study in any detail, who cares whether one particular form of peer review might be more effective than another in raising the standards of a few papers by some modest amount?

    • Dave Carter Says:

      Maybe there should be a quota of 4 per staff member per 6 years, that would make a REF co-ordinator’s job easier.

  6. isn’t the more important issue to ensure that poor papers aren’t cited (ie authors need to show they’ve read the papers they cite in their publications)?

    then the next step would be to include in researcher impact analyses a “negative H-index” based on the length of their tail of poorly cited papers (of sufficient age) – to try to dissuade people from publishing every thought/observation they have – no matter the quality.

  7. voxcharta.org is basically the reddit-like system you describe, along with some additional machinery to aid in creating an agenda for an astro-ph coffee discussion. Right now commenting is comparatively rare, though.

  8. While there are several good ideas to improve the current system (multiple referees, open access) surely the main point of the referee system is that most authors assume they will get a reasonably detailed report from a competent scientist. This motivates the author to get the paper to a certain minimum standard and to not leave any obvious weaknesses. Sometimes a referee may not scrutinise a given paper quite to the standard of a given journal, but the author has at least already attempted to do this in order to ease the papers passage to publication. Any alternate system would have to address this.

    Regarding John’s issue of too many papers that’s entirely our own (astronomers’) fault by building so many telescopes and collecting so much data!

  9. First, feel free to discuss open access and refereeing, but don’t mix the two. They are two separate issues. Yes, they both have to do with scientific papers, but that’s about it.

    Replacing refereeing by some sort of voting system, with the added disadvantage that only a subset (those with nothing better to do?) will vote (or write a virus which will infest computers and set up a botnet to skew the votes in a certain way) seems like a step back. If a journal doesn’t do good refereeing, don’t submit papers to it. You mentioned yourself that citations are the real indicator of quality, so why is any sort of refereeing needed? Yes, people can and do send in comments based on stuff they see on arXiv, and it is up to the author to take them into account. But reddit (have you actually read some of the drivel there) seems like a step back. Might as well base your taste in music on the singles charts.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 3,454 other followers

%d bloggers like this: