The ‘Danish Paper’ and How Science Works

I’m off work today but couldn’t resist posting a very quick update on the controversial claims of inconsistencies in the recent detection of gravitational waves by LIGO.

If you’re following the story you will know that it started with a paper on the arXiv by Cresswell et al.,  a group mainly based in Denmark, which is why the paper is now frequently referred to as ‘The Danish Paper’ although its authors actually come from all round the world.

Well the same group have now  written a rejoinder to the LIGO critique of their analysis. They’re clearly sticking to their guns, at least on their claim that the residuals left after removing the gravitational wave events from the two time series are correlated, which they should not be if they are simply noise.

Hopefully the public airing this controversy had received will lead to other independent groups downloading and analysing the data, which is all in the public domain, and we’ll eventually arrive at the truth.

Contrary to the opinion of one of my Cardiff colleagues I think this is how science works and, importantly, how it should be seen to work. Science  is a process of investigation, and it doesn’t come to an end when  results have been published in refereed journals. 

 The more the public see how science really works – warts and all – the better they will understand its strengths as well as its limitations.

Whatever the eventual outcome of this discussion I think we will find that the ‘Danish Paper’ has helped advance our understanding, and for that the authors deserve a great deal of credit.

 

Advertisements

21 Responses to “The ‘Danish Paper’ and How Science Works”

  1. I don’t think they’re completely sticking to their guns, as their rejoinder makes no mention of their previous claim about the Fourier phases, which was shown to be wrong.

    They do however stick to their guns on the question of whether correlations exist between the two cleaned (template-subtracted) signals. But the LIGO response by Ian Harry argued that whether or not such correlations exist doesn’t affect the detection claim.

    • telescoper Says:

      Yes, I’ve edited to clarify this.

      It may indeed turn out not to affect the detection claims but it is nevertheless a property of the data that needs to be understood.

    • Anton Garrett Says:

      It did strike me that the Danish paper was questioning only the accuracy of the time delay between the observations at the two detectors, and that the excellent match of the signal profile to one of the theoretical templates, together with the fact that both detectors saw it, was decisive evidence that it was gravitational radiation. All that the Danish paper therefore throws into renewed question is what part of the sky it came from (which anyway can’t be answered exactly with only two detectors even if the time delay is known with certainty).

      • telescoper Says:

        The simplest explanation for the observed correlations is just that the model for the signal is not perfect which means that when it is subtracted there is a residual present in both time series.

        This is perfectly possible – indeed, likely – because the model templates are derived from a grid constructed using simulations using different model parameters.

        I’m not sure however whether and how this information is used in determining the posterior distribution of model parameters.

      • Anton Garrett Says:

        That sounds like a crap way to do the data analysis! Subtract (ie, add the negative of) the wrong amplitude of the model and you will have superlative correlations! You have to estimate the amplitude somehow (and other parameters, of course, such as the masses of the merging black holes). Has to be Bayesian…

      • telescoper Says:

        You can use the LIGO data to get a posterior probability defined over the parameters of the models in a well-defined way, but you will inevitably find that the `best fit’ model leaves residuals when it is subtracted.

        A Bayesian approach of this sort would give a probability but would assume that the set of templates is exhaustive. I guess looking at the residuals is a kind of sanity check to see if there is any evidence that even the best-fitting template(s) can’t fit the data.

      • Anton Garrett Says:

        It’s an Ockham analysis. Suppose temporarily that there is only one detector. You throw into the ring (1) zero signal vs (2) black hole merger with parameters unknown, and calculate the probabilities of the two hypotheses, giving a prior to and marginalising over the parameters. Supposing that (2) is the clear winner, you then do parameter estimation.

        That should settle whether gravitational waves really have been detected. That the second detector saw something of similar profile at almost exactly the same time obviously firms it a good deal, and you can make that formal too with more work.

      • telescoper Says:

        I don’t think there’s any question that there’s been a detection. The question is whether there’s an effect on the parameter estimates..

      • Anton Garrett Says:

        I agree. But the Danish Abstract ended, “A clear distinction between signal and noise therefore remains to be established in order to determine the contribution of gravitational waves to the detected signals.” And the paper itself concluded, “The purpose in having two independent detectors is precisely to ensure that, after sufficient cleaning, the only genuine correlations between them will be due to gravitational wave effects. The results presented here suggest this level of cleaning has not yet been obtained and that the detection of the GW events needs to be re-evaluated with more careful consideration of noise properties.” If nobody is suggesting that gravitational waves have not been detected then these statements written as punchlines are, frankly, misleading.

      • Michel C. Says:

        The problem here is you cannot “throw into the ring zero signal”.

        I think that’s why you need a third detector to characterize the correlated noise between the other two detectors. Unless you can identify all possible sources of noise and distinguish it from a GW signal to establish a clear signal to noise ratio.

        I still think Ligo measured GWs though.

      • Shantanu Says:

        Peter: What you say is possible. However this is in conflict with the claim in the tests of GR paper arxiv:1602.03841 in the section titled “Residuals after subtracting the most-probable waveform model. ” It says “Our analysis reveals that the GW150914 residual favors the instrumental noise hypothesis over the presence of a coherent signal as well as the presence of glitches in either detectors;” It maybe that in this paper a more sensitive analysis has been done and traces of residual signal has been found.

  2. “Hopefully the public airing this controversy had received will lead to other independent groups downloading and analysing the data, which is all in the public domain, and we’ll eventually arrive at the truth.”

    Actually, this should be done even if there were no controversy. And journals should be obliged to publish confirmations of previously published results (perhaps with a limit of 2 or 3 such confirmations). I trust that, in the interest of good science, the Open Journal of Astrophysics will explicitly solicit such confirmation papers.

    Of course, as is the case with papers pointing out errors in other papers, such papers will not be highly cited. That doesn’t mean that they shouldn’t be written or published, though.

  3. Shantanu Says:

    One thing I would like really to see is if there are similar peaks in cross-correlation (after the 7 ms offset) in data not containing the signal. AFAIk, the Danes haven’t shown this in their paper.

  4. Shantanu Says:

    Also Harry hasn’t addressed the issue of correlations in the calibration lines between the two detectors after shifting the data by 7 milliseconds. Of course this could also be due to some trivial mistake or an artifact of how they have band-pass filtered the data/

    • In the LIGO response, Harry said

      “Nevertheless, if such a correlation were present it would suggest that we have not perfectly subtracted the real signal from the data, which would not invalidate any detection claim. There could be any number of reasons for this, for example the fact that our best-fit waveform will not exactly match what is in the data as we cannot measure all parameters with infinite precision. There might also be some deviations because the waveform models we used, while very good, are only approximations to the real signal (LIGO put out a paper quantifying this possibility). Such deviations might also be indicative of a subtle deviation from general relativity. These are of course things that LIGO is very interested in pursuing, and we have published a paper exploring potential deviations from general relativity (finding no evidence for that), which includes looking for a residual signal in the data after subtraction of the waveform (and again finding no evidence for that).”

      This seems to directly answer your point.

    • Oops, sorry – I see you are talking about correlation of the calibration lines, so the following quote is more relevant: “The claimed correlations between the two detectors due to resonance and calibration lines in the data would be present also in the time-shifted analyses—The calibration lines are repetitive lines, and so if correlated in the non-time shift analyses, they will also be correlated in the time-shift analyses as well.”

      • Shantanu Says:

        Yes, but again not obvious to me why the correlation is maximized at 6.9 milliseconds. Maybe there is some residual signal present when they band-passed the data around the calibration lines?

  5. Toffeenose Says:

    Absolutely, this is science working as it should!

  6. […] from LIGO and there’s been a lot of serious – but good-natured – discussion of `The Danish Paper‘ that came out some time ago and which questioned some aspects of the data analysis of the […]

  7. […] Bohr Institute between some members of the LIGO Scientific Collaboration and the authors of the `Danish Paper‘. As with the other one I attended last week it was both interesting and informative. […]

  8. […] Niels Bohr Institute and of the LIGO scientific collaboration concerning matters arising from the `Danish Paper‘. The most prominent among these appears to be the LIGO team and the Danish team have agreed […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: