Archive for LIGO

On the Time Lags of the LIGO signals

Posted in Bad Statistics, The Universe and Stuff with tags , , , on August 10, 2017 by telescoper

It seems that a lot of rumours are flying around on social media and elsewhere about the discussions that have been going on here in Copenhagen between members of the Niels Bohr Institute and of the LIGO scientific collaboration concerning matters arising from the `Danish Paper‘.  The most prominent among these appears to be the LIGO team and the Danish team have agreed on everything and that the Danish authors have conceded that they were mistaken in their claims. I have even been told that my recent blog posts gave the impression that this was the case. I’m not sure how, as all I’ve said is that the discussions reached agreement on some matters. I did not say what matters or whose position had changed.

I feel, therefore, that some clarification is necessary. Since I am a member of neither party to this controversy I have to tread carefully, and there are some things which I feel I should not discuss at all. I was invited to participate in the discussions as a neutral observer as a courtesy and I certainly don’t want to betray any confidences. On one thing, however, I can be perfectly clear. The Danish team (Cresswell et al.) have not retracted their claims and they reject the suggestion that their paper was wrong.

To reinforce this, I draw your attention to the fact that a revised version of `The Danish Paper’ has now been accepted for publication (in the Journal of Cosmology and Astroparticle Physics) and that this paper is now available on the arXiv. The referees raised a large number of queries, and in response to them all the revised version is almost double the length of the original.

Here is the arXiv entry page:

The main body of the paper has not been significantly modified and their main result – of an unexplained 7ms correlation in the background signal (referred to in the abstract as `noise’) – has not “gone away”. If you want to understand more, read the paper!

I’m sure there will be much more discussion of this and I will comment as appropriate when appropriate. In the meantime this remains very much a live issue.

P.S. In the interest of full disclosure I should mention that I did read over part of the revised version of the Danish paper and made some suggestions with regard to style and flow. I therefore have a mention in the acknowledgments of the final version. I was warned that I might expect some trouble for agreeing to be associated with the paper in this way but, as  Sam Spade says in The Maltese Falcon `I don’t mind a reasonable amount of trouble’…

LIGO and Open Science

Posted in Open Access, Science Politics, The Universe and Stuff with tags , , , , on August 8, 2017 by telescoper

I’ve just come from another meeting here at the Niels Bohr Institute between some members of the LIGO Scientific Collaboration and the authors of the `Danish Paper‘. As with the other one I attended last week it was both interesting and informative. I’m not going to divulge any of the details of the discussion, but I anticipate further developments that will put some of them into the public domain fairly soon and will comment on them as and when that happens.

I think an important aspect of the way science works is that when a given individual or group publishes a result, it should be possible for others to reproduce it (or not as the case may be). In normal-sized laboratory physics it suffices to explain the experimental set-up in the published paper in sufficient detail for another individual or group to build an equivalent replica experiment if they want to check the results. In `Big Science’, e.g. with LIGO or the Large Hadron Collider, it is not practically possible for other groups to build their own copy, so the best that can be done is to release the data coming from the experiment. A basic problem with reproducibility obviously arises when this does not happen.

In astrophysics and cosmology, results in scientific papers are often based on very complicated analyses of large data sets. This is also the case for gravitational wave experiments. Fortunately in astrophysics these days researchers are generally pretty good at sharing their data, but there are a few exceptions in that field. Particle physicists, by contrast, generally treat all their data as proprietary.

Even allowing open access to data doesn’t always solve the reproducibility problem. Often extensive numerical codes are needed to process the measurements and extract meaningful output. Without access to these pipeline codes it is impossible for a third party to check the path from input to output without writing their own version, assuming that there is sufficient information to do that in the first place. That researchers should publish their software as well as their results is quite a controversial suggestion, but I think it’s the best practice for science. In any case there are often intermediate stages between `raw’ data and scientific results, as well as ancillary data products of various kinds. I think these should all be made public. Doing that could well entail a great deal of effort, but I think in the long run that it is worth it.

I’m not saying that scientific collaborations should not have a proprietary period, just that this period should end when a result is announced, and that any such announcement should be accompanied by a release of the data products and software needed to subject the analysis to independent verification.

Now, if you are interested in trying to reproduce the analysis of data from the first detection of gravitational waves by LIGO, you can go here, where you can not only download the data but also find a helpful tutorial on how to analyse it.

This seems at first sight to be fully in the spirit of open science, but if you visit that page you will find this disclaimer:

 

In other words, one can’t check the LIGO data analysis because not all the data and tools necessary to do that are not publicly available.  I know for a fact that this is the case because of the meetings going on here at NBI!

Given that the detection of gravitational waves is one of the most important breakthroughs ever made in physics, I think this is a matter of considerable regret. I also find it difficult to understand the reasoning that led the LIGO consortium to think it was a good plan only to go part of the way towards open science, by releasing only part of the information needed to reproduce the processing of the LIGO signals and their subsequent statistical analysis. There may be good reasons that I know nothing about, but at the moment it seems to me to me to represent a wasted opportunity.

I know I’m an extremist when it comes to open science, and there are probably many who disagree with me, so I thought I’d do a mini-poll on this issue:

Any other comments welcome through the box below!

Copenhagen Again

Posted in Biographical, The Universe and Stuff with tags , , on August 1, 2017 by telescoper

As you might have inferred from my earlier post, I’m back again in the wonderful city of Copenhagen, as a guest of the Niels Bohr Institute. I’ve been here almost every year since my first visit here way back in the 1980s. I didn’t come here last summer, as I was too busy finishing off my duties at Sussex and relocating back to Cardiff so it’s nice to be back again now. I’m staying in one of the `9 small homes‘ that comprise a hotel near the NBI. I’ve stayed here before though not in my current small home, which is actually a self-contained apartment on the ground floor with its own front door. It’s also got a small kitchen so I can cook for myself when I don’t feel like eating out (like tonight). Incidentally, `hjem’ (the Danish word for `home’) is pronounced exactly as `home’ is pronounced in Geordie (i.e. as `hyem’). I did some shopping earlier this evening and attempted to speak Danish when I paid for my groceries. As always, however, I got a reply in English.

I realised only this morning that it’s a year since I left my previous job. I haven’t done half the things I had hoped to do in the year after stepping down as Head of School, but that’s partly because it took quite a while to get over certain health problems and also because quite a few things have come up that I didn’t anticipate. From what I’m told the old place is doing just fine without me!

Coincidently (?), I have arrived here at the Niels Bohr Institute at precisely the time that there is a delegation here from LIGO and there’s been a lot of serious – but good-natured – discussion of `The Danish Paper‘ that came out some time ago and which questioned some aspects of the data analysis of the first detection of gravitational waves. I think there are still quite a few issues to be resolved between the two groups. Although they do seem to be converging on what’s going on, I don’t think this controversy will be fully concluded until more data are made public, as the currently available time series are not exactly those used in the actual LIGO analysis.

I think this discussion can only be of benefit to the science community in the long run, especially if it encourages LIGO to get more fully into the spirit of open science, by releasing more data for use of researchers outside the consortium.

Questioning LIGO

Posted in The Universe and Stuff with tags , on June 17, 2017 by telescoper

Well. Cat, meet pigeons..

A paper appeared on the arXiv this week with the following abstract:

To date, the LIGO collaboration has detected three gravitational wave (GW) events appearing in both its Hanford and Livingston detectors. In this article we reexamine the LIGO data with regard to correlations between the two detectors. With special focus on GW150914, we report correlations in the detector noise which, at the time of the event, happen to be maximized for the same time lag as that found for the event itself. Specifically, we analyze correlations in the calibration lines in the vicinity of 35 Hz as well as the residual noise in the data after subtraction of the best-fit theoretical templates. The residual noise for the two more recent events, GW151226 and GW170104, exhibits equivalent behavior with respect to each of their time lags. A clear distinction between signal and noise therefore remains to be established in order to determine the contribution of gravitational waves to the detected signals.

I’m going to tread carefully here because (a) I have a number of colleagues at Cardiff who are directly involved in the analysis of LIGO data; (b) one of the authors of the new paper (Panel Naselsky) is a longstanding collaborator of mine; and (c) the new paper has not yet been refereed.

In fact I’m planning to visit Copenhagen in July/August and will catch up with Pavel and the other authors then.

Whether or not the points raised in the new paper are correct – and I am firmly agnostic, having not done the analysis myself – I think it’s entirely reasonable of the authors to subject the LIGO data to independent analysis. That’s how science is supposed to work; the relevant data are in the public domain now. 

No doubt the LIGO consortium will respond officially in due course. Of course, if anyone would like to comment unofficially then they are free to do so through the box below.

Update: Here is a fairly detailed rebuttal post.

Simulation of the binary black-hole coalescence GW170104

Posted in The Universe and Stuff with tags , , , on June 2, 2017 by telescoper

Via the Cardiff University news website, I found this video of a computer simulation of the binary black-hole coalescence that gave rise to the gravitational wave event GW170104 whose detection was announced yesterday, so I thought I’d share it here.

Here’s the  blurb accompanying the video:

The video shows a numerical simulation of a binary black-hole coalescence with masses and spins consistent with the GW170104 observation. The strength of the gravitational wave is indicated by elevation as well as color, with blue indicating weak fields and yellow indicating strong fields. We rescale the amplitude of the gravitational wave during the simulation to show the signal during the entire animation not only close to merger, where it is strongest. The sizes of the black holes are increased by a factor of two to improve visibility. The bottom panel in the video shows the gravitational waveform starting at frequency of 25Hz. The fade in of the video corresponds to a frequency of about 30Hz.

© Numerical-relativistic simulation: S. Ossokine, A. Buonanno (Max Planck Institute for Gravitational Physics) and the Simulating eXtreme Spacetimes project; scientific visualization: T. Dietrich (Max Planck Institute for Gravitational Physics), R. Haas (NCSA).

The colour scheme gives me a headache, and there’s no sountrack, but it’s quite instructive nonetheless.

 

New Astronomy at the New Year (GW170104)

Posted in The Universe and Stuff with tags , on June 1, 2017 by telescoper

The inside story of the detection of gravitational wave event announced earlier today..

Write Science

by Shane L. Larson

Newton’s portrait.

January 4 holds a special place in the hearts of scientists — it is Isaac Newton’s birthday (*). Newton stood at the crossroads that led to modern science, and astronomy in particular. He was the first person to build a workable reflecting telescope, a design that now bears his name and for the past 4 centuries has been the dominant type of telescope used by amateurs and professionals alike. Newtonian telescopes have revealed much about the Cosmos to our wondering minds. Newton was also responsible for the first formulation of a physical law that describes the working of gravity, called the Universal Law of Gravitation. Today we use the Universal Law to launch satellites, send astronauts into orbit, convert the force of your feet on the bathroom scale into your “weight“, and a thousand other applications.  There is much to celebrate…

View original post 1,623 more words

LIGO Echoes, P-values and the False Discovery Rate

Posted in Astrohype, Bad Statistics, The Universe and Stuff with tags , , , , on December 12, 2016 by telescoper

Today is our staff Christmas lunch so I thought I’d get into the spirit by posting a grumbly article about a paper I found on the arXiv. In fact I came to this piece via a News item in Nature. Anyway, here is the abstract of the paper – which hasn’t been refereed yet:

In classical General Relativity (GR), an observer falling into an astrophysical black hole is not expected to experience anything dramatic as she crosses the event horizon. However, tentative resolutions to problems in quantum gravity, such as the cosmological constant problem, or the black hole information paradox, invoke significant departures from classicality in the vicinity of the horizon. It was recently pointed out that such near-horizon structures can lead to late-time echoes in the black hole merger gravitational wave signals that are otherwise indistinguishable from GR. We search for observational signatures of these echoes in the gravitational wave data released by advanced Laser Interferometer Gravitational-Wave Observatory (LIGO), following the three black hole merger events GW150914, GW151226, and LVT151012. In particular, we look for repeating damped echoes with time-delays of 8MlogM (+spin corrections, in Planck units), corresponding to Planck-scale departures from GR near their respective horizons. Accounting for the “look elsewhere” effect due to uncertainty in the echo template, we find tentative evidence for Planck-scale structure near black hole horizons at 2.9σ significance level (corresponding to false detection probability of 1 in 270). Future data releases from LIGO collaboration, along with more physical echo templates, will definitively confirm (or rule out) this finding, providing possible empirical evidence for alternatives to classical black holes, such as in firewall or fuzzball paradigms.

I’ve highlighted some of the text in bold. I’ve highlighted this because as written its wrong.

I’ve blogged many times before about this type of thing. The “significance level” quoted corresponds to a “p-value” of 0.0037 (or about 1/270). If I had my way we’d ban p-values and significance levels altogether because they are so often presented in a misleading fashion, as it is here.

What is wrong is that the significance level is not the same as the false detection probability.  While it is usually the case that the false detection probability (which is often called the false discovery rate) will decrease the lower your p-value is, these two quantities are not the same thing at all. Usually the false detection probability is much higher than the p-value. The physicist John Bahcall summed this up when he said, based on his experience, “about half of all 3σ  detections are false”. You can find a nice (and relatively simple) explanation of why this is the case here (which includes various references that are worth reading), but basically it’s because the p-value relates to the probability of seeing a signal at least as large as that observed under a null hypothesis (e.g.  detector noise) but says nothing directly about the probability of it being produced by an actual signal. To answer this latter question properly one really needs to use a Bayesian approach, but if you’re not keen on that I refer you to this (from David Colquhoun’s blog):

One problem with all of the approaches mentioned above was the need to guess at the prevalence of real effects (that’s what a Bayesian would call the prior probability). James Berger and colleagues (Sellke et al., 2001) have proposed a way round this problem by looking at all possible prior distributions and so coming up with a minimum false discovery rate that holds universally. The conclusions are much the same as before. If you claim to have found an effects whenever you observe a P value just less than 0.05, you will come to the wrong conclusion in at least 29% of the tests that you do. If, on the other hand, you use P = 0.001, you’ll be wrong in only 1.8% of cases.

Of course the actual false detection probability can be much higher than these limits, but they provide a useful rule of thumb,

To be fair the Nature item puts it more accurately:

The echoes could be a statistical fluke, and if random noise is behind the patterns, says Afshordi, then the chance of seeing such echoes is about 1 in 270, or 2.9 sigma. To be sure that they are not noise, such echoes will have to be spotted in future black-hole mergers. “The good thing is that new LIGO data with improved sensitivity will be coming in, so we should be able to confirm this or rule it out within the next two years.

Unfortunately, however, the LIGO background noise is rather complicated so it’s not even clear to me that this calculation based on “random noise”  is meaningful anyway.

The idea that the authors are trying to test is of course interesting, but it needs a more rigorous approach before any evidence (even “tentative” can be claimed). This is rather reminiscent of the problems interpreting apparent “anomalies” in the Cosmic Microwave Background, which is something I’ve been interested in over the years.

In summary, I’m not convinced. Merry Christmas.