Archive for the Astrohype Category

LIGO Echoes, P-values and the False Discovery Rate

Posted in Astrohype, Bad Statistics, The Universe and Stuff with tags , , , , on December 12, 2016 by telescoper

Today is our staff Christmas lunch so I thought I’d get into the spirit by posting a grumbly article about a paper I found on the arXiv. In fact I came to this piece via a News item in Nature. Anyway, here is the abstract of the paper – which hasn’t been refereed yet:

In classical General Relativity (GR), an observer falling into an astrophysical black hole is not expected to experience anything dramatic as she crosses the event horizon. However, tentative resolutions to problems in quantum gravity, such as the cosmological constant problem, or the black hole information paradox, invoke significant departures from classicality in the vicinity of the horizon. It was recently pointed out that such near-horizon structures can lead to late-time echoes in the black hole merger gravitational wave signals that are otherwise indistinguishable from GR. We search for observational signatures of these echoes in the gravitational wave data released by advanced Laser Interferometer Gravitational-Wave Observatory (LIGO), following the three black hole merger events GW150914, GW151226, and LVT151012. In particular, we look for repeating damped echoes with time-delays of 8MlogM (+spin corrections, in Planck units), corresponding to Planck-scale departures from GR near their respective horizons. Accounting for the “look elsewhere” effect due to uncertainty in the echo template, we find tentative evidence for Planck-scale structure near black hole horizons at 2.9σ significance level (corresponding to false detection probability of 1 in 270). Future data releases from LIGO collaboration, along with more physical echo templates, will definitively confirm (or rule out) this finding, providing possible empirical evidence for alternatives to classical black holes, such as in firewall or fuzzball paradigms.

I’ve highlighted some of the text in bold. I’ve highlighted this because as written its wrong.

I’ve blogged many times before about this type of thing. The “significance level” quoted corresponds to a “p-value” of 0.0037 (or about 1/270). If I had my way we’d ban p-values and significance levels altogether because they are so often presented in a misleading fashion, as it is here.

What is wrong is that the significance level is not the same as the false detection probability.  While it is usually the case that the false detection probability (which is often called the false discovery rate) will decrease the lower your p-value is, these two quantities are not the same thing at all. Usually the false detection probability is much higher than the p-value. The physicist John Bahcall summed this up when he said, based on his experience, “about half of all 3σ  detections are false”. You can find a nice (and relatively simple) explanation of why this is the case here (which includes various references that are worth reading), but basically it’s because the p-value relates to the probability of seeing a signal at least as large as that observed under a null hypothesis (e.g.  detector noise) but says nothing directly about the probability of it being produced by an actual signal. To answer this latter question properly one really needs to use a Bayesian approach, but if you’re not keen on that I refer you to this (from David Colquhoun’s blog):

One problem with all of the approaches mentioned above was the need to guess at the prevalence of real effects (that’s what a Bayesian would call the prior probability). James Berger and colleagues (Sellke et al., 2001) have proposed a way round this problem by looking at all possible prior distributions and so coming up with a minimum false discovery rate that holds universally. The conclusions are much the same as before. If you claim to have found an effects whenever you observe a P value just less than 0.05, you will come to the wrong conclusion in at least 29% of the tests that you do. If, on the other hand, you use P = 0.001, you’ll be wrong in only 1.8% of cases.

Of course the actual false detection probability can be much higher than these limits, but they provide a useful rule of thumb,

To be fair the Nature item puts it more accurately:

The echoes could be a statistical fluke, and if random noise is behind the patterns, says Afshordi, then the chance of seeing such echoes is about 1 in 270, or 2.9 sigma. To be sure that they are not noise, such echoes will have to be spotted in future black-hole mergers. “The good thing is that new LIGO data with improved sensitivity will be coming in, so we should be able to confirm this or rule it out within the next two years.

Unfortunately, however, the LIGO background noise is rather complicated so it’s not even clear to me that this calculation based on “random noise”  is meaningful anyway.

The idea that the authors are trying to test is of course interesting, but it needs a more rigorous approach before any evidence (even “tentative” can be claimed). This is rather reminiscent of the problems interpreting apparent “anomalies” in the Cosmic Microwave Background, which is something I’ve been interested in over the years.

In summary, I’m not convinced. Merry Christmas.



A Non-accelerating Universe?

Posted in Astrohype, The Universe and Stuff with tags , , , , , on October 26, 2016 by telescoper

There’s been quite a lot of reaction on the interwebs over the last few days much of it very misleading; here’s a sensible account) to a paper by Nielsen, Guffanti and Sarkar which has just been published online in Scientific Reports, an offshoot of Nature. I think the above link should take you an “open access” version of the paper but if it doesn’t you can find the arXiv version here. I haven’t cross-checked the two versions so the arXiv one may differ slightly.

Anyway, here is the abstract:

The ‘standard’ model of cosmology is founded on the basis that the expansion rate of the universe is accelerating at present — as was inferred originally from the Hubble diagram of Type Ia supernovae. There exists now a much bigger database of supernovae so we can perform rigorous statistical tests to check whether these ‘standardisable candles’ indeed indicate cosmic acceleration. Taking account of the empirical procedure by which corrections are made to their absolute magnitudes to allow for the varying shape of the light curve and extinction by dust, we find, rather surprisingly, that the data are still quite consistent with a constant rate of expansion.

Obviously I haven’t been able to repeat the statistical analysis but I’ve skimmed over what they’ve done and as far as I can tell it looks a fairly sensible piece of work (although it is a frequentist analysis). Here is the telling plot (from the Nature version)  in terms of the dark energy (y-axis) and matter (x-axis) density parameters:


Models shown in this plane by a line have the correct balance between Ωm, and ΩΛ to cancel out the decelerating effect of the former against the accelerating effect of the latter (a special case is the origin on the plot, which is called the Milne model and represents an entirely empty universe). The contours show “1, 2 and 3σ” contours, regarding all other parameters as nuisance parameters. It is true that the line of no acceleration does go inside the 3σcontour so in that sense is not entirely inconsistent with the data. On the other hand, the “best fit” (which is at the point Ωm=0.341, ΩΛ=0.569) does represent an accelerating universe.

I am not all that surprised by this result, actually. I’ve always felt that taken on its own the evidence for cosmic acceleration from supernovae alone was not compelling. However, when it is combined with other measurements (particularly of the cosmic microwave background and large-scale structure) which are sensitive to other aspects of the cosmological space-time geometry, the agreement is extremely convincing and has established a standard “concordance” cosmology. The CMB, for example, is particularly sensitive to spatial curvature which, measurements tells us, must be close to zero. The Milne model, on the other hand, has a large (negative) spatial curvature entirely excluded by CMB observations. Curvature is regarded as a “nuisance parameter” in the above diagram.

I think this paper is a worthwhile exercise. Subir Sarkar (one of the authors) in particular has devoted a lot of energy to questioning the standard ΛCDM model which far too many others accept unquestioningly. That’s a noble thing to do, and it is an essential part of the scientific method, but this paper only looks at one part of an interlocking picture. The strongest evidence comes from the cosmic microwave background and despite this reanalysis I feel the supernovae measurements still provide a powerful corroboration of the standard cosmology.

Let me add, however, that the supernovae measurements do not directly measure cosmic acceleration. If one tries to account for them with a model based on Einstein’s general relativity and the assumption that the Universe is on large-scales is homogeneous and isotropic and with certain kinds of matter and energy then the observations do imply a universe that accelerates. Any or all of those assumptions may be violated (though some possibilities are quite heavily constrained). In short we could, at least in principle, simply be interpreting these measurements within the wrong framework, and statistics can’t help us with that!

New: Top Ten Gaia Facts!

Posted in Astrohype, The Universe and Stuff with tags , , , on September 14, 2016 by telescoper

After today’s first release of data by the Gaia Mission, as a service to the community, for the edification of the public at large, and by popular demand, here is a list of Top Ten Gaia Facts.

Gaia looks nothing like the Herschel Space Observatory shown here.

Gaia looks nothing like the Herschel Space Observatory shown here.


  1. The correct pronunciation of GAIA is as in “gayer”. Please bear this in mind when reading any press articles about the mission.
  2. The GAIA spacecraft will orbit the Sun at the Second Lagrange Point, the only place in the Solar System where the  effects of cuts in the UK science budget can not be felt.
  3. The data processing challenges posed by GAIA are immense; the billions of astrometric measurements resulting from the mission will be analysed using the world’s biggest Excel Spreadsheet.
  4. To provide secure backup storage of the complete GAIA data set, the European Space Agency has commandeered the world’s entire stock of 3½ inch floppy disks.
  5. As well as measuring billions of star positions and velocities, GAIA is expected to discover thousands of new asteroids and the hiding place of Lord Lucan.
  6. GAIA can measure star positions to an accuracy of a few microarcseconds. That’s the angle subtended by a single pubic hair at a distance of 1000km.
  7. The precursor to GAIA was a satellite called Hipparcos, which is not how you spell Hipparchus.
  8. The BBC will be shortly be broadcasting a new 26-part TV series about GAIA. Entitled WOW! Gaia! That’s Soo Amaazing… it will be presented by Britain’s leading expert on astrometry, Professor Brian Cox.
  9. Er…
  10. That’s it.

From Sappho to Babbage

Posted in Astrohype, Poetry, The Universe and Stuff with tags , , , on May 24, 2016 by telescoper

The English mathematician Charles Babbage, who designed and built the first programmable calculating machine, wrote to the (then) young poet Tennyson, whose poem The Vision of Sin he had recently read:


I like to think Babbage was having a laugh with Tennyson here, rather than expressing a view that poetry should be taken so literally, but you never know..

Anyway, I was reminded of the above letter by the much-hyped recent story of the alleged astronomical “dating” of this ancient poem (actually just a fragment) by Sappho:

Tonight I’ve watched
the moon and then
the Pleiades
go down

The night is now
half-gone; youth
goes; I am

in bed alone

It is a trivial piece of astronomical work to decuded that if the “Pleiades” does indeed refer to the constellation and “the night is now half-gone” means sometime around midnight, then the scene described in the fragment happened, if it happened at all, between January and March. However, as an excellent rebuttal piece by Darin Hayton points out, the assumptions needed to arrive at a specific date are all questionable.

More important, poetry is not and never has been intended for such superficial interpretation.  That goes for modern works, but is even more true for ancient verse. Who knows what the imagery and allusions in the text would have meant to an audience when it was composed, over 2500 years ago, but which are lost on a modern reader?

I’m not so much saddened that someone thought to study the possible astronomical interpretation an ancient text, even if they didn’t do a very thorough job of it. At least that means they are interested in poetry, although I doubt they were joking as Babbage may have been.

What does sadden me, however, is the ludicrous hype generated by the University of Texas publicity machine. There’s far too much of that about, and it’s getting worse.



On the Theory of Gravitational Wave Rumour Sources

Posted in Astrohype, The Universe and Stuff, Uncategorized with tags , , on January 12, 2016 by telescoper

There has been a great deal of excitement almost nowhere in the astrophysics community since it was announced recently that rumours of the detection of gravitational waves had yet again begun to circulate, so I thought I would add here a brief discussion of the theoretical background to these phenomena.

The standard theoretical model of such rumours is that they are  produced from time to time during the lifetime of a supermassive science project after periods of relative quiescence. It is thought that they are associated with a perceived lack of publicity which might threaten funding and lead to financial collapse of the project. This stimulates a temporary emission of hype produced by vigorous gossip-mongering which acts to inflate the external profile of the project, resisting external pressures and restoring equilibrium. This general phenomenon is not restricted to gravitational wave detection, but also occurs across many other branches of Big Science, especially cosmology and particle physics.

However, observations of the latest outburst suggest support for a rival theory, in which rumours are produced not by the project itself but by some other body or bodies in orbit around it or even perhaps entirely independent of it. Although there is evidence in favour of this theory, it is relatively new and many questions remain to be answered. In particular it is not known what the effect of rumours produced in this way might be on the long-term evolution of the project or on the source itself.

Lawrence Krauss is 61.



Evidence for Liquid Water on Mars?

Posted in Astrohype, The Universe and Stuff with tags , , , , , , , , on September 28, 2015 by telescoper

There’s been a lot of excitement this afternoon about possible evidence for water on Mars from the Compact Reconnaissance Imaging Spectrometer for Mars (CRISM) on board the Mars Reconaissance Orbiter (MRO). Unfortunately, but I suppose inevitably, some of the media coverage has been a bit over the top, presenting the results as if they were proof of liquid water flowing on the Red Planet’s surface; NASA itself has pushed this interpretation. I think the results are indeed very interesting – but not altogether surprising, and by no means proof of the existence of flows of liquid water. And although they may indeed provide evidence confirming that there is water on Mars,  we knew that already (at least in the form of ice and water vapour).

The full results are reported in a paper in Nature Geoscience. The abstract reads:

Determining whether liquid water exists on the Martian surface is central to understanding the hydrologic cycle and potential for extant life on Mars. Recurring slope lineae, narrow streaks of low reflectance compared to the surrounding terrain, appear and grow incrementally in the downslope direction during warm seasons when temperatures reach about 250–300K, a pattern consistent with the transient flow of a volatile species1, 2, 3. Brine flows (or seeps) have been proposed to explain the formation of recurring slope lineae1, 2, 3, yet no direct evidence for either liquid water or hydrated salts has been found4. Here we analyse spectral data from the Compact Reconnaissance Imaging Spectrometer for Mars instrument onboard the Mars Reconnaissance Orbiter from four different locations where recurring slope lineae are present. We find evidence for hydrated salts at all four locations in the seasons when recurring slope lineae are most extensive, which suggests that the source of hydration is recurring slope lineae activity. The hydrated salts most consistent with the spectral absorption features we detect are magnesium perchlorate, magnesium chlorate and sodium perchlorate. Our findings strongly support the hypothesis that recurring slope lineae form as a result of contemporary water activity on Mars.

Here’s a picture taken with the High Resolution Imaging Science Experiment (HIRISE) on MRO showing some of the recurring slope lineae (RSL):


You can see a wonderful gallery of other HIRISE images of other such features here.

The dark streaky stains in this and other examples are visually very suggestive of the possibility they were produced by flowing liquid. They also come and go with the Martian seasons, which suggests that they might involve something that melts in the summer and freezes in the winter. Putting these two facts together raises the quite reasonable question of whether, if that is indeed how they’re made, that liquid might be water.

What is new about the latest results that adds to the superb detail revealed by the HIRISE images – is that there is spectroscopic information that yields clues about the chemical composition of the stuff in the RSLs:



The black lines denote spectra that are taken at two different locations; the upper one has been interpreted as indicating the presence of some mixture of hydrated Calcium, Magnesium and Sodium Perchlorates (i.e. salts). I’m not a chemical spectroscopist so I don’t know whether other interpretations are possible, though I can’t say that I’m overwhelmingly convinced by the match between the data from laboratory specimens and that from Mars…

Anyway, if that is indeed what the spectroscopy indicates then the obvious conclusion is that there is water present, for without water there can be no hydrated salts. This water could have been absorbed from the atmospheric vapour or from the ice below the surface. The presence of salts would lowers the melting point of water ice, so this could explain how there could be some form of liquid flow at the sub-zero temperatures prevalent even in a Martian summer. It would not be pure running water, however, but an extremely concentrated salt solution, much saltier than sea water, probably in the form of a rather sticky brine. This brine might flow – or perhaps creep – down the sloping terrain (briefly) in the summer and then freeze. But nothing has actually been observed to flow in such a way. It seems to me – as a non-expert – that the features could be caused not by a flow of liquid, but by the disruption of the Martian surface, caused by melting and freezing, involving  movement of solid material, or perhaps localized seeping. I’m not saying that it’s impossible that a flow of briny liquid is responsible for the features, just that I think it’s far from proven. But there’s no doubt that whatever is going on is fascinatingly complicated!

The last sentence of the abstract quoted above reads:

Our findings strongly support the hypothesis that recurring slope lineae form as a result of contemporary water activity on Mars.

I’m not sure about the “strongly support” but “contemporary water activity” is probably fair as it includes the possibilities I discussed above, but it does seem to have led quite a few people to jump to the conclusion that it means “flowing water”, which I don’t think it does. Am I wrong to be so sceptical? Let me know through the comments box!



Gamma-Ray Bursts and the Cosmological Principle

Posted in Astrohype, Bad Statistics, The Universe and Stuff with tags , , , on September 13, 2015 by telescoper

There’s been a reasonable degree of hype surrounding a paper published in Monthly Notices of the Royal Astronomical Society (and available on the arXiv here). The abstract of this paper reads:

According to the cosmological principle (CP), Universal large-scale structure is homogeneous and isotropic. The observable Universe, however, shows complex structures even on very large scales. The recent discoveries of structures significantly exceeding the transition scale of 370 Mpc pose a challenge to the CP. We report here the discovery of the largest regular formation in the observable Universe; a ring with a diameter of 1720 Mpc, displayed by 9 gamma-ray bursts (GRBs), exceeding by a factor of 5 the transition scale to the homogeneous and isotropic distribution. The ring has a major diameter of 43° and a minor diameter of 30° at a distance of 2770 Mpc in the 0.78 < z < 0.86 redshift range, with a probability of 2 × 10−6 of being the result of a random fluctuation in the GRB count rate. Evidence suggests that this feature is the projection of a shell on to the plane of the sky. Voids and string-like formations are common outcomes of large-scale structure. However, these structures have maximum sizes of 150 Mpc, which are an order of magnitude smaller than the observed GRB ring diameter. Evidence in support of the shell interpretation requires that temporal information of the transient GRBs be included in the analysis. This ring-shaped feature is large enough to contradict the CP. The physical mechanism responsible for causing it is unknown.

The so-called “ring” can be seen here:

In my opinion it’s not a ring at all, but an outline of Australia. What’s the probability of a random distribution of dots looking exactly like that? Is it really evidence for the violation of the Cosmological Principle, or for the existence of the Cosmic Antipodes?

For those of you who don’t get that gag, a cosmic antipode occurs in, e.g., closed Friedmann cosmologies in which the spatial sections take the form of a hypersphere (or 3-sphere). The antipode is the point diametrically opposite the observer on this hypersurface, just as it is for the surface of a 2-sphere such as the Earth. The antipode is only visible if it lies inside the observer’s horizon, a possibility which is ruled out for standard cosmologies by current observations. I’ll get my coat.

Anyway, joking apart, the claims in the abstract of the paper are extremely strong but the statistical arguments supporting them are deeply unconvincing. Indeed, I am quite surprised the paper passed peer review. For a start there’s a basic problem of “a posteriori” reasoning here. We see a group of objects that form a map of Australia ring and then are surprised that such a structure appears so rarely in simulations of our favourite model. But all specific configurations of points are rare in a Poisson point process. We would be surprised to see a group of dots in the shape of a pretzel too, or the face of Jesus, but that doesn’t mean that such an occurrence has any significance. It’s an extraordinarily difficult problem to put a meaningful measure on the space of geometrical configurations, and this paper doesn’t succeed in doing that.

For a further discussion of the tendency that people have to see patterns where none exist, take a look at this old post from which I’ve taken this figure which is generated by drawing points independently and uniformly randomly:

pointaI can see all kinds of shapes in this pattern, but none of them has any significance (other than psychological). In a mathematically well-defined sense there is no structure in this pattern! Add to that difficulty the fact that so few points are involved and I think it becomes very clear that this “structure” doesn’t provide any evidence at all for the violation of the Cosmological Principle. Indeed it seems neither do the authors. The very last paragraph of the paper is as follows:

GRBs are very rare events superimposed on the cosmic
web identified by superclusters. Because of this, the ring is
probably not a real physical structure. Further studies are
needed to reveal whether or not the Ring could have been
produced by a low-frequency spatial harmonic of the large-
scale matter density distribution and/or of universal star
forming activity.

It’s a pity that this note of realism didn’t make it into either the abstract or, more importantly, the accompanying press release. Peer review will never be perfect, but we can do without this sort of hype. Anyway, I confidently predict that a proper refutation will appear shortly….

P.S. For a more technical discussion of the problems of inferring the presence of large structures from sparsely-sampled distributions, see here.