Archive for December 19, 2009

Dark Squib

Posted in Bad Statistics, Science Politics, The Universe and Stuff with tags , on December 19, 2009 by telescoper

After today’s lengthy pre-Christmas traipse around Cardiff in the freezing cold, I don’t think I can summon up the energy for a lengthy post today. However, today’s cryogenic temperatures did manage to remind me that I hadn’t closed the book on a previous story about rumours of a laboratory detection of dark matter by the experiment known as CDMS. The main rumour – that there was going to be a paper in Nature reporting the definite detection of dark matter particles – turned out to be false, but there was a bit of truth after all, in that they did put out a paper yesterday (18th December, the date that the original rumour suggested their paper would come out).  There’s also an executive summary of the results here.

It turns out that the experiment has seen two events that might, just might, be the Weakly Interacting Massive Particles (WIMPs) that are most theorists favoured candidate for cold dark matter. However, they might also be due to background events generated by other stray particles getting into the works. It’s impossible to tell at this stage whether the signal is real or not. Based on the sort of naive  frequentist statistical treatment of the data that for some reason is what particle physicists seem to prefer, there’s a 23% chance of their signal being background rather than dark matter. In other words, it’s about a one-sigma detection. In fact, if you factor in the possibility of a systematic error in the background counts – these are very difficult things to calibrate precisely – then the significance of the result decreases even further. And if you do it all properly, in a Bayesian way with an appropriate prior then the most probable result is no detection. Andrew Jaffe gives some details on his blog.

There is no universally accepted criterion for what constitutes a definite detection, but I’ve been told recently by the editor of Nature himself that if it’s less than 3-sigma (a probability of about 1% of it arising) then they’re unlikely to publish it. If it’s 2-sigma (5%) then it’s interesting, but not conclusive, but at 1-sigma it’s not worth writing home about never mind writing a press release.

I should  add that none of their results has yet been subject to peer review either. I can only guess that CDMS must be undergoing a funding review pretty soon and wanted to use the media to show it was producing the goods. I can’t say I’m impressed with these antics, and I doubt if the reviewers will be either.

Unfortunately, the fact that this is all so inconclusive from a scientific point of view hasn’t stopped various organs getting hold of the wrong end of the stick and starting to beat about the bush with it. New Scientist‘s Twitter feed screamed

Clear signal of dark matter detected in Minnesota!

although the article itself was a bit better informed. The Guardian ran a particularly poor story,  impressive only in the way it crammed so many misconceptions into such a short piece.

This episode takes me back to a theme I’ve touched on many times on this blog, which is that scientific results are very rarely black-and-white and they have to be treated carefully in appropriate probabilistic terms. Unfortunately, the media and the public have a great deal of difficulty understanding the subtleties of this and what gets across in the public domain can be either garbled or downright misleading. Most often in science the correct answer isn’t “true” or “false” but somewhere in between.

Of course, with more measurements, better statistics and stronger control of systematics this CDMS result may well turn into a significant detection. If it does then it will be a great scientific breakthrough and they’ll have my congratulations straight away, tempered with a certain amount of sadness that there will be no UK competitors in the race owing to our recent savage funding cuts. But we’re not there yet. So far, it’s just a definite maybe.