On (un)falsifiability of paradigms with Bayesian model selection …

Posted in Uncategorized on July 25, 2015 by telescoper


Yesterday’s post is generating quite a lot if traffic for a weekend so I thought I would reblog this piece on the same topic..

Originally posted on Another Astrostatistics Blog:

I noticed an unusual contribution on the philosophy of science with Bayesian model selection by Gubitosi et al. on astro ph the other day, in which some rather bold claims are made, e.g.

“By considering toy models we illustrate how unfalsifiable models and paradigms are always favoured by the Bayes factor.”

Despite the authors making a number of sniping comments about the sociology of “proof of inflation” claims in astronomy, their meta-reflections did not reach a point of self-awareness at which they were able to escape my own sociological observation: the bolder the claims made by astronomers about Bayes theorem, the narrower their reading of the past literature on the subject. Indeed, in this manuscript there are no references at all to any previous work on the role of Bayes factors in scientific decision making, even from within the astronomical canon (leaving beside the history of statistics); more precisely, it…

View original 572 more words

Falisifiability versus Testability in Cosmology

Posted in Bad Statistics, The Universe and Stuff with tags , , , , , on July 24, 2015 by telescoper

A paper came out a few weeks ago on the arXiv that’s ruffled a few feathers here and there so I thought I would make a few inflammatory comments about it on this blog. The article concerned, by Gubitosi et al., has the abstract:


I have to be a little careful as one of the authors is a good friend of mine. Also there’s already been a critique of some of the claims in this paper here. For the record, I agree with the critique and disagree with the original paper, that the claim below cannot be justfied.

…we illustrate how unfalsifiable models and paradigms are always favoured by the Bayes factor.

If I get a bit of time I’ll write a more technical post explaining why I think that. However, for the purposes of this post I want to take issue with a more fundamental problem I have with the philosophy of this paper, namely the way it adopts “falsifiablity” as a required characteristic for a theory to be scientific. The adoption of this criterion can be traced back to the influence of Karl Popper and particularly his insistence that science is deductive rather than inductive. Part of Popper’s claim is just a semantic confusion. It is necessary at some point to deduce what the measurable consequences of a theory might be before one does any experiments, but that doesn’t mean the whole process of science is deductive. As a non-deductivist I’ll frame my argument in the language of Bayesian (inductive) inference.

Popper rejects the basic application of inductive reasoning in updating probabilities in the light of measured data; he asserts that no theory ever becomes more probable when evidence is found in its favour. Every scientific theory begins infinitely improbable, and is doomed to remain so. There is a grain of truth in this, or can be if the space of possibilities is infinite. Standard methods for assigning priors often spread the unit total probability over an infinite space, leading to a prior probability which is formally zero. This is the problem of improper priors. But this is not a killer blow to Bayesianism. Even if the prior is not strictly normalizable, the posterior probability can be. In any case, given sufficient relevant data the cycle of experiment-measurement-update of probability assignment usually soon leaves the prior far behind. Data usually count in the end.

I believe that deductvism fails to describe how science actually works in practice and is actually a dangerous road to start out on. It is indeed a very short ride, philosophically speaking, from deductivism (as espoused by, e.g., David Hume) to irrationalism (as espoused by, e.g., Paul Feyeraband).

The idea by which Popper is best known is the dogma of falsification. According to this doctrine, a hypothesis is only said to be scientific if it is capable of being proved false. In real science certain “falsehood” and certain “truth” are almost never achieved. The claimed detection of primordial B-mode polarization in the cosmic microwave background by BICEP2 was claimed by some to be “proof” of cosmic inflation, which it wouldn’t have been even if it hadn’t subsequently shown not to be a cosmological signal at all. What we now know to be the failure of BICEP2 to detect primordial B-mode polarization doesn’t disprove inflation either.

Theories are simply more probable or less probable than the alternatives available on the market at a given time. The idea that experimental scientists struggle through their entire life simply to prove theorists wrong is a very strange one, although I definitely know some experimentalists who chase theories like lions chase gazelles. The disparaging implication that scientists live only to prove themselves wrong comes from concentrating exclusively on the possibility that a theory might be found to be less probable than a challenger. In fact, evidence neither confirms nor discounts a theory; it either makes the theory more probable (supports it) or makes it less probable (undermines it). For a theory to be scientific it must be capable having its probability influenced in this way, i.e. amenable to being altered by incoming data “i.e. evidence”. The right criterion for a scientific theory is therefore not falsifiability but testability. It follows straightforwardly from Bayes theorem that a testable theory will not predict all things with equal facility. Scientific theories generally do have untestable components. Any theory has its interpretation, which is the untestable penumbra that we need to supply to make it comprehensible to us. But whatever can be tested can be regared as scientific.

So I think the Gubitosi et al. paper starts on the wrong foot by focussing exclusively on “falsifiability”. The issue of whether a theory is testable is complicated in the context of inflation because prior probabilities for most observables are difficult to determine with any confidence because we know next to nothing about either (a) the conditions prevailing in the early Universe prior to the onset of inflation or (b) how properly to define a measure on the space of inflationary models. Even restricting consideration to the simplest models with a single scalar field, initial data are required for the scalar field (and its time derivative) and there is also a potential whose functional form is not known. It is therfore a far from trivial task to assign meaningful prior probabilities on inflationary models and thus extremely difficult to determine the relative probabilities of observables and how these probabilities may or may not be influenced by interactions with data. Moreover, the Bayesian approach involves comparing probabilities of competing theories, so we also have the issue of what to compare inflation with…

The question of whether cosmic inflation (whether in general concept or in the form of a specific model) is testable or not seems to me to boil down to whether it predicts all possible values of relevant observables with equal ease. A theory might be testable in principle, but not testable at a given time if the available technology at that time is not able to make measurements that can distingish between that theory and another. Most theories have to wait some time for experiments can be designed and built to test them. On the other hand a theory might be untestable even in principle, if it is constructed in such a way that its probability can’t be changed at all by any amount of experimental data. As long as a theory is testable in principle, however, it has the right to be called scientific. If the current available evidence can’t test it we need to do better experiments. On other words, there’s a problem with the evidence not the theory.

Gubitosi et al. are correct in identifying the important distinction between the inflationary paradigm, which encompasses a large set of specific models each formulated in a different way, and an individual member of that set. I also agree – in contrast to many of my colleagues – that it is actually difficult to argue that the inflationary paradigm is currently falsfiable testable. But that doesn’t necessarily mean that it isn’t scientific. A theory doesn’t have to have been tested in order to be testable.


Posted in Cricket, Poetry with tags , , , on July 23, 2015 by telescoper

Something rather different from my usual poetry postings. This poem was written in memory of celebrated cricketer Hedley Verity, who was wounded in action in Caserta, Sicily and taken prisoner; he later died of his wounds in a Prisoner-of-War camp at the age of 38. It was a tragic end to a life that had given so much to the world of cricket.

The following is a brief account of his playing career taken from the website where I found the poem. You can find a longer biography here.

Verity was born in 1905 within sight of Headingley Cricket Ground. It seems strange to think that Verity was originally turned down by Yorkshire at trials in 1926, but he was eventually given a chance by the county in 1930 and, of course, became a fixture until the start of the war. He was the natural successor to that other great Yorkshire left-arm spinner, Wilfred Rhodes, whose career drew to a close in 1930 after an amazing 883 games for the county. Verity was never going to get close – Hitler saw to that – but he did turn out for Yorkshire 278 times and in that time he produced some remarkable bowling analyses.

In 1931 he took ten for 36 off 18.4 overs against Warwickshire at Leeds, but incredibly he bettered these figures the following season by taking ten for ten in 19.4 overs against Nottinghamshire, also at Headingley. They remain the county’s best bowling figures for an innings while Verity’s 17 for 91 against Essex at Leyton in 1933 remain Yorkshire’s best bowling in a match. Verity claimed nine wickets in an innings seven times for Yorkshire. He took 100 wickets in a season nine times and took 200 wickets in three consecutive seasons between 1935-37. He ended with 1,956 first-class wickets at an average of 14.9, took five wickets in an innings 164 times and ten wickets in a match 54 times. On 1 September, 1939, in the last first-class match before war was declared, he took seven for nine at Hove against Sussex.

The year after he first appeared for Yorkshire, Verity made his England debut against New Zealand at The Oval, finishing the game with four wickets. After that summer he was ignored until 1932/33, the Bodyline Series, in which he took 11 wickets, including Bradman twice. By the time his career was over, Verity had dismissed Bradman ten times, a figure matched only by Grimmett. As with his domestic career, Verity’s international performances threw up some astonishing bowling figures. He took eight for 43 and finished with match figures of 15 for 104 against Australia at Lord’s in 1934. His stamina was demonstrated during the 1938-39 tour of South Africa when he bowled 95.6 eight-ball overs in an innings at Durban, taking four for 184. By the time war arrived, Verity had taken 144 wickets at an average of 24.37.

During the war he was a captain in the Green Howards. He sustained his wounds in the battle of Catania in Sicily and died on 31 July, 1943. His grave is at Caserta Military Cemetery, some 16 miles from Naples.

Ironically, the poet, Drummond Allison, was also killed in action during World War 2.

The ruth and truth you taught have come full-circle
On that fell island all whose history lies,
Far now from Bramhall Lane and far from Scarborough
You recollect how foolish are the wise.

On this great ground more marvellous than Lord’s
– Time takes more spin than nineteen thirty four –
You face at last that vast that Bradman-shaming
Batsman whose cuts obey no natural law.

Run up again, as gravely smile as ever,
Veer without fear your left unlucky arm
In His so dark direction, but no length
However lovely can disturb the harm
That is His style, defer the winning drive
Or shake the crowd from their uproarious calm.

by Drummond Allison (1921-1943).

Exciting Opportunity in Experimental Physics at the University of Sussex!

Posted in Education, The Universe and Stuff with tags , , , , on July 23, 2015 by telescoper

Just a quick update on the news that Department of Physics & Astronomy at the University of Sussex has an exciting opportunity in the form of a brand new Chair position in Experimental Physics. The advertisement appeared on the University of Sussex website somedays ago. But it has now appeared on Nature Jobs and the Times Higher websites. It is also in today’s print edition of the Times Higher. At least I think it is. I couldn’t find a copy in W.H. Smith’s when I went there today. Obviously it has sold out because word has got out about this job!

I’m taking the liberty of reposting a description of the new position here, but for fuller details please visit the formal advertisement.


The School of Mathematical and Physical Sciences seeks to appoint a Professor in Experimental Physics in the Department of Physics & Astronomy to lead the next phase of expansion and diversification of the research portfolio within the School by establishing an entirely new research activity in laboratory-based physics.

Sufficient resources will be made available to the selected candidate to establish a new group at Sussex in their field of experimental physics including, for example, condensed matter (interpreted widely), materials science, nanophysics or biophysics. Applicants in research areas with scope for interdisciplinary collaborations with other Schools at the University of Sussex (e.g. Life Sciences, Engineering & Informatics or Brighton and Sussex Medical School) are encouraged, especially  those in areas with potential for generating research impact, as defined in the context of the UK Research Excellence Framework.

The successful applicant will have a proven track-record of success in obtaining substantial external funding through research grants and/or industrial sponsorship.

The appointee will be supported with substantial (seven-figure) sum for start-up funding and an extensive newly-refurbished laboratory space. The financial package on offer will also support the appointment of at least two further experimental lectureships; the appointed professor is expected to be strongly involved in recruitment to these positions.

Informal (and confidential) enquiries may be addressed in the first instance to the Head of School, Professor Peter Coles (P.Coles@sussex.ac.uk).

The Curious Case of the 3.5 keV “Line” in Cluster Spectra

Posted in Bad Statistics, The Universe and Stuff with tags , , , , , , on July 22, 2015 by telescoper

Earlier this week I went to a seminar. That’s a rare enough event these days given all the other things I have to do. The talk concerned was by Katie Mack, who was visiting the Astronomy Centre and it contained a nice review of the general situation regarding the constraints on astrophysical dark matter from direct and indirect detection experiments. I’m not an expert on experiments – I’m banned from most laboratories on safety grounds – so it was nice to get a review from someone who knows what they’re talking about.

One of the pieces of evidence discussed in the talk was something I’ve never really looked at in detail myself, namely the claimed evidence of an  emission “line” in the spectrum of X-rays emitted by the hot gas in galaxy clusters. I put the word “line” in inverted commas for reasons which will soon become obvious. The primary reference for the claim is a paper by Bulbul et al which is, of course, freely available on the arXiv.

The key graph from that paper is this:


The claimed feature – it stretches the imagination considerably to call it a “line” – is shown in red. No, I’m not particularly impressed either, but this is what passes for high-quality data in X-ray astronomy!

There’s a nice review of this from about a year ago here which says this feature

 is very significant, at 4-5 astrophysical sigma.

I’m not sure how to convert astrophysical sigma into actual sigma, but then I don’t really like sigma anyway. A proper Bayesian model comparison is really needed here. If it is a real feature then a plausible explanation is that it is produced by the decay of some sort of dark matter particle in a manner that involves the radiation of an energetic photon. An example is the decay of a massive sterile neutrino – a hypothetical particle that does not participate in weak interactions –  into a lighter standard model neutrino and a photon, as discussed here. In this scenario the parent particle would have a mass of about 7keV so that the resulting photon has an energy of half that. Such a particle would constitute warm dark matter.

On the other hand, that all depends on you being convinced that there is anything there at all other than a combination of noise and systematics. I urge you to read the paper and decide. Then perhaps you can try to persuade me, because I’m not at all sure. The X-ray spectrum of hot gas does have a number of known emission features in it that needed to be subtracted before any anomalous emission can be isolated. I will remark however that there is a known recombination line of Argon that lies at 3.6 keV, and you have to be convinced that this has been subtracted correctly if the red bump is to be interpreted as something extra. Also note that all the spectra that show this feature are obtained using the same instrument – on the XMM/Newton spacecraft which makes it harder to eliminate the possibility that it is an instrumental artefact.

I’d be interested in comments from X-ray folk about how confident we should be that the 3.5 keV “anomaly” is real…

Software Use in Astronomy

Posted in Education, The Universe and Stuff with tags , , , , on July 21, 2015 by telescoper

I just saw an interesting paper which hit the arXiv last week and thought I would share it here. It’s called Software Use in Astronomy: An Informal Survey and the abstract is here:

softwareA couple of things are worth remarking upon. One concerns Python. Although I’m not surprised that Python is Top of the Pops amongst astronomers – like many Physics & Astronomy departments we actually teach it to undergraduates here at the University of Sussex – it is notable that its popularity is a relatively recent phenomenon and it’s quite impressive how rapidly it has caught on.

Another interesting thingis the continuing quite heavy use of Fortran. Most computer scientists would consider this to be an obsolete language, and is presumably mainly used because of inertia: some important and well established codes are written in it and presumably it’s too much effort to rewrite them from scratch in something more modern. I would have thought that Fortran would have been used primarily by older academics, i.e. old dogs who can’t learn new programming tricks. However, that doesn’t really seem to be the case based on the last sentence of the abstract.

Finally, it’s quite surprising that over 40% of astronomers claim to have had no training in software development. We do try to embed that particular skill in graduate programmes nowadays, but it seems that doesn’t always work!

Anyway, do read the paper yourself. It’s very interesting. Any further comments through the box below please, but please ensure they compile before submitting them…


The England Cricket Team – An Apology

Posted in Cricket with tags , on July 21, 2015 by telescoper

Some days ago I wrote a post on this blog about the 1st Ashes Test between England and Australia at Cardiff which resulted in an England victory. In that piece I celebrated the team spirit of England’s cricketers and some memorable performances with both bat and ball. I also suggested that England had a realistic prospect of regaining the Ashes.

However, in the light of Australia’s comprehensive victory in the 2nd Ashes Test at Lord’s during which the England bowlers were ineffectual, their batsmen inept and the team spirit non-existent, I now realize that my earlier post was misleading and that they actually have absolutely no chance of regaining the Ashes. I apologize for any inconvenience caused by my ealier error.

I hope this clarifies the situation.

P.S. Kevin Pietersen is 35.


Get every new post delivered to your Inbox.

Join 4,284 other followers