Bad Statistics, Bad Science

I saw an interesting article in Nature the opening paragraph of which reads:

The past few years have seen a slew of announcements of major discoveries in particle astrophysics and cosmology. The list includes faster-than-light neutrinos; dark-matter particles producing γ-rays; X-rays scattering off nuclei underground; and even evidence in the cosmic microwave background for gravitational waves caused by the rapid inflation of the early Universe. Most of these turned out to be false alarms; and in my view, that is the probable fate of the rest.

The piece goes on to berate physicists for being too trigger-happy in claiming discoveries, the BICEP2 fiasco being a prime example. I agree that this is a problem, but it goes far beyond physics. In fact its endemic throughout science. A major cause of it is abuse of statistical reasoning.

Anyway, I thought I’d take the opportunity to re-iterate why I statistics and statistical reasoning are so important to science. In fact, I think they lie at the very core of the scientific method, although I am still surprised how few practising scientists are comfortable with even basic statistical language. A more important problem is the popular impression that science is about facts and absolute truths. It isn’t. It’s a process. In order to advance it has to question itself. Getting this message wrong – whether by error or on purpose -is immensely dangerous.

Statistical reasoning also applies to many facets of everyday life, including business, commerce, transport, the media, and politics. Probability even plays a role in personal relationships, though mostly at a subconscious level. It is a feature of everyday life that science and technology are deeply embedded in every aspect of what we do each day. Science has given us greater levels of comfort, better health care, and a plethora of labour-saving devices. It has also given us unprecedented ability to destroy the environment and each other, whether through accident or design.

Civilized societies face rigorous challenges in this century. We must confront the threat of climate change and forthcoming energy crises. We must find better ways of resolving conflicts peacefully lest nuclear or conventional weapons lead us to global catastrophe. We must stop large-scale pollution or systematic destruction of the biosphere that nurtures us. And we must do all of these things without abandoning the many positive things that science has brought us. Abandoning science and rationality by retreating into religious or political fundamentalism would be a catastrophe for humanity.

Unfortunately, recent decades have seen a wholesale breakdown of trust between scientists and the public at large. This is due partly to the deliberate abuse of science for immoral purposes, and partly to the sheer carelessness with which various agencies have exploited scientific discoveries without proper evaluation of the risks involved. The abuse of statistical arguments have undoubtedly contributed to the suspicion with which many individuals view science.

There is an increasing alienation between scientists and the general public. Many fewer students enrol for courses in physics and chemistry than a a few decades ago. Fewer graduates mean fewer qualified science teachers in schools. This is a vicious cycle that threatens our future. It must be broken.

The danger is that the decreasing level of understanding of science in society means that knowledge (as well as its consequent power) becomes concentrated in the minds of a few individuals. This could have dire consequences for the future of our democracy. Even as things stand now, very few Members of Parliament are scientifically literate. How can we expect to control the application of science when the necessary understanding rests with an unelected “priesthood” that is hardly understood by, or represented in, our democratic institutions?

Very few journalists or television producers know enough about science to report sensibly on the latest discoveries or controversies. As a result, important matters that the public needs to know about do not appear at all in the media, or if they do it is in such a garbled fashion that they do more harm than good.

Years ago I used to listen to radio interviews with scientists on the Today programme on BBC Radio 4. I even did such an interview once. It is a deeply frustrating experience. The scientist usually starts by explaining what the discovery is about in the way a scientist should, with careful statements of what is assumed, how the data is interpreted, and what other possible interpretations might be and the likely sources of error. The interviewer then loses patience and asks for a yes or no answer. The scientist tries to continue, but is badgered. Either the interview ends as a row, or the scientist ends up stating a grossly oversimplified version of the story.

Some scientists offer the oversimplified version at the outset, of course, and these are the ones that contribute to the image of scientists as priests. Such individuals often believe in their theories in exactly the same way that some people believe religiously. Not with the conditional and possibly temporary belief that characterizes the scientific method, but with the unquestioning fervour of an unthinking zealot. This approach may pay off for the individual in the short term, in popular esteem and media recognition – but when it goes wrong it is science as a whole that suffers. When a result that has been proclaimed certain is later shown to be false, the result is widespread disillusionment.

The worst example of this tendency that I can think of is the constant use of the phrase “Mind of God” by theoretical physicists to describe fundamental theories. This is not only meaningless but also damaging. As scientists we should know better than to use it. Our theories do not represent absolute truths: they are just the best we can do with the available data and the limited powers of the human mind. We believe in our theories, but only to the extent that we need to accept working hypotheses in order to make progress. Our approach is pragmatic rather than idealistic. We should be humble and avoid making extravagant claims that can’t be justified either theoretically or experimentally.

The more that people get used to the image of “scientist as priest” the more dissatisfied they are with real science. Most of the questions asked of scientists simply can’t be answered with “yes” or “no”. This leaves many with the impression that science is very vague and subjective. The public also tend to lose faith in science when it is unable to come up with quick answers. Science is a process, a way of looking at problems not a list of ready-made answers to impossible problems. Of course it is sometimes vague, but I think it is vague in a rational way and that’s what makes it worthwhile. It is also the reason why science has led to so many objectively measurable advances in our understanding of the World.

I don’t have any easy answers to the question of how to cure this malaise, but do have a few suggestions. It would be easy for a scientist such as myself to blame everything on the media and the education system, but in fact I think the responsibility lies mainly with ourselves. We are usually so obsessed with our own research, and the need to publish specialist papers by the lorry-load in order to advance our own careers that we usually spend very little time explaining what we do to the public or why.

I think every working scientist in the country should be required to spend at least 10% of their time working in schools or with the general media on “outreach”, including writing blogs like this. People in my field – astronomers and cosmologists – do this quite a lot, but these are areas where the public has some empathy with what we do. If only biologists, chemists, nuclear physicists and the rest were viewed in such a friendly light. Doing this sort of thing is not easy, especially when it comes to saying something on the radio that the interviewer does not want to hear. Media training for scientists has been a welcome recent innovation for some branches of science, but most of my colleagues have never had any help at all in this direction.

The second thing that must be done is to improve the dire state of science education in schools. Over the last two decades the national curriculum for British schools has been dumbed down to the point of absurdity. Pupils that leave school at 18 having taken “Advanced Level” physics do so with no useful knowledge of physics at all, even if they have obtained the highest grade. I do not at all blame the students for this; they can only do what they are asked to do. It’s all the fault of the educationalists, who have done the best they can for a long time to convince our young people that science is too hard for them. Science can be difficult, of course, and not everyone will be able to make a career out of it. But that doesn’t mean that it should not be taught properly to those that can take it in. If some students find it is not for them, then so be it. I always wanted to be a musician, but never had the talent for it.

I realise I must sound very gloomy about this, but I do think there are good prospects that the gap between science and society may gradually be healed. The fact that the public distrust scientists leads many of them to question us, which is a very good thing. They should question us and we should be prepared to answer them. If they ask us why, we should be prepared to give reasons. If enough scientists engage in this process then what will emerge is and understanding of the enduring value of science. I don’t just mean through the DVD players and computer games science has given us, but through its cultural impact. It is part of human nature to question our place in the Universe, so science is part of what we are. It gives us purpose. But it also shows us a way of living our lives. Except for a few individuals, the scientific community is tolerant, open, internationally-minded, and imbued with a philosophy of cooperation. It values reason and looks to the future rather than the past. Like anyone else, scientists will always make mistakes, but we can always learn from them. The logic of science may not be infallible, but it’s probably the best logic there is in a world so filled with uncertainty.




16 Responses to “Bad Statistics, Bad Science”

  1. Reblogged this on Marty McFly and commented:
    Wise words 🙂

  2. Reblogged this on James's thinking space and commented:
    Science is a process, not a method for establishing ‘truth’. Quite.

  3. Peter Main Says:

    Well, many of the problems raised are certainly true but just to clarify a few things. First, the numbers of people doing chemistry and physics at A-level and university are higher than a decade ago. Physics has gone up by about 25% at A-level and 50% at university. Second, there is no evidence to suggest that sending all researchers into schools would be a good thing. Communicating with school age children is tough and not everyone is able to do it – what’s more, if it is done badly is is likely and probably does, serve to reduce the numbers taking physics and the other subjects. Third, research suggests that the reasons why people do or do not choose to study the sciences are not really connected with a lack of interest.

    But I do agree on the “priesthood” argument and that scientists are partly to blame for most of that impression. But it is also the fault of science documentaries, popular books and a whole lot more. There are many exceptions but the broad line of “this is the way that it is and only very clever people like me can properly understand it” is how it is: I liken it to the situation before the English revolution, when only priests were allowed to read the Bible.

    What to do? There is no simple answer but I am absolutely sure that more researchers going into schools is not the solution. Perhaps we move towards a baccalaureate system where science becomes more a part of everyday discourse. Perhaps we become properly serious about science communication. In schools, we can try to change the way that the sciences are presented, with a greater emphasis on the process rather than the accumulation of facts.

    The requirement is for a cultural change (e.g. I would bet there are more MPs that believe in homeopathy than have a science degree) and that is slow. But there is another point: given that science is funded and is so successful at changing the world, why do we worry so much if some people are antipathetic? All the surveys show that the majority of the population value and respect the sciences.

    As a final, mischievous point, I am amazed that so many of (we) scientists complain that people do not understand the way we think and the importance of evidence and doubt but when it comes to people, we ignore the evidence from the social sciences and psychology and carry on doing what our prejudices tell us is right. Should we not be embarrassed?

  4. Phillip Helbig Says:

    I agree in general, as almost always with your posts, of course.

    You seem to imply that there is a lack of blogs by people in other branches of science. I think there are more than enough. Just in my own RSS feed I have at least one by a biologist, a climate scientist, a mathematician, a chemist, a nuclear physicist, etc.

  5. Phillip Helbig Says:

    What has been the most visible science-related thing in cyberspace recently? Probably the fiasco involving Tim Hunt.

    I don’t claim to know all the facts, but at the very least it seems clear that a) he did not “rant for several minutes” but rather the objectionable stuff was just a couple of sentences, b) it was a (perhaps ill-executed) joke, and c) the prime witness appears to have Mark Brake as her role model in some respects, so might not be a completely reliable source.

    This has been one of the most obvious examples of trigger-happy scientists recently. It would be nice if Hunt’s critics owned up to the fact that they might have over-reacted and/or reacted too quickly. It’s sort of like BICEP2, actually: there was never any sound evidence, yet many people behaved as if they was. In both cases, many of these had axes to grind.

    What happened to Bayesian reasoning? Had his remarks been off-the-record at some other conference after a few beers, I might be inclined to believe them. However, they were part of his presentation at a conference which explicitly dealt with women in science, and he was asked to speak on the topic. It is difficult for me to believe that all of Hunt’s critics really think that even a male-chauvinist pig could be that stupid, i.e. to make such remarks not in jest.

    Jokes don’t always work—just ask any comedian. Some aren’t always appropriate. But it does seem that the reaction was too harsh. Note that many women have come out to defend Hunt, whereas as far as I know none has said “Hunt didn’t hire me but hired a less qualified man instead”.

    I think we should be at least as careful in terms of making evidence-based decisions with respect to colleagues as we should be when interpreting observational data about the universe.

    Science blogger Phil Plait has stooped really low, caricaturing Hunt’s supporters to be all right-wing and/or incompetent and/or fans of the Daily Mail and discrediting people who support Hunt on the grounds that they are otherwise not reliable—but he thinks it is bad to criticize St. Louis for not having been honest about her qualifications. Double standard?

    The faster-than-light–neutrino people admitted that it was a loose cable. Maybe some of Hunt’s detractors should admit that stripping people of (even unpaid) positions as the result of a tweet from an otherwise questionable journalist is not rigorous enough.

    I don’t think that any otherwise sensible person thinks that the reaction was inappropriate if the initial reports had been true. That is not the point. The point is that they should have been checked before acting on them.

    The alternative is that anyone can make any claim, tweet it, and the arbitrary opponent loses his job, or life, because of something which later turns out to be false. But by that time so many people have lost their faces while jumping on the bandwagon that reinstatement is not possible, much less bringing someone back to life.

    • Phillip Helbig Says:

      Just to be clear: I am not trying to defend Hunt. What I want to criticize is the fact that many in the blogosphere are properly sceptical about claims they don’t like, but too quick to accept, on flimsy evidence, claims they do like. Maybe it turns out that he really did mean it. But next time, the tweet might be completely false, so the reaction was still too quick, even if it turns out to be justified. (The fact remains, though, that the original claim that he went on a several-long-minute chauvinist rant is wrong.)

  6. Phillip Helbig Says:

    The Nature article makes an interesting demand: “Academic metrics need to be devised that distinguish citations of discredited claims so that it is not more advantageous to state and retract a result than to make a solid discovery.”

    I recommend reading the article and the comments.

    (In an interesting twist of fate, just so happens that I know the author.)

  7. Anton Garrett Says:

    It’s a bit much of Nature to grumble about physicists when so many drug efficacy experiments prove not to be replicable.

    I can see both sides of the “radio interview” problem.

    I certainly agree that Hawking’s use of the phrase “the mind of God” at the end of Brief History of Time was ill-advised. He seemed to use it simply as a metaphor for the laws of physics – certainly he’s not religious – and it is unpalatable to theists and atheists alike (albeit for differing reasons). Too bad it caught on.

    Generalising the last subject, I recently saw and much recommend the new feature-length documentary “Going Clear” about scientology.

    • Phillip Helbig Says:

      Another grungeworthy meme is “God particle”. What rubbish.

      Any truth to the rumour that this is a euphemism for “God-damn particle”?

      • Anton Garrett Says:

        I groaned when nontheist Paul Davies, a decent enough physicist, followed up Hawking’s phrase by writing a science-and-religion book called “The mind of God”.

  8. Hmm. Only 4 examples are cited in the Nature article. As a percentage of published work, that’s pretty small (in comparison with bio-medical studies for example) – hardly evidence of a ‘slew’. A second point is that none of these studies were regarded by any scientist I know as likely to be true at the time of publication, and were soon found wanting. Agreed, they were reported with great fanfare by science journalists as a’good story’ , but they were not accepted by the establishment. So perhaps we simply need to consider how results are communicated to the media…

    • Phillip Helbig Says:

      ” Only 4 examples are cited in the Nature article. As a percentage of published work, that’s pretty small”

      The piece says “The list includes faster-than-light neutrinos; dark-matter particles producing γ-rays; X-rays scattering off nuclei underground; and even evidence in the cosmic microwave background for gravitational waves caused by the rapid inflation of the early Universe.” My emphasis; it is clear that the 4 are just examples. Prominent ones, to be sure, but just examples. I’m sure that there are more.

      Yes, they were greeted sceptically by the community and were more journalistic hype than the community being too accepting. However, because they were prominent, they were checked by others. Less prominent papers are not checked in the same detail.

      I’ve written a few papers over the years which point out that some other paper is wrong. No-one has ever shown that my debunking was wrong, but as Jan points out, in terms of citations, a sensationalist but wrong and later retracted claim will get more than a solid, but less sensational, discovery. Those pointing out that the original claim was wrong will probably get even less.

      With BICEP2, there was so much buzz about it that the papers shooting it down couldn’t be ignored. But for less sensational but still important stuff, many people probably aren’t aware of claims which have been debunked and still cite the original stuff.

  9. Much of that comment may well be true, Philip, but note that your comment begins and ends with the ‘and that’s just the ones we know about’ argument. It’s this part of the argument that I’d like to see more evidence for., i.e., are these the exceptions rather thn the rule? It’s interesting that in at least 2 of the cases cited, many of the authors themselves were not convinced by the results, but wanted to show them to the community…

    • Phillip Helbig Says:

      Most papers are not wrong. So, yes, these examples are exceptions and not the rule.

      How many papers are wrong? I don’t know. The only way to find out is to check them thoroughly, perhaps try to reproduce them in their entirety. This is rarely done. Unfortunately, debunking a paper is not very well honoured, and a confirmation probably wouldn’t be published at all.

      However, it is definitely not the case that one can publish just any old rubbish.

      There are certainly negative aspects to “impact”, and pressure to have “impact” is probably what caused the BICEP2 people to jump the smoking gun, to mix a metaphor or two. The faster-than-light–neutrino stuff, however, is different. They really didn’t understand what was going on, and a straightforward interpretation led to the faster-than-light result. However, no-one in the collaboration really believed it, or said they did. This was a case of “we’re stuck; maybe someone can help us out”, which is why they went public.

  10. Reblogged this on SENSE: Science Education as a Non-Sighted Experience and commented:
    I wish I could communicate my thoughts on science as good as the post I came across not long ago does. Though the issue is not related to accessibility of science, I couldn’t possibly find better words why we need to do science and boost its “popularity” at all levels.

  11. Yes, agreed, there’s quite a difference between the neutrino and BICEP 2 cases.
    The expression ‘god particle’ comes from the particle physicist Leon Lederman, who wanted to call it ‘that goddamn particle’ but his US publisher advised against it. I think it’s quite a harmless term if a small g is used, as he intended originally…

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: