Politics, Polls and Insignificance

In between various tasks I had a look at the news and saw a story about opinion polls that encouraged me to make another quick contribution to my bad statistics folder.

The piece concerned (in the Independent) includes the following statement:

A ComRes survey for The Independent shows that the Conservatives have dropped to 27 per cent, their lowest in a poll for this newspaper since the 2010 election. The party is down three points on last month, while Labour, now on 33 per cent, is up one point. Ukip is down one point to 17 per cent, with the Liberal Democrats up one point to eight per cent and the Green Party up two points to seven per cent.

The link added to ComRes is mine; the full survey can be found here. Unfortunately, the report, as is sadly almost always the case in surveys of this kind, neglects any mention of the statistical uncertainty in the poll. In fact the last point is based on a telephone poll of a sample of just 1001 respondents. Suppose the fraction of the population having the intention to vote for a particular party is p. For a sample of size n with x respondents indicating that they hen one can straightforwardly estimate p \simeq x/n. So far so good, as long as there is no bias induced by the form of the question asked nor in the selection of the sample, which for a telephone poll is doubtful.

A  little bit of mathematics involving the binomial distribution yields an answer for the uncertainty in this estimate of p in terms of the sampling error:

\sigma = \sqrt{\frac{p(1-p)}{n}}

For the sample size given, and a value p \simeq 0.33 this amounts to a standard error of about 1.5%. About 95% of samples drawn from a population in which the true fraction is p will yield an estimate within p \pm 2\sigma, i.e. within about 3% of the true figure. In other words the typical variation between two samples drawn from the same underlying population is about 3%.

If you don’t believe my calculation then you could use ComRes’ own “margin of error calculator“. The UK electorate as of 2012 numbered 46,353,900 and a sample size of 1001 returns a margin of error of 3.1%. This figure is not quoted in the report however.

Looking at the figures quoted in the report will tell you that all of the changes reported since last month’s poll are within the sampling uncertainty and are therefore consistent with no change at all in underlying voting intentions over this period.

A summary of the report posted elsewhere states:

A ComRes survey for the Independent shows that Labour have jumped one point to 33 per cent in opinion ratings, with the Conservatives dropping to 27 per cent – their lowest support since the 2010 election.

No! There’s no evidence of support for Labour having “jumped one point”, even if you could describe such a marginal change as a “jump” in the first place.

Statistical illiteracy is as widespread amongst politicians as it is amongst journalists, but the fact that silly reports like this are commonplace doesn’t make them any less annoying. After all, the idea of sampling uncertainty isn’t all that difficult to understand. Is it?

And with so many more important things going on in the world that deserve better press coverage than they are getting, why does a “quality” newspaper waste its valuable column inches on this sort of twaddle?

5 Responses to “Politics, Polls and Insignificance”

  1. Anton Garrett Says:

    “why does a “quality” newspaper waste its valuable column inches on this sort of twaddle?”

    It’s every editor’s favourite reporter Phil Space again.

  2. George Jones Says:

    Maybe a little off-topic, but potentially more serious: it seems that many medical doctors are statistically illiterate,


  3. Andrew Liddle Says:

    You say `There’s no evidence of support for Labour having “jumped one point” ‘, but are you sure? The one-point jump in Labour’s support is the most likely explanation for the two polls, even if there is indeed quite a substantial probability that there was no jump. There’s also quite a substantial probability that the jump was *more than* one percent; indeed this is more probable than that there has been no gain.

    Since you are a committed Bayesian I don’t think you can say there is no evidence at all, as you then wouldn’t be able to deal with the cumulative effect of several one-percent shifts all in the same direction, as may happen if there is a long-term trend in favour of Labour that pollsters are sampling on a shorter timescale.

    It’s also worth bearing in mind that in the UK voting system the number of parliamentary seats Labour wins is an extraordinarily sensitive function of the percentage of votes, so relatively small changes in the probability distribution for the fraction of the vote can lead to enormous changes in the expected number of seats won, and hence in the probability of a Labour absolute majority. It seems to me possible that a change which looks statistically insignificant in terms of vote share could be significant in terms of the changed probability of a victorious outcome (though I don’t know whether this could be true, any ideas anyone?).

    • telescoper Says:

      The last three ComRes polls for the Independent give the percentage intending to vote Labour as 35,32 and 33 (latest). I don’t think these data give any evidence in support of an increase that’s anything other than just noise. One could easily do a proper test between two hypothesis, one in which nothing changes and another in which there is a change by an amount x which a parameter to be estimated using Bayesian methods. I haven’t done this test, but it seems likely that the former model would be strongly preferred for any sensible prior on x.

      • Andrew Liddle Says:

        Mmm, certainly if you introduce the 35 datapoint there’s not going to be evidence of an upward trend. But generally speaking I’m not sure it is always true that results deemed statistically insignificant are completely uninteresting, and this might be a case due to the outcome sensitivity I mentioned above. Likewise, if you were offered a choice of two treatments for a terminal illness whose assessed difference in survival rates was statistically insignificant, I’m guessing you’d still choose the one with the higher survival rate rather than flip a coin.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: