Archive for February, 2009

False Positives

Posted in Bad Statistics with tags , , on February 27, 2009 by telescoper

There was an interesting article on the BBC website this week that, for once, contains an example of a reasonable discussion of statistics in the mass media. I’m indebted to my friend Anton for pointing it out to me. I’ve filed it along with examples of Bad Statistics because the issue is usually poorly explained. I don’t think the article itself is bad. In fact, it’s rather good.

The question is all about cancer screening, specifically for breast cancer, but the lesson could apply to a host of other situations. In the original context, the question goes as follows:

Say that routine screening is 90% accurate. Say you have a positive test. What’s the chance that your positive test is accurate and you really have cancer?

Presumably there will be many of you that think the answer is 90%. Hands up if you think this!

If you don’t think it’s 90% then what do you think it is?

The correct answer is that you have no idea. I haven’t given you enough information.

To see why, imagine that the prevalence of cancer in the population is such that 1% of a randomly selected sample will have it. Out of a thousand people one would expect that, on average, ten would have cancer. If the test is 90% accurate then 9 of these will show positive signs and only one won’t.

However, 990 people out of the original thousand don’t have cancer. If the test is only 90% accurate then 10%, i.e. 99 of these will show a false positive.

Thus the total number of positive tests is 108 and only 9 of the individuals concerned actually have cancer. The odds are therefore 9/108. That’s only about a 1-in-12 chance that you have cancer.

But that depends on my assumption about the overall rate in the population. If that number is different it changes the odds. Without this information, the problem is ill-posed.

The more general way of looking at this is in terms of conditional probabilities. What you are given is that P(positive test| cancer)=P(+|C)=0.9 and P(negative test|no cancer)=0.9, while P(negative test|cancer)= 0.1 and P(positive test|no cancer)=P(+|N)=0.1. What you want to know is P(cancer|positive test)=P(C|+). This can be obtained from Bayes’ Theorem but only if you know P(cancer)=P(C)=1-P(N), since people either have cancer or they don’t.

The answer is given by P(C|+)=P(C)P(+|C)/[P(C)P(+|C)+P(N)P(+|N)], which for the numbers I gave above= 0.01 x 0.9/[0.01 x 0.9 + 0.99 x 0.1]=0.009/[0.009+0.099], which gives the same answer as before.

So the moral is that you shouldn’t panic if you get a positive test from a screening test of this type. As long as the condition being tested is relatively rarer than the likelihood of an error in the test result then the chances are high that you’ve got nothing to worry about. But of course, you should take more detailed tests.

The Bayesian way is the easy way!

A Welsh Affair

Posted in Science Politics with tags , , , , on February 26, 2009 by telescoper

Today I had the “pleasure” of attending a day-long conference called Funding, Risk and Innovation: Wales’s Engagement with Science Policy organized by the Institute of Welsh Affairs and sponsored by, amongst other organizations, the Institute of Physics. I had hoped that this would give me an insight into the landscape of Welsh science politics which might bear fruit in the future. As if.

Unfortunately, but alas predictably, there wasn’t much of interest. I think the first presentation of the day perfectly  illustrated the whole problem. Opening the batting was Ieuan Wyn Jones, Deputy First Minister and Minister for Economic Development, from the Welsh Assembly Government or WAG.  He gave a motherhood-and-apple-pie talk about how important science was to the future of Wales, took a few questions and then left. Those of us scientists who had gone to the meeting hoping for engagement between  politicians and scientists were left to discuss things between ourselves. Hardly the point.

Next was Phil Gummett, Chief Executive of HEFCW who gave the results of the latest Research Assessment Exercise (which I’ve blogged about here, there and everywhere). To my dismay he announced that HEFCW are indeed going to use the 0:1:3:7 weighting formula adopted by HEFCE, but has found a bit more cash which it will add to the pot of money allocated to 4*. However, unlike in the case of English universities, HEFCW is not going to apply any protection to STEM subjects (Science, Technology & Medicine). In the case of my own department at Cardiff University, which got a very low assessment  of 4* research, this is very bad news.

When I got home this evening I read the same news in the Times Higher. I could have found this out without wasting a day sitting  in a ghastly conference room in the soulless Cardiff Novotel. Still, the lunch wasn’t bad.

Phil Gummett struck me as quite a reasonable chap who is trying to do the right thing, but whose hands are tied by the Welsh Assembly which has decided that Higher Education in Wales is not as high a priority as Further Education, with the result that the funds available to HEFCW for research is less than it would be for English universities. University STEM departments in Wales altogether receive about £10M less from HEFCW than they would get from HEFCE if they were in England. For physics, this will probably get worse after the 2008 RAE.

The reason for this pessimism is that, as I’ve noted before, Physics did rather badly in the RAE compared to other discplines, with a significantly lower fraction of work assessed at 4* (world-leading). Since the funding formula is heavily weighted by the 4* element, physics will suffer relative to other disciplines. HEFCW will not attempt to correct this. I think the Chair of the Physics panel, Sir John Pendry, must shoulder at least some of the blame for the gross anomaly that this represents.

It remains to be seen what happens to physics nationally, but I fear the RAE may undo a decade of very effective positive campaigning about the importance of physics. I’ve already heard from various Heads of Physics departments around the country (even those who have done well in the RAE)  who have been asked by their Vice-Chancellors why they have done so much worse than other disciplines.

The final thing he said was that HEFCW would make its allocations as block grants to the Universities concerned and that they should make their own decisions as to how to allocate the funds to the departments. This sort of thing always annoys me. It’s admitting that the formula is probably stupid, so passing the buck to the institutions to sort out the mess themselves.

I spoke to a nice lady from Cardiff University’s planning department in the afternoon who said that they weren’t sure how they were going to allocate funds to Schools after the HEFCW grant was announced, and that the University as a whole was probably going to lose out in research funds, despite having 54% of all the 4* research in Wales.

The big problem is the funding gap caused by the WAG’s policy. Devolution has had a negative effect in science funding in Wales, while in Scotland it has had the opposite effect. The Scottish parliament seems much more interested in science than does the Welsh Assembly. Indeed, per capita, Scottish Universities have a much heavier level of research investment even than those in England, which in turn are much higher than in Wales.

EPSRC‘ recently allocated £82M to UK universities to fund  doctoral training centres. In all, it allocated grants to 45 universities. Wales has 5% of the UK population, but not a single grant went to a Welsh university. Of the 1200 or so students these centres will train, not a single one will be in Wales.  I can’t believe the Scottish assembly would have let such an outcome happen in Scotland.

Further strangulation of research funds is inevitable unless the WAG is persuaded to change its mind about the importance of science. But if the politicians don’t stay to listen to the arguments, how will this happen?

Over lunch I chatted to various physicists from Swansea University. Several of them had come to the meeting, but I was the only representative from Cardiff. There was a strong steer from the RAE panel for physics in terms of closer collaboration so we chatted a bit about possibilities for that. I think the consensus was that we’re probably going to be bounced down the road anyway so the best way forward would be to come up with a plan of our own instead of having someone else’s.

I promise not to mention the RAE again, until the final allocations are published in April!

Blackbird has Spoken

Posted in Biographical, Poetry with tags , , , on February 25, 2009 by telescoper

Over the last few days we’ve been having something approximating springtime here in Cardiff. It has been sunny and quite warm, my garden has started to come to life, and the crocuses have appeared in Bute Park. It’s also getting to the time when I won’t feel guilty for walking home in daylight. Soon I’ll even be able to walk home through Bute Park, which closes when it gets dark, currently at 5.15.

I hope this all continues into a pleasant spring and summer, without the heavy continuous rain we had last year. I’m not betting on it though.

However, the clement weather has given me one headache recently. With sunrise happening a bit earlier and the good weather giving the local wildlife something to shout about, the dawn chorus has been waking me up around 4am.

Or, actually, it’s not so much a chorus as a solo. A very loud blackbird has taken to sitting right next to my bedroom window and singing at the top of its voice.

I’m very fond of blackbirds. Once while I was in the garden in my old house in Beeston, a blackbird flew onto a fence post about a yard away from me and sat there looking at me as I stood with a spade in my hand. I looked back. We looked at each other for ages, the blackbird turning its head every now and again so as to peer at me with a different eye. I slowly raised my arm and extended a palm. To my absolute delight the bird hopped onto my open hand. It stayed there only a minute or so, probably until it realised my fingers weren’t actually big fat worms like it thought. For that moment, though, I felt a bit like a latter-day St Francis of Assisi.

Blackbirds have a very attractive song, but this one seems particularly loud and he certainly does go on a bit. For about a week now I’ve been unable to get back to sleep after being woken by this critter, and instead got up and had a cup of tea while he says what he has to say. Columbo finds his song quite interesting too, although the bird is always out of reach…

Years ago, I used to suffer very badly from insomnia so being awake at 4am is not an unfamiliar experience to me, although it’s much nicer to be woken by birdsong than to be unable to sleep in the first place. This all reminded me of a devastatingly brilliant poem called Aubade and written by Philip Larkin that was published in the Times Literary Supplement in 1977. This is one of the last poems written by Larkin, and is undoubtedly one of the greatest.

Written by a jazz-loving bachelor who drank too much, someone not unlike myself in some respects, I found it uncanningly accurate in its depiction of the bleak thoughts that tend to engulf you when you’re alone and awake in the silence before dawn. But I can assure you the mood is a whole lot lighter when you have a blackbird (and a cat) for company!

I work all day, and get half drunk at night.
Waking at four to soundless dark, I stare.
In time the curtain edges will grow light.
Till then I see what’s really always there:
Unresting death, a whole day nearer now,
Making all thought impossible but how
And where and when I shall myself die.
Arid interrogation: yet the dread
Of dying, and being dead,
Flashes afresh to hold and horrify.

The mind blanks at the glare. Not in remorse
– The good not used, the love not given, time
Torn off unused – nor wretchedly because
An only life can take so long to climb
Clear of its wrong beginnings, and may never:
But at the total emptiness forever,
The sure extinction that we travel to
And shall be lost in always. Not to be here,
Not to be anywhere,
And soon; nothing more terrible, nothing more true.

This is a special way of being afraid
No trick dispels. Religion used to try,
That vast moth-eaten musical brocade
Created to pretend we never die,
And specious stuff that says no rational being
Can fear a thing it cannot feel, not seeing
that this is what we fear – no sight, no sound,
No touch or taste or smell, nothing to think with,
Nothing to love or link with,
The anaesthetic from which none come round.

And so it stays just on the edge of vision,
A small unfocused blur, a standing chill
That slows each impulse down to indecision
Most things may never happen: this one will,
And realisation of it rages out
In furnace fear when we are caught without
People or drink. Courage is no good:
It means not scaring others. Being brave
Lets no-one off the grave.
Death is no different whined at than withstood.

Slowly light strengthens, and the room takes shape.
It stands plain as a wardrobe, what we know,
Have always known, know that we can’t escape
Yet can’t accept. One side will have to go.
Meanwhile telephones crouch, getting ready to ring
In locked-up offices, and all the uncaring
Intricate rented world begins to rouse.
The sky is white as clay, with no sun.
Work has to be done.
Postmen like doctors go from house to house.

Fat Tuesday

Posted in Jazz with tags , on February 24, 2009 by telescoper

Today’s  the day we call in England  Shrove Tuesday. We’re apparently all supposed to get shriven by doing a pennance before Lent . Another name for the occasion is Pancake Day, although I’m not sure what sort of pennance it is to be forced to eat pancakes.

Further afield the name for this day is a bit more glamorous. Mardi Gras, which I translated for the title using my schoolboy French, doesn’t make me think of pancakes but of carnivals. And being brought up in a house surrounded by Jazz, it makes me think of New Orleans and the wonderful marching bands that played not just during the Mardi Gras parades but at  just about every occasion for which they could find an excuse, including funerals.

The Mardi Gras parades gave rise to many of the great tunes of New Orleans Jazz, many of them named after the streets through which the parade would travel, mainly in  the famous French Quarter. Basin Street, South Rampart Street, and Bourbon Street are among the names redolent with history for Jazz fans and musicians around the world. I also remember a record by Humphrey Lyttelton‘s 1950s band called Fat Tuesday.

The New Orleans Mardi Gras has on recent occasions sometimes got a bit out of hand, and you probably wouldn’t want to take kids into the French Quarter for fear they would see things they shouldn’t. Personally, though, I’d love the chance to savour the atmosphere and watch the parades.

The  clip I’ve chosen is of Bourbon Street Parade. The one and only time I went to New Orleans I felt a real thrill walking along this street, just because I’ve heard the tune so many times on old records.  I didn’t go in Mardi Gras time, however, but in the middle of summer. The heat was sweltering and the humidity almost unbearable, but the air was filled with music as well as moisture. It was impossible to sleep in the heat, so I stayed up moving from bar to bar, drinking and listening to music until I was completely exhausted.

The tune was written by the late Paul Barbarin, who died in 1969 during a street parade in New Orleans. What a way to go. He also plays on the clip I included here.

I picked this particular clip because it features a much underrated British musician, Sammy Rimmington (although the notes on Youtube have muddled it up; he plays saxophone on this, not clarinet). My dad once played with Sammy Rimmington and I remember the unqualified admiration with which he (my dad) spoke of his (Sammy’s) playing.

The Problem of the Steady State

Posted in Science Politics with tags , , on February 24, 2009 by telescoper

Just as a quick postscript to my recent item about proposed changes to the method of funding PhD students by STFC, let me point out the following simple calculation.

Assume that the number of permanent academic positions in a given field (e.g. astronomy) remains constant over time. If that is the case, each retirement (or other form of departure) from a permanent position will be replaced by one, presumably junior, scientist.

This means that over an academic career, on average, each academic will produce just one PhD who will get a permanent job. This of course doesn’t count students coming in from abroad, or those getting faculty positions abroad but in the case of the UK these are probably relatively small corrections.

Under the present supply of PhD studentships an academic can expect to get a PhD student at least once every three years or so. At a minimum, therefore, over a 30 year career one can expect to have ten PhD students. A great many supervisors have more PhD students than this, but this just makes the odds worse. The expectation is that only one of these will get a permanent job in the UK. The others (nine out of ten, according to my conservative estimate) above must either leave the field or the country to find permanent employment.

The arithmetic of this situation is a simple fact of life, but I’m not sure how many prospective PhD students are aware of it.

Scientiae Doctores

Posted in Science Politics with tags , , , on February 22, 2009 by telescoper

The season for recruiting new research students is well and truly upon us and at the same the Science & Technology Facilities Council (STFC) is consulting about changing the way that it allocates PhD studentships to departments.

Most postgraduate students studying for PhDs in Astronomy are funded by STFC (although some Universities also fund their own internal studentships). The result of this arrangement is that successful applicants to a PhD course can receive a stipend which amounts to about £13K per annum. It’s not a huge amount of money, but it is a stipend rather than a salary so it’s tax-free. Since a PhD student also remains a student and therefore qualifies for various other fringe benefits (Council Tax, student discounts, etc), it’s not actually a bad deal for the student. Anyway, if it were significantly more then it’s possible PhD students would have to start paying back their student loans, which would make things worse. STFC also pays a tuition fee to the University concerned, but this is done directly and the student doesn’t even see that element of the funding.

Since about 1995, PPARC and then STFC has funded research studentships in areas within its remit by means of peer review. Departments have bid for studentships (every two years) and a panel awards an allocation depending on the quality of the bid. Of course, everyone asks for many more studentships than are available so what you get is a fraction of what you ask for. I wrote the application for the first ever quota studentships for the Astronomy group at the University of Nottingham, and did it again a couple of times after that. Each time, despite going into best bullshit mode to write the case, I was frustrated by the relatively small number of studentships we were awarded. Although we succeeded in building up gradually from zero to 2-3 per year, it was a very slow process.

In recent years, the funding mechanism has evolved slightly so that studentship fees and stipends were devolved to the departments concerned in terms of Doctoral Training Grants (DTGs) rather than being administered centrally by PPARC/STFC. In the old days, students used to get their stipend from PPARC/STFC whereas now they are paid by their department from a cash grant.

Anyway, for various reasons (chief among them being no doubt to save administrative costs) STFC has decided to consult on changes to the mechanism for allocating the DTGs to the various departments around the country. The most serious proposed change is to follow the practice at the Engineering and Physical Sciences Research Council (EPSRC) and dispense with peer review. Instead, the proposal is to award studentships based on a formula involving how successful the department is at obtaining postdoctoral research assistant (PDRA) support from STFC.

Here is the proposed formula:

 Specifically, the studentship award per department should be proportional to the product of volume and average quality per academic within the department, that is to:

 

V * Q

 

The Committee has followed guidance in developing measures of V and Q that are non-subjective, repeatable and transparent.  The volume V is defined as the number of academics (including Fellows) eligible to hold STFC research grants. The  quality Q is measured by the number of STFC-funded PDRAs (P) awarded per academic (i.e. P/V), since this measures the success of the academic staff in securing STFC funding for PDRAs through peer-review.  More precisely we define quality per academic as Q = [1 +(P/V)].

 

Although the Committee felt this definition of quality applied primarily to responsive-mode PDRAs, it agreed that PDRAs on project grants should be included, but with a weighting, relative to responsive-mode, of 0.33.

 

Using these definitions, the Committee recommends that the studentship award per department should be proportional to a simple product of volume and average quality per academic within the department, that is to:

 

N(students) µ V * Q

 

where Q = [1 + (P/V)]

 

And so the departmental quota is proportional to: 

 

V[1+(P/V)] = V+P

 

In addition, recognising that very small departments offer more limited training opportunities on their own, a threshold is proposed, such that no studentships are awarded for V < 3. Instead, these very small departments/groups would be able to collaborate with other larger departments in seeking STFC studentship support.

 

Hence

 

         N(students) µ V+P  for V ³ 3

                             = 0        for V < 3

 

The constant of proportionality is chosen such that the total number of studentships equals the number available for allocation.

 

 

I think this is a fairly reasonable proposal, actually. The one thing I don’t really understand relates to the fact that STFC doesn’t just fund PDRAs on its grants, but under the Full Economic Cost regime (FEC), it also pays for fractions of academic staff effort for people working on its projects. On my recent successful STFC grant, for example, I was awarded 25% of my time (i.e. 0.25 FTE, full-time-equivalent) to do the research as well as a PDRA. Since the proposal above will have to cope with the question of what staff are “eligible” then why not make the quantity V proportional to the total FTEs funded, or at least only count those for whom some FEC time is allocated? And why not include staff FTE in the Q-factor too?

My guess is that such a modification wouldn’t make much difference to astronomy departments, but the original proposal has caused cries of anguish from particle physicists. This is because the number of PDRAs in particle physics is much smaller than in astronomy, so many large groups face a big reduction in their PhD quota. Including FEC numbers in the mix might well smooth the transition for them. For your information, the number of PDRAs per active astronomy researcher  is around 0.5 at present.

Anyway, the deadline for consulting on this has passed (on February 20th) so we now wait to see what STFC actually does. Probably the consultation period is a purely cosmetic exercise anyway and what will emerge is exactly what was proposed.

If you ask me (and nobody did), all this is mere tinkering. I think there are serious problems with graduate funding in the UK and these require much more radical remedies. At the risk of (and indeed with the intention of) being provocative, here is my diagnosis and suggested remedies:

  • There are too many PhDs in astronomy. STFC funded 160 studentships in 2006, compared with 88 in 2000. There are nowhere near enough PDRA positions to accommodate this number of PhDs in academic research. And even those who get their first PDRA position have very limited prospects of getting a permanent job. The result is a generation of disaffected students employed as low-paid assistants for 3-4 years and then thrown aside when they have got their PhD.
  • Of course, applicants for PhD places don’t know what research is really like and some will leave academia of their own volition when they find out that it’s not for them. In my experience, though, most graduate applicants simply don’t realise how heavily the odds are stacked against them. Less than one in ten can possibly stay in research in the long term, and the more PhDs are funded the worse the odds against them become.
  • The short duration of a British PhD disadvantages our students with respect to those from the USA or continental europe, who all do a lengthy Masters course before taking their PhD. These take at least 5 years to complete.  The result is that our home-grown PhDs are seriously disadvantaged in the job market against competitors from abroad. Similar points have been made forcefully by Ian Halliday.
  • My remedy is simple. Reduce the number of studentships but extend each one to five years and require each hosting department to provide a proper graduate school with intensive graduate-level courses to make up for the progressive reduction in content of undergraduate physics courses.
  • Even more unpopularly, I think the UK should scrap 4 years Masters (MPhys) programmes and embrace the structure of the Bologna agreement, i.e. a universal 3+2+3 structure of 3 years Bachelors, 2 years’ Masters and three years PhD.
  • Currently STFC stipends can only be paid to UK nationals and residents. It’s an open secret that most departments would preferentially recruit European physics graduates to their PhD positions if they were allowed to do so, because their undergraduate preparation is much better than that provided in UK universities. I propose that we abandon this protectionism and open up PhD opportunities to European applications, just as we would legally have to do if a PhD were considered to be a job.
  • Finally, I think the UK should consider the introduction of a common graduate entrance examination, perhaps based on the US GRE, to ensure the maintenance of appropriate standards for postgraduate entry and eligibility for STFC funding.

There are of course some advantages to the current British PhD system. For one thing, the PhD is earned very quickly. I was 25 when I got my PhD, and already had several publications. Most of my European collaborators were at least 30 before they got theirs (additional years have to be added for national service in many countries, but we don’t have it in the UK). But I am painfully aware that my technical knowledge outside the immediate area of my PhD is much thinner than most academics in the field. Now, in middle age, I feel like a long-distance runner who had inadequate preparation, went off too fast at the start of the race, and is now struggling along while people overtake him with monotonous regularity.

The nature of research in astronomy and cosmology has changed so much in the 20 years since I got my PhD that the old system has to go. Instead of tinkering with funding formula, driven principally by the need to save adminstrative costs within STFC, we need a radical overhaul of the entire graduate education system in the UK, involving all research councils and their political masters.

Unfortunately, though, for the time being at least the politicians have other more pressing matters to worry about, such the collapse of the economy.

Anthropology

Posted in Jazz with tags , , on February 21, 2009 by telescoper

Just to balance the books here’s a wonderful version of the classic bebop tune Anthropology, performed by Bud Powell who I mentioned in my previous post about Thelonious Monk.

This really is authentic bebop, with its complex yet propulsive drum patterns and jagged melodic lines characterized by unusual intervals and little punctuating tags at the end of each phrase in the solo. Also brilliant, but quite different to Monk, Bud Powell was very much a direct translation onto the keyboard of the saxophone and trumpet styles of Charlie Parker and Dizzy Gillespie.

The tune was written by them too, although it uses essentially the same chords as George Gershwin’s I Got Rhythm. It  was a standard bebop procedure to write a new melodic line on top of an existing harmonic structure and give the result a new name. Jazz improvisation is always based on the harmonies (chord changes) rather than the tune, so this is a way to build new compositions quickly with the knowledge that the harmonic foundations would be sound. In the  40s there was a classic Charlie Parker recording session during which the band decided to do such a variation of the standard Cherokee (written by Ray Noble). In the middle of the performance they absent-mindedly played the Cherokee theme and there was a cry of anguish from the control room by the Producer who obviously hoped that if they stayed off the actual tune of Cherokee he wouldn’t have to pay composer’s royalties. So that take was abandoned, and they did another. The final version was called Koko and it’s one of the Charlie Parker classics. Many other classic bop tunes were made in this way, with different standard tunes underneath them. Necessity and budget restrictions are the mothers of invention.

Oh, and just listen to the fantastic double bass by the brilliant (and then very young) Niels-Henning Orsted Pedersen who drives it along like the clappers!

When you read about the structure of bebop it seems so complicated and obscure that it’s hard to believe that the result is such a thrill to listen to. I think it  is the highest point of twentieth century music-making, both in the creativity and skill of its proponents and in the sheer excitement of its sound which is made all the more remarkable when you realise how difficult it is to play!