Archive for October, 2011

Hallowe’en

Posted in Biographical, Music with tags , on October 31, 2011 by telescoper

Oh no. The plethora of pumpkins staring at me through  my office window tells me that the  dreaded Hallowe’en is upon me again.

In fact it has been upon me for three nights running already. My enjoyment of a pre-concert drink in the Poet’s Corner on Friday was interrupted by an invasion of costumed children all demanding money with menaces. I had similar encounters at my own doorstep on Saturday and Sunday too. Excuse me, but Hallowe’en is on October 31st. Not the day before. Or the day before that. Go away.

Hopefully, I won’t be in tonight to be pestered by any more little horrors, but if I am I’ll offer them this treat from the strange and wonderful musical world of one of my favourite composers, Charles Ives. That should scare them off.

Advertisements

Advice for the REF Panels

Posted in Finance, Science Politics with tags , , , , , on October 30, 2011 by telescoper

I thought I’d post a quick follow-up to last week’s item about the Research Excellence Framework (REF). You will recall that in that post I expressed serious doubts about the ability of the REF panel members to carry out a reliable assessment of the “ouputs” being submitted to this exercise, primarily because of the scale of the task in front of them. Each will have to read hundreds of papers, many of them far outside their own area of expertise. In the hope that it’s not too late to influence their approach, I thought I’d offer a few concrete suggestions as to how things might be improved. Most of my comments refer specifically to the Physics panel, but I have a feeling the themes I’ve addressed may apply in other disciplines.

The first area of  concern relates to citations, which we are told will be used during the assesment, although we’re not told precisely how this will be done. I’ve spent a few hours over the last few days looking at the accuracy and reliability various bibliometric databases and have come to the firm conclusion that Google Scholar is by far the best, certainly better than SCOPUS or Web of Knowledge. It’s also completely free. NASA/ADS is also free, and good for astronomy, but probably less complete for the rest of physics. I therefore urge the panel to ditch its commitment to use SCOPUS and adopt Google Scholar instead.

But choosing a sensible database is only part of the solution. Can citations be used sensibly at all for recently published papers? REF submissions must have been published no earlier than 2008 and the deadline is in 2013, so the longest time any paper can have had to garner citations will be five years. I think that’s OK for papers published early in the REF window, but obviously citations for those published in 2012 or 2013 won’t be as numerous.

However, the good thing about Google Scholar (and ADS) is that they include citations from the arXiv as well as from papers already published. Important papers get cited pretty much as soon as they appear on the arXiv, so including these citations will improve the process. That’s another strong argument for using Google Scholar.

The big problem with citation information is that citation rates vary significantly from field to field sit will be very difficult to use bibliometric data in a formulaic sense, but frankly it’s the only way the panel has to assess papers that lie far from their own expertise. Unless anyone else has a suggestion?

I suspect that what some panel members will do is to look beyond the four publications to guide their assessment. They might, for example, be tempted to look up the H-index of the author if they don’t know the area very well. “I don’t really understand the paper by Professor Poindexter but he has an H-index of 95 so is obviously a good chap and his work is probably therefore world-leading”. That sort of thing.

I think this approach would be very wrong indeed. For a start, it seriously disadvantages early career researchers who haven’t had time to build up a back catalogue of high-impact papers. Secondly, and more fundamentally still, it is contrary to the stated aim of the REF, which is to assess only the research carried out in the assessment period, i.e. 2008 to 2013. The H-index would include papers going back far further than 2008.

But as I pointed out in my previous post, it’s going to be impossible for the panel to perform accurate assessments of all the papers they are given: there will just be far too many and too diverse in content. They will obviously therefore have to do something other than what the rest of the community has been told they will do. It’s a sorry state of affairs that dishonesty is built into the system, but there you go. Given that the panel will be forced to cheat, let me suggest that they at least do so fairly. Better than using the H-index of each individual, use the H-index calculated over the REF period only. That will at least ensure that only research done in the REF period will count towards the REF assessment.

Another bone of contention is the assessment of the level of contribution authors have made to each paper, in other words the question of attribution. In astronomy and particle physics, many important papers have very long author lists and may be submitted to the REF by many different authors in different institutions. We are told that what the panel will do is judge whether a given individual has made a “significant” contribution to the paper. If so, that author will be accredited with the score given to the paper. If not, the grade assigned will be the lowest and that author will get no credit at all. Under this scheme one could be an author on a 4* paper but be graded “U”.

This is fair enough, in that it will penalise the “lurkers” who have made a career by attaching their names to papers on which they have made negligible contributions. We know that such people exist. But how will the panel decide what contribution is significant and what isn’t? What is the criterion?

Take the following example. Suppose the Higgs Boson is discovered at the LHC duringthe REF period. Just about every particle physics group in the UK will have authors on the ensuing paper, but the list is likely to be immensely long and include people who performed many different roles. Who decides where to draw the line on “significance”. I really don’t know the answer to this one, but a possibility might be to found in the use of the textual commentary that accompanies the submission of a research output. At present we are told that this should be used to explain what the author’s contribution to the paper was, but as far as I’m aware there is no mechanism to stop individuals hyping up their involvement.What I mean is I don’t think the panel will check for consistency between commentaries submitted by different people for the same institution.

I’d suggest that consortia  should be required to produce a standard form of words for the textual commentary, which will be used by every individual submitting the given paper and which lists all the other individuals in the UK submitting that paper as one of their four outputs. This will require co-authors to come to an agreement about their relative contributions in advance, which will no doubt lead to a lot of argument, but it seems to me the fairest way to do it. If the collaboration does not produce such an agreement then I suggest that paper be graded “U” throughout the exercise. This idea doesn’t answer the question “what does significant mean?”, but will at least put a stop to the worst of the game-playing that plagued the previous Research Assessment Exercise.

Another aspect of this relates to a question I asked several members of the Physics panel for the 2008 Research Assessment Exercise. Suppose Professor A at Oxbridge University and Dr B from The University of Neasden are co-authors on a paper and both choose to submit it as part of the REF return. Is there a mechanism to check that the grade given to the same piece of work is the same for both institutions? I never got a satisfactory answer in advance of the RAE but afterwards it became clear that the answer was “no”. I think that’s indefensible. I’d advise the panel to identify cases where the same paper is submitted by more than one institution and ensure that the grades they give are consistent.

Finally there’s the biggest problem. What on Earth does a grade like “4* (World Leading)” mean in the first place? This is clearly crucial because almost all the QR funding (in England at any rate) will be allocated to this grade. The percentage of outputs placed in this category varied enormously from field to field in the 2008 RAE and there is very strong evidence that the Physics panel judged much more harshly than the others. I don’t know what went on behind closed doors last time but whatever it was, it turned out to be very detrimental to the health of Physics as a discipline and the low fraction of 4* grades certainly did not present a fair reflection of the UK’s international standing in this area.

Ideally the REF panel could look at papers that were awarded 4* grades last time to see how the scoring went. Unfortunately, however, the previous panel shredded all this information, in order, one suspects, to avoid legal challenges. This more than any other individual act has led to deep suspicions amongs the Physics and Astronomy community about how the exercise was run. If I were in a position of influence I would urge the panel not to destroy the evidence. Most of us are mature enough to take disappointments in good grace as long as we trust the system.  After all, we’re used to unsuccessful grant applications nowadays.

That’s about twice as much as I was planning to write so I’ll end on that, but if anyone else has concrete suggestions on how to repair the REF  please file them through the comments box. They’ll probably be ignored, but you never know. Some members of the panel might take them on board.

Ein deutsches Requiem

Posted in Music with tags , , , , , on October 29, 2011 by telescoper

Last night was time for another injection of culture, so I went again to St David’s Hall in Cardiff for a programme of music played by the Orchestra and Chorus of Welsh National Opera conducted by musical director Lothar Koenigs.

The first item on the programme was the set of five Kindertotenlieder (Songs of the Death of Children) by Gustav Mahler, settings of poems by Friedrich Rückert drawn from a huge collection of tragic verse the poet produced in reaction to the death of his children from scarlet fever. Mahler’s daughter Maria herself suffered the same fate in 1909, four years after the first performance of the Kindertotenlieder. Of course these works are immensely poignant, but the pervading atmosphere is not just of  melancholy but also of resignation. The soloist last night was mezzo soprano Sarah Connolly who gave a performance of great dignity and emotional power. She has a simply gorgeous voice, with lovely velvety chest tones as well as strength and clarity in the upper register. She looked the part too, her facial expressions adding to the sense of tragedy underlying the music. One for Mahler fans only, I suspect, but I loved it.

The next piece before the interval was quite new to me, A Survivor from Warsaw, written by Arnold Schoenberg in 1947 as a reaction to the persecution of jews in the Warsaw ghetto. In addition to the orchestra this work features a male chorus and a narrator (WNO regular David Soar) who recounts the story of a massacre in the declamatory Sprechstimme that Schoenberg used in several works. I was surprised to learn from the programme that the narration was actually written in English (as it was performed last night), but I don’t think the texture of the English language really suits this style of vocalisation. The male chorus sings a setting of the Shema Yisrael amidst sounds representing the violence of the attacking soldiers. The music is rigorously atonal: disturbing, agonized and entirely appropriate to the subject. Not exactly easy listening, but why on Earth should it be?

After the interval we heard the main piece of the evening, Ein deutsches Requiem (A German Requiem) by Johannes Brahms. Regular readers of this blog (both of them) will know that I’m not exactly a devout follower of Brahms, but he is a composer I somehow feel I ought to persevere with. The German Requiem is, like the preceding pieces, a reaction to loss; in this case it was probably the deaths of Brahms’ mother and of his friend Robert Schumman that led Brahms to compose the work. It’s not a traditional Requiem, in the sense of being a liturgical setting, but it does take its text from the scriptures. It’s also a very large work, comprising seven movements lasting well over an hour altogether, and is Brahms’ longest composition. Soloists were David Soar (bass-baritone) and Laura Mitchell (soprano); the latter wore a white dress and black shoes to the consternation of the fashion-conscious members of the audience. Apparently that’s a no-no.

This is not a work that I’m familiar with amd it’s such a long piece that it’s difficult to take it all in during one performance. Inevitably, therefore, there are parts that stand out in my memory better than others. The orchestral playing was very tight, full of colour, and never lost momentum. However, I would say that the Chorus of Welsh National Opera were absolutely magnificent; the dramatic intensity they achieved during the crescendi in the 2nd movement (Denn alles Fleisch, with text drawn from Psalm 126) definitely raised the hairs on the back of my neck. That alone was enough to make me want to listen to this again.  I’d therefore like to ask any readers of this blog please to help by suggesting good recordings of this work through the Comments box.

Here’s a version of the 2nd Movement I found on youtube, just to give you an idea of its sombre majesty, but last night’s rendition was better. Try to imagine what the crescendo that grows from about 3.00 sounds like live…

Serious Brain Teaser

Posted in Cute Problems with tags , , , on October 28, 2011 by telescoper

This one has been doing the rounds this morning, so I couldn’t resist posting it here:

PS. If anyone knows where this originated please let me know and I’ll give proper credit!

PPS. Note that the Mike Disney option (300%) is missing…

Serious Brain Power

Posted in Uncategorized with tags on October 28, 2011 by telescoper

I can’t resist doing a bit of advertising on behalf of Cardiff University’s new recruitment campaign which has the slogan Serious Brain Power.  A major initiative is under way to attract high quality researchers to Cardff (either at Chair level for established academics or at the level of a Fellowship for those earlier in their careers)  across a range of academic disciplines, including STEM subjects.

In the School of Physics & Astronomy we’ve already appointed four new lecturers in Physics over the last year, and will also be joined by a new Professor of Experimental Physics next year, all independently of this scheme, but it would be great if we could attract even more excellent new people into the School via the new initiative; for an advert see here.

At fellowship level the positions  provide a greater degree of independence than a normal postdoctoral research assistantship, including the possibility to direct one’s own research programme. The number of  similar positions funded by research councils  is  dwindling owing to cutbacks in the research council budgets, making such a post a particularly valuable and attractive proposition.

Although this is a personal blog, and therefore not officially part of the recruitment campaign, it occurred to me that readers of this blog might well be interested in these opportunities, hence the reason for posting this message. Applicants for astronomy and cosmology would be welcomed,  by me at any rate! It’s a rare opportunity to join a Physics department that’s actually growing in size…

To find out more about the Fellowships and Chairs, see here. Feel free to contact me informally if you have any questions, and  please also feel free to pass this on to anyone you think might be interested!

Kielder Star Camp

Posted in The Universe and Stuff with tags , , , on October 27, 2011 by telescoper

I  came across a story in the Grauniad about the Kielder Forest Star Camp at which scores of amateur astronomers are gathering along with their tents this week to exploit the darkest skies in England.  The skies are pretty dark above  Cardiff right now, but that’s because of the thick cloud rather than lack of light pollution. I hope they have better weather in Kielder which, if you didn’t know, is in Northumberland. With an area of 250 square miles, Kielder Forest is  England’s largest forest (although it’s actually more of a plantation, being man-made under the auspices the Forestry Commission) and it surrounds Kielder Water, the largest man-made reservoir in the UK. Anyway, as the time-lapse video shows, it’s  a fine spot for astronomy when the clouds stay away; at the end you’ll see the excellent new Kielder Observatory too!

Good luck to all the participants (and, more importantly, clear skies…) .

Come off it, REF!

Posted in Science Politics with tags , , , , , , on October 27, 2011 by telescoper

Yesterday we all trooped off to the Millennium Stadium in Cardiff for a Staff Away Day. We didn’t actually get to play on the pitch of course, which wasn’t even there, as it had been removed to reveal a vast expanse of soil. Instead we were installed in the “Dragon Suite” for a discussion about our preparation for the forthcoming Research Excellence Framework.

Obviously I can’t post anything about our internal deliberations, but I’m sure departments up and down the United Kingdom are doing similar things so I thought I’d mention a few things which are already in the public domain and my personal reactions to them. I should also say that the opinions I express below are my own and not necessarily those of anyone else at Cardiff.

The first thing is the scale of the task facing members of the panel undertaking this assessment. Each research active member of staff is requested to submit four research publications (“outputs”) to the panel, and we are told that each of these will be read by at least two panel members. The panel comprises 20 members.

As a rough guess I’d say that the UK has about 40 Physics departments, and the average number of research-active staff in each is probably about 40. That gives about 1600 individuals for the REF. Actually the number of category A staff submitted to the 2008 RAE was 1,685.57 FTE (Full-Time Equivalent), pretty  close to this figure. At 4 outputs per person that gives 6400 papers to be read. We’re told that each will be read by at least two members of the panel, so that gives an overall job size of 12800 paper-readings. There are 20 members of the panel, so that means that between 29th November 2013 (the deadline for submissions) and the announcement of the results in December 2014 each member of the panel will have to have read 640 research papers. That’s an average of about two a day…

Incidentally, as I’ve mentioned before, the Physics REF panel includes representatives from institutions in England, Scotland and Northern Ireland, but not Wales. The decision to exclude representation from Welsh physics departments was a disgrace, in my view.

Now we are told the panel will use their expert judgment to decide which outputs belong to the following categories:

  • 4*  World Leading
  • 3* Internationally Excellent
  • 2* Internationally Recognized
  • 1* Nationally Recognized
  • U   Unclassified

There is an expectation that the so-called QR  funding allocated as a result of the 2013 REF will be heavily weighted towards 4*, with perhaps a small allocation to 3* and probably nothing at all for lower grades. “Internationally recognized” research is probably worthless in the view of HEFCE, in other words. Will the papers belonging to the category “Not really understood by the panel member” suffer the same fate?

The panel members will apparently know enough about every single one of the papers they are going to read in order to place them  into one of the above categories, especially the crucial ones “world-leading” or “internationally excellent”, both of which are obviously defined in a completely transparent and objective manner. Not.

We are told that after forming this judgement based on their expertise the panel members will “check” the citation information for the papers. This will be done using the SCOPUS service provided (no doubt at considerable cost) by   Elsevier, which by sheer coincidence also happens to be a purveyor of ridiculously overpriced academic journals. I’ve just checked the citation information for some of my papers on SCOPUS, and found an alarming number of errors. No doubt Elsevier are  on a nice little earner peddling meaningless data for the HECFE bean-counters, but I haven’t any confidence that it will add much value to the assessment process.

There have been high-profile statements to the effect that the REF will take no account of where the relevant “outputs”  are published, including a recent pronouncement by David Willetts. On the face of it, that would suggest that a paper published in the spirit of Open Access in a free archive would not be disadvantaged. However, I very much doubt that will be the case.

I think if you look at the volume of work facing the REF panel members it’s pretty clear that citation statistics will be much more important for the Physics panel than we’ve been led to believe. The panel simply won’t have the time or the breadth of understanding to do an in-depth assessment of every paper, so will inevitably in many cases be led by bibliometric information. The fact that SCOPUS doesn’t cover the arXiv means that citation information will be entirely missing from papers just published there.

The involvement of  a company like Elsevier in this system just demonstrates the extent to which the machinery of research assessment is driven by the academic publishing industry. The REF is now pretty much the only reason why we have to use traditional journals. It would be better for research, better for public accountability and better economically if we all published our research free of charge in open archives. It wouldn’t be good for academic publishing houses, however, so they’re naturally very keen to keep things just the way they are. The saddest thing is that we’re all so cowed by the system that we see no alternative but to participate in this scam.

Incidentally we were told before the 2008 Research Assessment Exercise that citation data would emphatically not be used;  we were also told afterwards that citation data had been used by the Physics panel. That’s just one of the reasons why I’m very sceptical about the veracity of some of the pronouncements coming out from the REF establishment. Who knows what they actually do behind closed doors?  All the documentation is shredded after the results are published. Who can trust such a system?

To put it bluntly, the apparatus of research assessment has done what most bureaucracies eventually do; it has become  entirely self-serving. It is imposing increasingly  ridiculous administrative burdens on researchers, inventing increasingly  arbitrary assessment criteria and wasting increasing amounts of money on red tape which should actually be going to fund research.

And that’s all just about “outputs”. I haven’t even started on “impact”….