Taking down their Particulars and Examining their Testimonials

If you’ve looked at Cosmic Variance recently you will know that has almost gone up in flames (metaphorically speaking) . The incendiary item was what I thought was a gently humorous post on the subject of recommendation letters for entry into graduate schools, which evolved to include postdoctoral positions too. This item has generated nearly a hundred comments so far, some of which are quite sensible and interesting but others worryingly vitriolic. One correspondent in particular got hold of the wrong end of the stick and proceeded to beat wildly about the bush with it,  accusing academics of everything from intellectual snobbery to the Whitechapel Murders.

I’m actually quite pleased that the more extremist comments are there, as they make mine look quite sensible which they perhaps wouldn’t if they were on their own. I’ve therefore collected my thoughts here to see if they generate any reaction.

A follow-up post attempted to defuse the issue with an injection of common sense, but it remains to be seen whether this will indeed steady the ship. (I’m proud of that multiply mixed metaphor.)

The principal bone of contention is the matter of “recommendation letters” and whether a Professor should ever write negative comments when asked to recommend a student for a place on a graduate course.

In the UK we generally don’t have “recommendations” but “references” or “testimonials” which are supposed to describe the candidate’s character and abilities in a manner that is useful to those doing the recruitment. They are not meant to be written in absurdly hyperbolic terms and they are not meant to ignore any demonstrable shortcomings of the applicant. They are supposed to advise the people doing the recruitment of the suitability of the candidate in a sober, balanced and objective way. Fortunately, most students applying to graduate schools are actually rather good so there are many more positives than negatives, but if there are weaknesses in my view these  must be mentioned, even this turns out to be the kiss of death to their application.

Another objection is to recommendation letters that include statements of the form Aaron is better than Brenda but not as good as Charlie. I don’t object at all to the idea of a reference that includes some form of  ordering like this. Since there are inevitably more applicants than places the panel will have to make a ranking, so why not help them by giving your input? After all, you know the candidates better than the panel does.

The point is that the referee is not only providing a service for the student but also for the recruiting school. On this basis, it is, I think, perfectly valid to include negative points as long as they can be justified objectively.

British professors are often criticized by our colleagues over the pond for writing very reserved recommendation letters, but having one year received references from a US institution on behalf of 4 different students who were all apparently the best student that institution had ever had in physics, I think I prefer the old-fashioned British understatement.

However, references transcripts and other paperwork can only establish whether a student has reached the threshold level of technical competence that is needed to commence a research degree. That’s a necessary but not sufficient condition for their success as a scientist. The other factors – drive, imagination, commitment, and diligence (which apparently is a term of abuse in the USA) are much harder to assess. I think this part has to be done at interview. You can’t just rely on examination results because it’s by no means true that the best students at passing examinations turn into the best graduate students. Research is a whole different ball game.

I also think there’s a difference between how references are used for making a job appointment versus a place at graduate school. Where I come from, in the UK, graduate study is funded by a studentship which pays a stipend rather than a salary and the successful applicants are not formally employed by the university. It is rather different in the case of a postdoc where the successful candidate is an employee of the institution.

Owing to recent changes in employment legislation in the UK, best practice for the process of appointing staff is now considered to not even ask for references until after short-listing or even after interview. The purpose of references is simply to verify that the information the applicant has given in the application is complete and correct; they are not to be used in deciding on quality. Shortlisting is done on the basis of whether the applicant can show in the application that they have skills that match the requirement of the post. The final decisions is made after interview of shortlisted candidates.

Many academics hate this new-fangled way of doing things partly because its a new-fangled way of doing things but also because they tend to rely heavily on references in judging the relative quality of candidates for PDRA appointments. Personally, however, I don’t find references particularly helpful in this context – especially those from America where the language is so inflated as to be laughable – so, unlike most of my colleagues, I’m quite happy to embrace the “new” approach. I think relying too much on references is a tantamount to wanting other people to make difficult decisions for you rather than making them yourself.

On top of this, modern protocol requires the use of a standard application form rather than just a CV and list of research interests. That way the relative merits of all candidates can be judged on the basis of the same pieces of information and answers to the  same questions. Whatever you think about this process, it certainly does make things more transparent.

I’m currently advertising a postdoc job and will be shortlisting for this position on the basis I have described, i.e. without asking for references uprfront. I’ll only look at references later on, after shortlisting.  This is the first time I will have done it this way, so I am interested to see how it works.

But I can’t see at all how one could possibly make decisions concerning entry of an undergraduate student onto a graduate programme without using references earlier on in the process.

About these ads

17 Responses to “Taking down their Particulars and Examining their Testimonials”

  1. I think you are missing the point about negative comments. There is nothing wrong in mentioning particular weakness within an overall “positive” picture. But if the negative comments are so severe as to jeopardize the job…then why would you want to write the letter in the first place? If your opinion is that the applicant is not suited or qualified for the job, then why not just say this to the applicant’s face. There may be another prof who disagrees with your opinion and can write a more positive letter for that student…why would you want to deprive the student of that possibility by writing negative comments that could jeopardize his job.

    The debate over “negative comments” is referring to comments that will jeopardize chances of getting the job. If your negative comments are simply there for providing an overall picture, without hurting job prospects, then thats a different matter.

  2. At the end of the day your “testimonial” is only an opinion. If you think the student is not qualified, why not just refuse to write the letter? Then the student can find a Prof. who may have a different and more positive opinion. By writing negative comments that jeopardize a students chances you are doing two things:

    1. You are actively taking part in ensuring that the student doesn’t get the job.

    2. You are taking away an opportunity from the student to get a more positive letter from someone else who has a different opinion than you.

    One can point out a students weakness in the context of a letter that is still overwhelmingly positive and recommends the student for the job. But to take deliberate action to jeopardize the student with negative comments, instead of just being honest with the srtudent, in my opinion is wrong.

  3. — “1. You are actively taking part in ensuring that the student doesn’t get —–the job. ”

    yes, actively taking part in ensuring that a student doesn’t get the job based on an OPNION.

  4. one more point:

    “They are supposed to advise the people doing the recruitment of the suitability of the candidate in a sober, balanced and objective way.”

    …this may be. But your primary responsibility is the welfare of the student. This is way more important than any service you are providing to hiring institutions…they’ll do just fine without your letter…trust me. It is the student’s welfare that is most important.

    If you plan to write many negative comments that can hurt a students chances, it is in the best interest of the student that he knows about this upfront so that he may seek other options. Also, if you think the student is not qualified you should advise the student accordingly. It is also part of the job “counsel” to the student.

  5. telescoper Says:

    I thought this would happen.

    Of course, a Professor should have the best interests of their students at heart. If he/she can’t support an application to a graduate school he/she should discuss the matter with the student, explain why, perhaps persuade them not to apply. If they insist, he/she should suggest they find an alternative referee. I said all this over on cosmic variance, in fact. But if they insist on going ahead one must be honest.

    A reference is an opinion, but it will be based on detailed knowledge of the student. Ultimately the selection committee will make a decision based on an opinion too. The referee’s opinion should help them form theirs, but it won’t necessarily be the same. References are only part of the process. But at the end of the day opinions are all we have. We can’t admit every student to graduate school. There aren’t enough places.

    As I said in my post, most applicants for graduate school are suitably qualified and my references are appropriately very positive in the vast majority of cases. It would be horrendous to write negative comments without backing them up or to write such things out of malice or spite. I don’t really think this happens to any significant extent. Perhaps unsuccessful students blame their referees when actually there are other reasons for not being selected.

    One can point out a students weakness in the context of a letter that is still overwhelmingly positive and recommends the student for the job. But to take deliberate action to jeopardize the student with negative comments, instead of just being honest with the student, in my opinion is wrong.

    I agree (except that a place in graduate school is not a “job”); see my post. Believe me, Professors do try to do everything they can to support the students. However, some students seem to expect glowing references regardless of their ability. If a student with poor grades insists on applying to graduate school (after being advised that perhaps they are not sufficiently qualified), then it is unreasonable for them to expect to receive such a wonderful reference that it puts them straight to the top of the heap. Of course if there are reasons for poor performance, such as illness or other extenuating circumstances, then the referee can help by commenting on those in a positive light. But nevertheless the referee is supposed to write a balanced, objective view of the candidate. This may be more positive for some students than others. If everyone got equally positive references regardless of ability then the reference would be of no use to anyone at all.

    If, on the basis of all the information available, the committee decides not to admit a given student then it is probably not because a Professor wrote an honest reference, it’s more likely that the places went to better candidates. That’s understandably disappointing for the student, but those are the realities of life.

    Finally, let me say that I certainly do have the welfare of students at heart. But not just students at my own institution. In order for the system to work it has to be fair. How fair is it for student A to get into graduate school based on grossly misleading references, thus depriving student B of a place?

  6. Anton Garrett Says:

    Important to tell the (whole) truth when writing a reference, not only to assist the institution considering who to take on, but for your own reputation for truthfulness. It seems there are ways to get across negative info in the US system but they are a bit more subtle – learn the code. With freedom of info legislation causing students to demand to see references and even threaten litigation, the system might have to revert to phone calls between profs. I think that the student should be told in *broad* terms what you intend to say

    Anton

    • telescoper Says:

      Agreed. The effect of recent legislation is to reduce references to statements of demonstrable fact only. They then become simple validations of information already contained in the application. That’s fine in my opinion for job references, where one really should go and read the papers and form one’s own decision. Hence my acceptance of the policy of only taking references after shortlisting. But it’s very tricky for graduate admissions because there is no research profile or publications to go on (at least in most cases). Remember too that the students haven’t even graduated at the point when they are applying for PhDs.

      I tend to go very strongly on references from the undergraduate project supervisor as that is often a good indicator of aptitude for research. If the supervisor can’t support the application, which would be the case if the project work was of poor quality and declines to write a reference then even if someone else does write one, it still decreases the chance that the student will be accepted.

      It’s no wonder students feel alienated if the references have to be written in code. I’m all for having frank discussions with the student beforehand about the prospects and then writing an honest and clear reference.

      In most cases in my experience, though, it’s not the reference that kills the application anyway. It’s the academic profile in the case of graduate applicants or track record in the case of PDRAs.

      Given the intense competition there have to be losers. And not getting into an academic job is not the end of the world anyway, many excellent people don’t even try because they have other plans.

  7. Adrian Burd Says:

    As we too are going through this annual ritual, I find that one of the starkest differences between my experiences in the UK and the US is the interview. The interviews I had for graduate school in the UK were real interviews; I recall vividly that my future supervisor asked me problems that I had to solve on the board in front of him and a couple of other professors. This may no longer be standard practice however.

    Over this side of the pond, the interview for graduate school is much more of a meet and greet affair. One can ask oblique questions about a student’s interests, ambitions etc. But I have been warned off from asking proper questions gauged at assessing how a student thinks. So, I find that the interviews give me little more than an impression of how personable the student is, and how enthusiastic they are.

    As a result, I end up making my assessment based almost entirely on the student’s transcript and reference letters.

    Adrian

  8. telescoper Says:

    Adrian,

    You and I were both interviewed at Sussex (probably by the same people!) so we both know that was a pretty tough ordeal. Times have changed over here and we generally focus less on solving problems on the board and more on finding out about the candidate’s enthusiasm for the research topic through their more general background knowledge. (If any UK undergraduates are reading this my advice is to make sure you know something about the area you’re applying for…).

    People assume that UK applicants for US graduate school positions are doomed because of the references they get. I’m not sure about this. I think the main problem is the GRE which is unlike anything our students have taken before and which they consequently struggle with. US candidates will probably be much better prepared for this test.

    Of course personality is a non-negligible component of the candidate’s profile. He/she will be working very closely with the supervisor over a long time. It just won’t work if they can’t tolerate each others company.

    Peter

  9. Rob Ivison Says:

    Put simply, if professors don’t give wholly honest references then they are disadvantaging the other applicants. Also (this will annoy some of your correspondents, hopefully), being unable to judge whether your mentor/professor/whatever will write a decent reference for you is a weakness.

    It’s worth mentioning that the US system of tenure review does no-one any favours, these days. I was once happy to write a letter, when asked by a department head, but an unfortunate incident re: the Freedom of Information Act led me to adopt a blanket policy of refusal. What’s in it for me? Usually the answer would be “the chance to ensure justice is done, help the right person get the job, and keep a fair system alive”. All these should benefit me in the long run. Sadly, the answer can now be “a gob-full of abuse”.

    I wonder how many folk give over-the-top references to help poor students find work with people they don’t like? ;-)

  10. telescoper Says:

    Rob,

    I doubt if there are many smart enough to have hit on your Machiavellian scheme. But if they start doing it now we know who to blame.

    Peter

  11. Rob Ivison Says:

    Mm. I had assumed it was common practice, but then some of the folk I work with make Niccolo look like a pussy cat…

  12. Adrian Burd Says:

    Peter,
    I seem to recall there being three in the interview – John, Roger Taylor, and a third, Robert Smith I think but it might have been Leon Mestel.

    It sounds as if the UK system has changed into something akin to the US system. I agree with you that both student and supervisor need to get along. However, it would be nice to have an acceptable means of gauging the extent of their knowledge and ability – I still find it impossible to figure out just what a B+ in Calculus III from a small liberal arts college I’ve never heard of actually means. I know some of these places are extremely good, but others I’m not so sure of.

    I also agree about UK students taking the GRE – I’m sure I would do poorly! I do find it interesting that one gets applicants with very high scores in the GRE quantitative section, and yet get poor grades in any quantitative-based courses. I do know that people practice taking GRE questions, so I’m not sure what it actually tells one.

    I think that one has to look at the whole package – transcript, GRE scores, letters, student statements, interview etc.

    Adrian

  13. “Also (this will annoy some of your correspondents, hopefully), being unable to judge whether your mentor/professor/whatever will write a decent reference for you is a weakness.”

    It’s certainly a weakness. I don’t see how that statement could be annoying. What could be annoying is the implication that this strength/weakness is strongly weighted when selecting an undergraduate student for admission for a graduate course. Is this really what you’re trying to select for? Surely not.

  14. Rob Ivison Says:

    Whilst reading a bad reference I wouldn’t be able to tell whether the student had chosen poorly, or whether every conceivable referee would have said similar things… it’s not a weakness that can be weighted.

    Telescoper: if you could have the perfect reference, revealing every aspect of a potential student’s ability and character, what would you be hoping to find? Creativity blended with a anal streak? Blonde streak blended with – I’ll stop there…

  15. telescoper Says:

    I really just look for evidence that they have sufficient technical knowledge and skill to make them worth interviewing. Evidence that they’re not a gormless troglodyte helps, as does that tell-tale bit of quirkiness that suggests they might actually be mad enough to accept an offer of doing a PhD under my supervision.

  16. Why should a prospective student be expected to own a gorm? Or is it sufficient merely to have access to one? And am I to infer you are biased towards Celtic fans? Did Newcastle once lose a top player to Rangers or something like that?

    (Maybe these are the sort of questions which would sort the wheat from the goats at interview.)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 3,271 other followers

%d bloggers like this: