Archive for A-levels

The Case of Norm versus Criterion

Posted in Education with tags , , on August 15, 2019 by telescoper

I saw a news item last night that revealed the grade boundaries for some of this year’s A-level examinations in the United Kingdom. Among the surprises were that for one board an A-grade in Mathematics corresponded to a mark of 55% and an A-grade in Physics was 59%. I’m sure I’m not the only one to find these results a bit disturbing.

The explanation given for these figures is basically that there’s a new style of A-level examination this year and the boundaries were adjusted so as not to penalize the current set of students with respect to previous years. In other words, students did much worse than expected on the new examinations so the grade boundaries were lowered.

Most assessments of academic performance such as A-levels can be classified into two broad types: criterion-referenced and norm-referenced. In the former performance is measured relative to defined goals, whereas in the latter it is defined relative to the performance of others taking the same test.

University assessments in the UK and Ireland (especially in the sciences) are usually criterion-based, meaning that the % score (and hence the grade) is determined by comparing the marks with a pre-prepared marking scheme based on the expected learning outcomes of the course. It is possible for all students taking an examination to get A-grades if they all meet the criteria. On a norm-referenced system one allocates the grades based on the distribution of scores and not on the absolute level of performance attained. In reality when marking examinations under a criterion-based system, one has a bit of discretion in how to award partial credit when a question is not done completely, so this is a bit of a simplification but those are the two approaches in a nutshell.

Some universities allow marks for a component of assessment to be scaled up (for example if there is a problem with an examination paper), which has the same effect as lowering the grade boundary, but this does not usually apply to the entire mark. Some universities don’t allow scaling at all in any circumstances.

In the universities with which I am familiar, an A-grade (corresponding to First-class performance) is fixed at a mark >70%, a B (2.1) is 60-69%, a C (2.2) is 50-59%, and so on. The pass mark at undergraduate level is 40%; it might be 50% at Masters level.

This is why the figure of 55% being an A-grade at mathematics comes as a such a shock to university-based academics: that score would be in the middle of the lower-second class range in a university examination.

This episode demonstrates one of the serious issues with A-levels as preparation for university entrance. On the one hand, especially in the sciences,  we want students to be equipped with certain basic skills and knowledge to enable them to cope with their course. That calls for a criterion-based system of assessment. On the other hand, in any one year, the `top’ universities want to recruit the `top’ students (in many cases because they want to have a high position the league tables because their entry tariff is high). It’s hardly surprising that the system is dysfunctional when it is being pulled in two mutually incompatible directions.

The upshot of this year’s mathematics and physics A-levels is that universities that take in students with an A-grade in Mathematics -can’t really have much confidence in what they have learned. To make matters worse, the grade boundaries differ from one Examination Board to the next. It’s a mess. At least here in Ireland there is a truly national examination system: there is a single Leaving Certificate examination in each subject that all students take.

While I am on about A-levels I’ll just mention another disadvantage that they have compared to the Leaving Certificate (and, for that matter, the International Baccalaureate) which is that they force students to choose a very narrow post-16 curriculum. Most students take three A-levels (not counting the useless `General Studies’) which for science students often means three science subjects (e.g. Maths, Physics & Chemistry). In the Leaving Certificate students take six or seven subjects and in the IB they take six. I’ve been in Ireland for less than two years so I’m not so familiar with the system here, but my experience with the IB over about 30 years in UK higher education, is that students are certainly no less prepared for university study if they took that than if they did A-levels.

There is currently a major review of the Irish Leaving Certificate going on. One of the things that surprises me about Ireland is that, despite its hard-won independence from the United Kingdom, it has a tendency to copy slavishly many of the silly things that the UK introduces, especially in higher education. I sincerely hope that the review of the Leaving Certificate does not fall into the trap of making it more like A-levels.

Advertisements

Failing A-levels

Posted in Education with tags , , on April 17, 2019 by telescoper

As it is tangentially related to yesterday’s post I thought I’d comment on an article in The Times with the headline

University grades: firsts for quarter of students with lowest A levels

The piece (which you probably won’t read as it is behind a paywall) goes on to imply that the success at degree level of students who got poor A-levels results is the result of ‘grade inflation’.

My take on this is somewhat different. To me it just confirms what I’ve thought for years, namely that A-levels are virtually useless, either as a preparation for undergraduate study or as an indicator of academic potential. If they are are a guide to anything at all, it is to the quality of the school the student was lucky enough to attend.

It’s not only a shame that UK universities rely on A-levels so heavily for student recruitment but also a disgrace that institutions are punished in league tables whenever they take on students with low results. And if they do a good job educating such students to high levels of achievement they get attacked in churlish articles accusing them of lowering standards.

The assumption behind this is that there should be a near-perfect correlation between entry and exit qualifications. That is not the case at all, and why on earth should it be?

Look at this the other way round. Oxbridge only accepts students with the highest A-levels results, so why do these Universities not award more first-class degrees? Dare I suggest that perhaps not all the students they select have the aptitude their school qualifications suggest?

I noticed this the other day. It’s a list of skills needed for the job market in 2020.

Strangely, ‘rote learning’, ‘uncritical regurgitation of factoids’, and ‘ability to perform formulaic tasks’ are not on the list. They’re not much use as a preparation for university study either. So why does the UK school education system place such an emphasis on precisely these useless activities, to the exclusion of actually useful things?

Answers on a postcard please.

Qualification Matters

Posted in Biographical, Education with tags , , , on January 18, 2014 by telescoper

Yesterday evening I had a very enjoyable dinner with a seminar speaker and group of students from the Kobayashi-Maskawa Institute at Nagoya University. On the way back to the guest house I’m staying in, at about 9.30pm, we passed a group of young kids in uniform apparently returning from school. I was told that they were students from a Junior High School (chūgaku) who had been studying late in preparation for an entrance examination to the (selective) Senior High Schools (kōtōgakkō). I was a bit surprised to see young teenagers putting in such long hours, but such behaviour seems quite normal here in Japan.

One of the things I have to get to grips with when I get back to the School of Mathematical Sciences at the University of Sussex on Monday is the annual admissions cycle. I have meetings in my calendar for next week about this, and we are hosting an applicant visit day on Saturday 25th January; we have of course already made offers to a large number of students based on their predicted grades at GCE A-level. This process is fraught with difficulties, not least because there some schools seem to over-predict the grades that their pupils achieve in order to maximise the chance of them getting into University. Predicted grades being rather unreliable, it would of course be fairer on all concerned to use actual grades, i.e. to defer university applications until after the A-level results come out, but that would be far too sensible so obviously will not happen in the United Kingdom. I’m actually quite sceptical of the usefulness of A-levels for preparing students for University in general, but that’s another matter.

Anyway, the mini-rant above isn’t main point of this post, which is instead to issue a warning to those  students taking A-levels later this year. It is  based on anecdotal evidence, but quite a lot thereof. The point is that universities will often reduce their usual offer at A-level when they find an applicant with very high predicted grades, sometimes even making an unconditional offer (actually the minimum is two E grades) to high-fliers. This happened a lot in the old days when I was applying for a place at university. Though it is less common nowadays the government’s policy of lifting controls on universities’ ability to recruit students with three A-level grades at ABB or better is bound to increase the amount of game-playing as competition between recruiters intensifies.

This creates a number of issues, but the one I want to pick out arises when a student is predicted to get three As at A-level, but his/her first-choice university gives them an unconditional offer. Their first reaction will obviously be “whoopee! I’ll accept that offer and, what’s more, I don’t need to fret too much about my summer examinations..”.

In fact I’ve known plenty of students who came into university with quite modest A-level grades, but did brilliantly well at their studies and ended up with first-class degrees. There can be many explanations behind such cases quite apart from the relaxation effect. Some schools don’t have good specialist science teachers, for example; this can mean that a student’s interest is really only ignited once they get into university.

The problem is, though, that A-levels results are not just for university entry, they’re for life. A student may graduate with a good degree, even a First,  but so do many others. When selecting for postgraduate jobs, or even postgraduate study, recruiters often have a large number of people with excellent degrees; that’s an obvious consequence of the expansion of the Higher Education sector in recent years. What happens, therefore, is that employers and PG admissions tutors have to look at other factors; naturally, that includes A-level results but also even GCSEs. You might be surprised to learn that even if you get a first-class degree, your chances of getting a PhD place at a top institution depends on your performance at School but they definitely do. I’m not saying that this should, be the case, just that it in practice it is.

I recall hearing recently from one former student who had a first-class degree in Physics and a PhD from an excellent University, and who was applying for a job in a research institute (not in the university sector). The application form he received asked him to list not only his A-levels but also his GCSE results!

I hope I’ve made my point to any prospective students, but I’ll summarize it in case I haven’t. If you get a generous offer from your chosen university then that’s a good thing, because it means that they want you. You should be happy. But don’t ease off on your studies because if you end up getting poorer A-levels than you deserve, they may one day come back to haunt you.

Value Added?

Posted in Bad Statistics, Education with tags , , , , , on October 22, 2012 by telescoper

Busy busy busy. Only a few minutes for a lunchtime post today. I’ve a feeling I’m going to be writing that rather a lot over the next few weeks. Anyway, I thought I’d use the opportunity to enlist the help of the blogosphere to try to solve a problem for me.

Yesterday I drew attention to the Guardian University league tables for Physics (purely for the purposes of pointing out that excellent departments exist outside the Russell Group). One thing I’ve never understood about these legal tables is the column marked “value added”. Here is the (brief) explanation offered:

The value-added score compares students’ individual degree results with their entry qualifications, to show how effective the teaching is. It is given as a rating out of 10.

If you look at the scores you will find the top department, Oxford, has a score of 6 for “value added”;  in deference to my alma matter, I’ll note that Cambridge doesn’t appear in these tables.  Sussex scores 9 on value-added, while  Cardiff only scores 2. What seems peculiar is that the “typical UCAS scores” for students in these departments are 621, 409 and 420 respectively. To convert these into A-level scores, see here. These should represent the typical entry qualifications of students at the respective institutions.

The point is that Oxford only takes students with very high A-level grades, yet still manages to score a creditable 6/10 on “value added”.  Sussex and Cardiff have very similar scores for entry tariff, significantly lower than Oxford, but differ enormously in “value added” (9 versus 2).

The only interpretation of the latter two points that makes sense to me would be if Sussex turned out many more first-class degrees given its entry qualifications than Cardiff (since their tariff levels are similar, 409 versus 420). But this doesn’t seem to be the case;  the fraction of first-class degrees awarded by Cardiff Physics & Astronomy is broadly in line with the rest of the sector and certainly doesn’t differ by a factor of several compared to Sussex!

These aren’t the only anomalous cases. Elsewhere in the table you can find Exeter and Leeds, which have identical UCAS tariffs (435) but value added scores that differ by a wide margin (9 versus 4, respectively).

And if Oxford only accepts students with the highest A-level scores, how can it score higher on “value added” than a department like Cardiff which takes in many students with lower A-levels and turns at least some of them into first-class graduates? Shouldn’t the Oxford “value added” score be very low indeed, if any Oxford students at all fail to get first class degrees?

I think there’s a rabbit off. Can anyone explain the paradox to me?

Answers on a postcard please. Or, better, through the comments box.

Will University Swapping Work?

Posted in Education with tags , , , , , on October 21, 2012 by telescoper

Yesterday’s crossword having been more straightforward than usual, I found myself with time to peruse the Independent newspaper at my leisure. While doing so I came across a little item describing a plan suggested by Lord Rees that students from “disadvantaged backgrounds” should be allowed to swap universities after two years of a three-year degree and transfer to a Russell group institution. Apparently this idea is based on a scheme that runs “successfully” in the University of California.

The purported aim of this is to give “a second chance” to students who didn’t do well enough at A-level to get into an “elite” university – which is laudable – but it doesn’t deal with the underlying problem, namely that our pre-university education system is a mess, for two reasons.  First, students can have the misfortune to attend a school where certain subjects are taught badly or not at all. This is a particular problem in my own field, physics. Second, the A-level examinations on which most institutions base their entry criteria do not provide a reasonable assessment of a candidate’s suitability for university study.

Because of these problems many students either don’t apply to top universities or fail to make the grades required. Such universities are reluctant to drop their grades to make special allowance because they would then get penalised in the league tables –  a high entry requirement at A-level is perceived to be a mark of quality. I’m convinced that this is a major flaw in the system. Some of the very best students I’ve had the pleasure to work with at Cardiff, for example, came in at a time when our recruitment team was struggling to meet its quota,  with modest A-level scores that would not normally have been high enough to get in. I worry a great deal about how many more talented young people there are out there who lacked that bit of luck and missed out entirely.

Lord Rees is correct in saying that it will take a very long time to fix the pre-university education system, and his proposal is an attempt to provide a sticking-plaster solution later on. If you like, it’s an admission of defeat. Elite universities will be allowed to carry on using inappropriate criteria to reject talented students applying to join the first year of a degree, but will be allowed to cherry-pick the best performers from other institutions into Year 3.

Although I think this proposal contains some good ingredients, there are several things about it that worry me. I don’t know how many students will want to move after two years in the first place. They will have made friends, formed relationships, and generally settled in at their original university and to up sticks in order to travel to another university for their final year would be very disruptive. Steps would have to be taken to ensure continuity of curriculum too. And what about the financial and other implications for the original institution, which would have to be prepared to lose an indeterminate number of its best students at the end of Year 2, with consequent impact on the quality of its graduating class?

I don’t think it’s fair for the so-called “elite” to exploit the hard work put in by other departments and institutions in order to mask its own failure to recruit appropriately. The only fair solution is to fix the university admission system, which means fixing our  broken A-levels.

And another thing. I’m shortly moving from Cardiff (which is a member of the Russell group) to Sussex (which isn’t).  Look at the league tables for Physics and tell me which one should be regarded as “elite”. Should students choose their University on the basis of which one provides the best education, or on the basis that it provides membership of a prestigious club?

On balance, I don’t think this scheme is workable in the way suggested. There is a variant, however, which I think is more promising. I think we should scrap the current confused system of 4-year undergraduate degrees (MPhys, MSci, etc) and adopt a standard system of 3-year Bachelors degrees. The next level of degree should be standalone postgraduate Masters. I’d prefer these to be two years, actually, but that’s not essential to this argument. Students could then transfer after their Bachelors’ degree into an “elite” university for their Masters if they so wish.

Qualifying Standards

Posted in Biographical, Education, Sport with tags , , on August 13, 2012 by telescoper

Well, the Olympics are finally over. I have to say I didn’t see much of the games themselves, although I did catch Mo Farah’s excellent run in the 5000m final and afterwards reminding us all that one can be a great athlete as well as humble and likeable individual. I see that Team GB (and NI) have done remarkably well in landing a haul of 29 gold medals, well up on Beijing 2008.

Many of these were in sports I know nothing about (such as Keirin and Dressage, both of which sound to me like items of IKEA furniture) but I’m perfectly happy to accept that winning any Olympic Gold medal is a remarkable achievement and requires not just talent but dedication and hard work. I hope the success of Team GB inspires others with the thought that succeeding in doing what’s difficult can be rewarding in itself, whether or not it leads to personal wealth.

I have just a couple more days here in Copenhagen, where the weather has been lovely throughout my visit. Here’s a gratuitous picture of one of the city’s lovely parks in the sunshine:

I’m feeling a lot better for having been here for the last week or so. The people here have been so very kind and understanding. I have to admit, though, I’m a bit nervous about going back because: (a) I have more medical tests to go through before I start on a proper programme; (b) quite a big backlog has built up of things I have to do; and (c) I have to face the colleagues and students I’ve let down so badly over the last few weeks and try to find a way of making up for my dereliction of duty.

The next big thing when I get back to work will be admissions. On Thursday (16th August) the A-level examination results will be officially announced and the clearing system opens for business. Only then will we find out how many students we’ll have entering the first year in October. We think things have gone pretty well on the recruitment front, but you never know until you see the final numbers. Fingers crossed.

Anyway, with the results having been published, there’ll no doubt be the usual discussion in the newspapers about whether the Olympic Games were easier this year than they were in our day….

By Gove, I agree!

Posted in Education with tags , , , on April 4, 2012 by telescoper

I never thought the day would come, but I have to admit it. I agree with Michael Gove. There. I said it.

Not with everything he says, of course. But I do think that universities should take over responsibility for the examinations required for University Entrance, currently known as A-levels. Here is an excerpt from an old post on this, and I’ve said much the same thing on several other occasions:

So what’s the solution? I think it is to scrap A-levels entirely, and give the system of pre-university qualifications over to the people who actually know what students need to know to cope with their courses, i.e. the universities. There should be a single national system of University Entrance Examinations, set and moderated by an Examination Board constituted by university teachers. This will provide the level playing field that we need. No system can ever be perfect of course, but this is the best way I can think of to solve the biggest problem with the current one. Not that it will ever happen. There are just too many vested interests happy with the status quo despite the fact that it is failing so many of our young people.

But lest you all think I’ve turned into a Conservative, let me point out that the fault with the current system is precisely that market forces have operated to the detriment of educational standards. The GCE examination boards compete for customers by offering easier and easier examinations each year, regardless of what students need to know to cope with University courses. What I advocate is renationalisation.  I bet Mr Gove doesn’t like it put that way…

Oh and another thing. I think universities should be given this task, but should also be paid for doing it just as the examination boards now are. That way it will not be treated as yet another imposition from the top, but an important task that has a similar status within a university as teaching and research.