The Case of Norm versus Criterion

I saw a news item last night that revealed the grade boundaries for some of this year’s A-level examinations in the United Kingdom. Among the surprises were that for one board an A-grade in Mathematics corresponded to a mark of 55% and an A-grade in Physics was 59%. I’m sure I’m not the only one to find these results a bit disturbing.

The explanation given for these figures is basically that there’s a new style of A-level examination this year and the boundaries were adjusted so as not to penalize the current set of students with respect to previous years. In other words, students did much worse than expected on the new examinations so the grade boundaries were lowered.

Most assessments of academic performance such as A-levels can be classified into two broad types: criterion-referenced and norm-referenced. In the former performance is measured relative to defined goals, whereas in the latter it is defined relative to the performance of others taking the same test.

University assessments in the UK and Ireland (especially in the sciences) are usually criterion-based, meaning that the % score (and hence the grade) is determined by comparing the marks with a pre-prepared marking scheme based on the expected learning outcomes of the course. It is possible for all students taking an examination to get A-grades if they all meet the criteria. On a norm-referenced system one allocates the grades based on the distribution of scores and not on the absolute level of performance attained. In reality when marking examinations under a criterion-based system, one has a bit of discretion in how to award partial credit when a question is not done completely, so this is a bit of a simplification but those are the two approaches in a nutshell.

Some universities allow marks for a component of assessment to be scaled up (for example if there is a problem with an examination paper), which has the same effect as lowering the grade boundary, but this does not usually apply to the entire mark. Some universities don’t allow scaling at all in any circumstances.

In the universities with which I am familiar, an A-grade (corresponding to First-class performance) is fixed at a mark >70%, a B (2.1) is 60-69%, a C (2.2) is 50-59%, and so on. The pass mark at undergraduate level is 40%; it might be 50% at Masters level.

This is why the figure of 55% being an A-grade at mathematics comes as a such a shock to university-based academics: that score would be in the middle of the lower-second class range in a university examination.

This episode demonstrates one of the serious issues with A-levels as preparation for university entrance. On the one hand, especially in the sciences,  we want students to be equipped with certain basic skills and knowledge to enable them to cope with their course. That calls for a criterion-based system of assessment. On the other hand, in any one year, the `top’ universities want to recruit the `top’ students (in many cases because they want to have a high position the league tables because their entry tariff is high). It’s hardly surprising that the system is dysfunctional when it is being pulled in two mutually incompatible directions.

The upshot of this year’s mathematics and physics A-levels is that universities that take in students with an A-grade in Mathematics -can’t really have much confidence in what they have learned. To make matters worse, the grade boundaries differ from one Examination Board to the next. It’s a mess. At least here in Ireland there is a truly national examination system: there is a single Leaving Certificate examination in each subject that all students take.

While I am on about A-levels I’ll just mention another disadvantage that they have compared to the Leaving Certificate (and, for that matter, the International Baccalaureate) which is that they force students to choose a very narrow post-16 curriculum. Most students take three A-levels (not counting the useless `General Studies’) which for science students often means three science subjects (e.g. Maths, Physics & Chemistry). In the Leaving Certificate students take six or seven subjects and in the IB they take six. I’ve been in Ireland for less than two years so I’m not so familiar with the system here, but my experience with the IB over about 30 years in UK higher education, is that students are certainly no less prepared for university study if they took that than if they did A-levels.

There is currently a major review of the Irish Leaving Certificate going on. One of the things that surprises me about Ireland is that, despite its hard-won independence from the United Kingdom, it has a tendency to copy slavishly many of the silly things that the UK introduces, especially in higher education. I sincerely hope that the review of the Leaving Certificate does not fall into the trap of making it more like A-levels.

2 Responses to “The Case of Norm versus Criterion”

  1. Absolutely. We like to look carefully at whatever daft change our large neighbour carried out, and then confirm for ourselves that it was a bad idea. With the exception of Brexit, thank God

  2. Reblogged this on Symptoms Of The Universe and commented:
    Peter Coles making perfect sense, yet again. This time he highlights the clear superiority of the Leaving Certificate system over A-levels (which I similarly lauded here: )

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: