Archive for Examination

The Case of Norm versus Criterion

Posted in Education with tags , , on August 15, 2019 by telescoper

I saw a news item last night that revealed the grade boundaries for some of this year’s A-level examinations in the United Kingdom. Among the surprises were that for one board an A-grade in Mathematics corresponded to a mark of 55% and an A-grade in Physics was 59%. I’m sure I’m not the only one to find these results a bit disturbing.

The explanation given for these figures is basically that there’s a new style of A-level examination this year and the boundaries were adjusted so as not to penalize the current set of students with respect to previous years. In other words, students did much worse than expected on the new examinations so the grade boundaries were lowered.

Most assessments of academic performance such as A-levels can be classified into two broad types: criterion-referenced and norm-referenced. In the former performance is measured relative to defined goals, whereas in the latter it is defined relative to the performance of others taking the same test.

University assessments in the UK and Ireland (especially in the sciences) are usually criterion-based, meaning that the % score (and hence the grade) is determined by comparing the marks with a pre-prepared marking scheme based on the expected learning outcomes of the course. It is possible for all students taking an examination to get A-grades if they all meet the criteria. On a norm-referenced system one allocates the grades based on the distribution of scores and not on the absolute level of performance attained. In reality when marking examinations under a criterion-based system, one has a bit of discretion in how to award partial credit when a question is not done completely, so this is a bit of a simplification but those are the two approaches in a nutshell.

Some universities allow marks for a component of assessment to be scaled up (for example if there is a problem with an examination paper), which has the same effect as lowering the grade boundary, but this does not usually apply to the entire mark. Some universities don’t allow scaling at all in any circumstances.

In the universities with which I am familiar, an A-grade (corresponding to First-class performance) is fixed at a mark >70%, a B (2.1) is 60-69%, a C (2.2) is 50-59%, and so on. The pass mark at undergraduate level is 40%; it might be 50% at Masters level.

This is why the figure of 55% being an A-grade at mathematics comes as a such a shock to university-based academics: that score would be in the middle of the lower-second class range in a university examination.

This episode demonstrates one of the serious issues with A-levels as preparation for university entrance. On the one hand, especially in the sciences,  we want students to be equipped with certain basic skills and knowledge to enable them to cope with their course. That calls for a criterion-based system of assessment. On the other hand, in any one year, the `top’ universities want to recruit the `top’ students (in many cases because they want to have a high position the league tables because their entry tariff is high). It’s hardly surprising that the system is dysfunctional when it is being pulled in two mutually incompatible directions.

The upshot of this year’s mathematics and physics A-levels is that universities that take in students with an A-grade in Mathematics -can’t really have much confidence in what they have learned. To make matters worse, the grade boundaries differ from one Examination Board to the next. It’s a mess. At least here in Ireland there is a truly national examination system: there is a single Leaving Certificate examination in each subject that all students take.

While I am on about A-levels I’ll just mention another disadvantage that they have compared to the Leaving Certificate (and, for that matter, the International Baccalaureate) which is that they force students to choose a very narrow post-16 curriculum. Most students take three A-levels (not counting the useless `General Studies’) which for science students often means three science subjects (e.g. Maths, Physics & Chemistry). In the Leaving Certificate students take six or seven subjects and in the IB they take six. I’ve been in Ireland for less than two years so I’m not so familiar with the system here, but my experience with the IB over about 30 years in UK higher education, is that students are certainly no less prepared for university study if they took that than if they did A-levels.

There is currently a major review of the Irish Leaving Certificate going on. One of the things that surprises me about Ireland is that, despite its hard-won independence from the United Kingdom, it has a tendency to copy slavishly many of the silly things that the UK introduces, especially in higher education. I sincerely hope that the review of the Leaving Certificate does not fall into the trap of making it more like A-levels.

Space without Physics…

Posted in Uncategorized with tags , , , , , on March 24, 2010 by telescoper

I’m indebted to a colleague (Annabel Cartwright) for sending me this (coincidentally topical) sample question, illustrating the quality of a modern British school science examination.

Since it’s now clear  that there is no room for science in the new era of the UK Space Agency, I suppose we should get used to the removal of science from other things too. Starting with science exams.

This question is taken from a GCSE Physics examination.

Some people think that governments spend too much money on space research.

Which ONE of the following statements is true?

  1. Science can tell us what the planets are made of, and whether they ought to be explored.
  2. Science can tell us what the planets are made of, but not whether they ought to be explored.
  3. Science cannot tell us what the planets are made of but can tell us whether they ought to be explored.
  4. Science cannot tell us what the planets are made of, nor whether they ought to be explored.

Apparently one (and only one) answer is correct. Any offers?