Science Examination Blues

I woke up this morning …

.. to the 7am news on BBC Radio 3, including a story about how GCSE science examinations are not “sufficiently rigorous”. Then, on Twitter, I saw an example of an Edexcel GCSE (Multiple-choice) Physics paper.  It’s enough to make any practising physicist weep.

Most of the questions are very easy, but there’s just as many that are so sloppily put together that they  don’t make any sense at all. Take Question 1:

I suppose the answer is meant to be C, but since it doesn’t say that A is the orbit of a planet, as far as I’m concerned, it might just as well be D. Are we meant to eliminate D simply because it doesn’t have another orbit going through it?

On the other hand, the orbit of a moon around the Sun is in fact similar to the orbit of its planet around the Sun, since the orbital speed and radius of the moon around its planet are smaller than those of the planet around the Sun. At a push, therefore you could argue that A is the closest choice to a moon’s orbit around the Sun. The real thing would be something close to a circle with a 4-week wobble variation superposed.

You might say I’m being pedantic, but the whole point of exam questions is that they shouldn’t be open to ambiguities like this, at least if they’re science exams. I can imagine bright and knowledgeable students getting thoroughly confused by this question, and many of the others on the paper.

Here’s a couple more, from the “Advanced” section:

The answer to Q30 is, presumably, A. But do any scientists really think that galaxies are “moving away from the origin of the Big Bang”?  I’m worried that this implies that the Big Bang was located at a specific point. Is that what they’re teaching?

Bearing in mind that only one answer is supposed to be right, the answer to Q31 is presumably D. But is there really no evidence from “nebulae” that supports the Big Bang theory? The expansion of the Universe was discovered by observing things Hubble called “nebulae”..

I’m all in favour of school students being introduced to fundamental things such as cosmology and particle physics, but my deep worry is that this is being done at the expense of learning any real physics at all and is in any case done in a garbled and nonsensical way.

Lest I be accused of an astronomy-related bias, anyone care to try finding a correct answer to this question?

The more of this kind of stuff I see, the more admiration I have for the students coming to study physics and astronomy at University. How they managed to learn anything at all given the dire state of science education in the UK is really quite remarkable.

About these ads

37 Responses to “Science Examination Blues”

  1. [...] This post was mentioned on Twitter by Edd Edmondson, Peter Coles. Peter Coles said: Science Examination Blues: http://wp.me/pko9D-1BK [...]

  2. Oh shit. Thats awful.

  3. telescoper Says:

    This sort of thing is why I think the exam boards should be scrapped and the task of setting schools examinations given to people who actually know something about the subject(s) …

  4. I don’t disagree with the spirit of this post at all. But I think this *good* post would be transformed into an *excellent* post if you had included how *you* would have written the questions yourself, given the constraint of multiple choice answers. It would also be useful to provide a mark scheme, and list the aspects of the curriculum (which can be your dream, idealised curriculum, of course!) that you think your questions test.

    Cheers, Tom

  5. Brendan Says:

    So in question 31, answer C isn’t referring to BAOs?

    This is pretty terrifying to read, but no great surprise. It was terrible when I studied it a decade ago; it’s got steadily worse over the years.

  6. telescoper Says:

    Tom,

    Your suggestion presupposes that I would have asked questions like this in the first place! Why should we accept multiple choice questions as a given? I didn’t have any at O-level.

    I’m not in favour of multiple choice questions unless the question is phrased in such a way that one, and only one, answer could possibly be construed as correct. None of these questions satisfy that constraint, whatever part of whichever urriculum they’re intended to test. They’re just sloppy.

    I’d be happy to write an entire physics GCSE examination given an appropriately exorbitant fee….

    Peter

    PS. I don’t know what you mean by a mark scheme for multiple choice examinations.

  7. telescoper Says:

    Brendan,

    I notice that whoever wrote this paper has a thing about ultrasound. It keeps popping up as one of the choices.

    There’s also a point in Q30 that some scientists who believe in a multiverse might plausibly pick C…

    Peter

  8. Chris North Says:

    I’ve taken part in marking research for a GCSE exam board, and found it a very interesting experience – I could write a book about my thoughts on that experience, but I don’t have time. One of the major failings is that the students tend to be taught “soundbites”, upon learning which the answers to the questions are presumably more obvious than to someone who doesn’t know which soundbites to use…

    For example, Q1 would be more obvious (to me) by using 4 different sketches, one with just A, another with A+C, a third with just B, and a fourth that isn’t D (, a less eccentric elipse, or just get out the spirograph!). Bear in mind that this is a foundation question, where the goal is to identify students who know that a moon goes round a planet (or some other object).

    Q30 should be removed or completely re-written, as the whole premise for the question is flawed. I’m sure there’s a multiple-choice question about cosmology that would fit in – given sufficient time to think about it. Having whole-sentence answers would give more versatiliy as well.

    Q31 could be clarified by saying “most strongly supported” or somthing of that nature. After all, the composition of nebulae could be a factor in analysis of BBN, and even the age of neutron stars could in some cases probably be used to constrain the age of the Universe.

    Buzz-words and soundbites are, in my opinion, a major problem in school education at the moment. Not in the fact that they exist (they are extremely useful for getting people to remember things) but for the fact that the soundbite is meant to help you remember a fact – not just the fact itself.

    Having said that, writing exam questions isn’t trivial, and there are a few quite nice questions in there (I was pleasantly surprised at some). There’s one about units which is good, and a couple about the life cycle of stars.

    Another really apalling thing is the fact that equations are given. While I wouldn’t expect them to remember the Planck function or anything (that’s normally given at degree level), I would say that higher tier students (A-C grades) should not need to be told that speed = distance/time, or force = mass * acceleration. That’s exactly the sort of thing they should be expected to remember from exams. These questions become an exercise in English comprehension and the ability to push calculator buttons, rather than physics questions.

    Cheers,
    Chris

    • telescoper Says:

      Chris,

      I think Q1 could be a good question if it were written with a little more care. I agree that your suggestion of 4 separate diagrams is a good one. Q30 is not only based on a shaky premise, it hasn’t even been checked for basic grammar. Which of the following is … implies an answer in the singular, and “galaxies” is plural. Q31 could be also a sensible question with a few edits. But it’s been chucked together so sloppily as to be deeply unsettling.

      What’s really lacking in this paper is that very few of the questions involve any real thinking rather than just regurgitating soundbites. In that sense the examination is probably not the fundamental problem. The curriculum is. You can’t fix this just by having exams in a different format. I think multiple choice questions are fine if you have to solve a problem and pick one of four possible numerical answers, for example.

      Peter

  9. Brendan Says:

    Peter,

    There was a massive amount about ultrasound at A-level too, with a different exam board. I think there is some idea that it is a clear ‘real world’ application of physics and it needs to be pushed to the forefront of every student’s mind.

    Still, I think we’re mainly just being picky here; as Chris says, these questions are probably fine if you know the ‘sound-bite’. The problems that implies are far worse than the ambiguities a bunch of trained physicists can conjure up from sloppy questions.

  10. Hi Peter,

    Heh – yes, the mark scheme for a multiple choice would be rather trvial ;-) But this does highlight why multiple choice questions are used in examinations – they are very easy for busy, tired and (sometimes) non-specialist teachers to mark the papers. I’m not saying this justifies their use – it merely explains why exam boards use them.

    I guess my worry is that in the whole “science exams are rubbish” debate, it’s very easy to criticise what’s there without being constructive. I wasn’t asking for a whole GCSE paper, just one example of what you, as a scientist involved in the field, might have done.

    So, for example, I would have had Q31 the same, with the answers:

    * A) The velocity of galaxies, as measured by telescopes;
    * B) The relative abundance of elements observed in the Universe;
    * C) Measurements of the Cosmic Microwave Background radiation;
    * D) All of the above.

    With the answer D. The object of the question, for me, would have been to test if students have understood that, in science, we use data from many sources to support or reject a model. This would have been a “How Science Works” type question, using current topics in science.

    I doubt this question would be used (the language is probably too complicated, for a start!), but you can see how by offering up an example, you can tell how I’m thinking about the problem – which, as you rightly say, does urgently need addressing.

    Cheers, Tom

  11. Thanks Chris – that’s the sort of thing I meant. I also hadn’t looked at the paper for examples of the good questions. This is another problem with the debate – often the crap examples are derided, without a celebration of the good stuff.

    Cheers, Tom

  12. Ricardohere Says:

    I think one of the problems with this kind of sloppiness is that it militates against the more able students who might expect to receive a higher grade.

    For instance, a student who understands the ideas that lie behind the theory rather than simply remembering soundbites, may be dumbfounded by Qs 30 & 31 and possibly give a ‘wrong’ answer when in reality he or she could quite easily provide a correct explanation for the answer chosen, demonstrating a genuine understanding and, presumably, attracting a good grade.

    Viva-style examination is obviously not possible, but I feel that multiple-choice questions do not allow for the most able to demonstrate the breadth or depth of their knowledge & understanding, no matter how well-written.

    I know there are many reasons for the use of multiple-choice papers, but I do think they let down the more able in favour of a maximin educational strategy.

    (I do agree, however, that other questions on the paper are quite adequate!)

    As someone who didn’t take Physics ‘O’ Level all those years ago (because of option-clashes, I hasten to add!), I’m at least comforted to know that I’m confident I could tackle one of these papers with ease.

  13. Chris North Says:

    It would be very interesting to sit in on one of the panels writing these questions. A brief anecdote from the research I took part in:

    Question (paraphrased):
    Ahmed is doing an experiment ueing electrolysis with HCl. What are the a) positive and b) negative charge carriers in operation?

    a)______________ (1mark) , b) ________________ (1 mark)

    Acceptable answers: positive ions/negative ions, H+/Cl-, Hydrogen ion/Chlorine ion.
    Unaccptable answer: protons/electrons

    Despite our howled protests that you couldn’t allow “H+” but disallow “proton”, we were overruled! Apparently, students are taught that it’s ions, and that if they put electrons/protons they’ve probably got the wrong idea. No one put protons/Cl- so we couldn’t prove our point.

    The research was into whether you needed to be a physicist and/or a physicist to mark physics papers. What I took away was that you didn’t need to be a physicist to mark them, but that they should have actually got a physicist to write some of the questions!

  14. Garret Cotter Says:

    Peter – heartily seconded about the students. They are worked so heard and have to jump through countless hoops, and at the end of the game they know so little physics. I am getting very depressed by the increasing numbers who arrive at university to study physics and find it is not the subject they thought. I have seen some very very clever and hard-working people get depressed, disillusioned and drop out. These people are our future and they deserve better.

    So what _can_ we do? Who sets these papers? How does one get involved? Can we have any influence on the syllabus or at least on checking the papers?

  15. No wonder pupils opt for GCSE Leisure and Tourism instead (the classic figure is at http://web.aqa.org.uk/qual/appliedgcse/qp-ms/AQA-38401-W-QP-JUN06.PDF).

    Of course papers are set in response to the curriculum (which no doubt Gove will shake up as soon as). But the boards must have some freedom in the type of questions, no? Are they really just tailoring to the markers?

    So, who are the people who actually set the papers, and how do we get to them?

    For 30, what on earth is wrong with, “Give two pieces of evidence for the Big Bang (one mark each)”..? Is that so onerous? Are these exams designed to be done without any revision?

    In 31, what happens to Jas?

    Call in Colin Dexter I say.

  16. Garret, how dare you pip me.

  17. Garret Cotter Says:

    Sorry JZ. But you got your retaliation in – I started reading your link and I just wet myself.

  18. telescoper Says:

    I don’t know what kind of person works for these exam boards but they’re run for profit and compete with each other on the basis that the more students do well the more money they will make. The effect on standards has been entirely predictable.

    It has been suggested that universities should take over the business of setting national exams, but only A-levels:

    http://www.telegraph.co.uk/education/educationnews/7353466/Universities-should-set-A-levels-say-Tories.html

    I think this isn’t a daft idea but I think A-levels are so damaged now we’d be better off binning them and going for the International Baccalaureate which at least will show up how poor our standards are compared with other countries.

    But what is the alternative for GCSE? Since just about every European country now does science and maths education better than the UK, perhaps we should find out how it is done elsewhere? I’ll bet it’s not done by private companies.

  19. Peter: Excellent point re. the terrible questions being a symptom of a terrible curriculum – I think you’ve hit the nail on the head with this. A paragraph or so at the end of the post highlighting this point would have (rightly) made example questions irrelevant.

    Garret, jz: @AlomShaha (on Twitter) is a physics teacher (and friend of mine) who has put a lot of time and effort into answering these questions. If you want an idea of the nature of the problems, read the “Judging a book by its cover” chapter in “Surely You’re Joking Mr. Feynman”…

  20. Peter: Private schools are already going down the route of the IB and the IGCSE for precisely the reasons you’ve highlighted. Interestingly, it would appear state schools are now allowed offer them too:

    http://news.bbc.co.uk/1/hi/education/10258324.stm

    Perhaps not the answer, but a step in the right direction?

  21. telescoper Says:

    Tom,

    Indeed. The political reason for not embracing the IB in the state sector so far is that it would lay bare the flaws in our system and make it clear we’ve been deluding ourselves into thinking standards have been rising. Compared to the rest of the world they’ve undoubtedly been going in the opposite direction.

    Smart politicians in the new government could seize this opportunity to do some good for UK education while at the same time scoring political points off the people who let it get into such a state.

    Peter

  22. Peter – re universities, aren’t we just going round in circles? e.g. Oxford, Cambridge Exam Boards -> OCR -> AQA -> … Or is the Telegraph’s suggested scheme different to what used to happen?

    Are you referring to a specific IB? Some of these are set by the exam boards, but perhaps there isn’t the (alarming!) market pressure on the IB..?

    Trebles all round.

    PS Garret – slightly more information.

  23. Anton Garrett Says:

    So who DOES set these questions? (I can see why they want to remain anonymous, but I mean the question seriously.)

    Multiple choice is probably popular today because of speed of marking – it can even be done by a computer. It’s not so bad provided that plenty of calculational questions are included, with the ‘wrong’ answers corresponding to likely howlers.

    Anton

  24. Anton – someone called the “Principal Examiner” I think. We should flood the place with applications – http://web.aqa.org.uk/over/vacexp_vacancy.php. The only setting job I can see is for GCSE Expressive Arts, but a number of candidates have ably demonstrated their abilities in this very discussion.

    I can’t find any names (perhaps for good reason), except this one:

    http://iopscience.iop.org/0031-9120/19/4/318/pdf/0031-9120_19_4_318.pdf

  25. Peter,

    A perceived difficulty with the political route is that sometimes it’s unclear who would be scoring the points. Does science education come under the jurisdiction of the “science” minister or the “education/schools” minister? It’s easy to say that it’s both – and just as easy for the respective departments to fob you off to the other.

    Alternatively, you could see it as an opportunity. I don’t know what you make of the #scivote movement on Twitter, but that appears to have a bit of momentum behind it that could be harnessed. For example, it would be interesting to see what David Willetts MP thinks about how pre-university education is affecting science in universities.

    Cheers, Tom

    • telescoper Says:

      Tom: By scoring points, I meant that the new government could blame it all on the previous lot and use this as justification for making a fresh start while further discrediting their predecessors…

      Richard: If that’s what the QCDA was supposed to do, nobody is going to miss it because it clearly wasn’t doing it.

      Anton: I agree that MCQ can be very good for maths and some elements of physics. But these questions are very good illustrations of how not to test physics using them.

  26. Ricardohere Says:

    A couple of sources re: advantages / disadvantages of MCQs (one Wiki, the other more reliable):

    http://en.wikipedia.org/wiki/Multiple_choice#Advantages

    http://www.ukcle.ac.uk/resources/trns/mcqs/index.html

    If MCQs are to be used, they need to be set by people who know how to set good MCQs (& preferably trialled to weed out ambiguities not obvious to the setter) – this has obviously gone wrong in many cases.

    I’m not sure the problem can be laid wholly at the door of the curriculum, although it certainly needs revisiting in some cases.

    As to the IB programmes, I can’t see any problem with adopting them, but am still staggered at how the UK isn’t able to provide better examinations than the shoddy example given here; is there no QA process? I was under the impression that that was what QCDA et al. had been for.

  27. Peter: Absolutely – and I think this is exactly what the scrapping of QCDA Richardohere is all about, with the added bonus that the QCDA has been scrapped. This article is just a summary, but the QCDA rep in the actual debate was pretty useless, prepped with management speak, and largely failed to answer many of the questions asked of him:

    http://www.timesonline.co.uk/tol/news/science/eureka/article7042478.ece

    The points here clearly go to the education minister – which is fine, as the QCDA was perceived as a problem for all subjects. So one approach may be to lobby for practising experts in the associated field to act in a sort of “peer review” capacity to make sure the bad questions don’t slip through the net (assuming, of course, that this doesn’t happen already…!).

    The point remains, of course, that there are good questions in there, and I don’t think we should focus entirely on the bad ones (is “many cases” fair?). Though one bad question is obviously one too many.

    Cheers, Tom

  28. That should have read “scrapping of QCDA, as mentioned by Richardohere,”. Whoops!

  29. Ricardohere Says:

    8oD!
    No problem Tom!

    I couldn’t agree more that QCDA wasn’t fit for purpose, so it’s good to see the back of it in its previous incarnation; I think the QCF needs a good looking at as well.
    However, as with many such things; they’re both intrinsically good ideas sullied by poor execution. I suspect we need some good QA & curriculum development more than ever at the moment.

    Again, Tom, you’re right that there are many good questions in there as well, but shoddy bad questions deserve some focus; not only do they demonstrate the lack of QA, they can also mean the difference between grades for a student.
    (Pure ‘anecdata’, but I suffered from it recently in a college course; I explained afterwards why I’d given the answers I had, thus demonstrating a clear understanding of the subject, but as they weren’t the ‘expected’ answers, I still dropped a grade).

    Fair point about my use of ‘many’.
    I don’t have the evidence to back that up at all!

    @telescoper
    MCQs may well be OK for some elements of maths, but I still remember the days of ‘showing your working to gain partial credit’ with affection.

  30. Alan Heavens Says:

    Peter,

    This is an example of a broader worry which I have about declining standards in education. The first feature which appears is that people coming out of school are not well prepared. The next feature is that people who have been through a poor education system are then found in decision-making roles (e.g. in assessment, as here, or job appointments). The decisions become unreliable, and this problem is much harder to fix, as the assessments/appointments themselves become unreliable and propagate the problem.

    Alan

  31. [...] science education laid bare. Indeed was so staggered to discover how poorly conceived the current GCSE science examinations are that I forgot that I’d already blogged about the lamentable tendency of the modern [...]

  32. Well this certainly doesn’t bode too well for the future of science in this country, although I do wish the exam had been that easy when I was taking it.

  33. [...] we have an article here about the exams regulator proposing that they’re too easy and a blog post here from a month earlier in time saying the same [...]

  34. [...] You might want to compare this with a recent example of an Edexcel GCSE (Multiple-choice) Physics paper, about which I have also posted previously. [...]

  35. [...] you don’t believe this, take a look at this GCSE Science Examination. A truly intelligent student would struggle to find any correct answer for many of the questions on [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 3,269 other followers

%d bloggers like this: