The Culture of Over-Assessment in STEM

This afternoon I went to yet another meeting about assessment and feedback in University teaching involving members of staff and students from the School of Physics & Astronomy here at Cardiff University as well as some people from other schools and departments. Positive though this afternoon’s discussion was, it didn’t do anything to dissuade me from a long-held view that the entire education system holds back the students’ ability to learn by assessing them far too much. This is a topic that I’ve blogged about a few times before over the years (see, e.g., here) but given that the problem hasn’t gone away (and indeed is probably going to get worse as a result of the Teaching Excellence Framework which the Westminster government is trying to impose on universities),  I make no apologies for repeating the main points here.

One important point we need to resolve to pin down essentially what is meant by “Research-led Teaching”, which is what we’re supposed to be doing at universities. In my view too much teaching is not really led by research at all, but mainly driven by assessment. The combination of the introduction of modular programmes and the increase of continuously assessed coursework has led to a cycle of partial digestion and regurgitation that involves little in the way of real learning and certainly nothing like the way research is done. I don’t know why we’ve got into this situation but it can’t be allowed to continue.

I’m not going to argue for turning the clock back entirely but, for the record, my undergraduate degree involved no continuous assessment at all (apart from a theory project I opted for in my final year. Having my entire degree result based on the results of six three-hour unseen examinations in the space of three days is not an arrangement I can defend, but note that despite the lack of continuous assessment I still spent less time in the examination hall than present-day students.

That’s not to say I didn’t have coursework. I did, but it was formative rather than summative; in other words it was for the student to learn about the subject, rather for the staff to learn about the student. I handed in my stuff every week, it was marked and annotated by a supervisor, then returned and discussed at a supervision.

People often tell me that if a piece of coursework “doesn’t count” then the students won’t do it. There is an element of truth in that, of course. But I had it drummed into me that the only way really to learn my subject (Physics) was by doing it. I did all the coursework I was given because I wanted to learn and I knew that was the only way to do it. I think we need to establish that as a basic principle of education in physics (and similar subjects).

The very fact that coursework didn’t count for assessment made the feedback written on it all the more useful when it came back because if I’d done badly I could learn from my mistakes without losing marks. This also encouraged me to experiment a little, such as using a method different from that suggested in the question. That’s a dangerous strategy nowadays, as many seem to want to encourage students to behave like robots, but surely we should be encouraging students to exercise their creativity rather than simply follow the instructions? The other side of this is that more challenging assignments can be set, without worrying about what the average mark will be or what specific learning outcome they address.

I suppose what I’m saying is that the idea of Learning for Learning’s Sake, which is what in my view defines what a university should strive for, is getting lost in a wilderness of modules, metrics, percentages and degree classifications. We’re focussing too much on those few aspects of the educational experience that can be measured, ignoring the immeasurable benefit (and pleasure) that exists for all of us humans in exploring new ways to think about the world around us.

Advertisements

11 Responses to “The Culture of Over-Assessment in STEM”

  1. As one sitting in the same meeting today I completely agree that we have far too much “summative” continual assessment in our modules. It’s part of the current insane preoccupation with measurement, questionnaires and league tables, which the Teaching so-called Excellence Framework will make even worse.

    But I’m not sure we can change things too much: we might easily get a lower score on the NSS “assessment and feedback” question if we have less of it. That shouldn’t really be a motivation, but it is. The very questions chosen in the NSS are designed to set the agenda – it’s all about control.

    True, in an idealised view of university education, every student would be self motivated to learn. Like you, I had only formative assessment when a student at university. It was also the norm when I joined Cardiff as a lecturer. Sadly, many students I talk to nowadays say that they will only work through problems if they get marks “which count”. (And they are very concerned about every fraction of a mark.) Their expectations have been distorted by the continual pressure to get marks. I hope we *can* somehow change this, and we should try, but the lack of real control by those doing the teaching and researching in universities doesn’t give me cause for optimism!

  2. John Peacock Says:

    Peter: I agree 100%. And to the charge sheet should be added the increasing tendency to examine courses instantly: in Edinburgh, we’re currently fighting a proposal from the centre that 50% of courses should be examined prior to Christmas, in every year. This is educational madness, as it leaves no time for understanding to mature, or for connections between courses to become apparent. Feeling such connections emerge during bulk revision for finals over the Easter holidays was and remains one of the greatest joys of my academic career.

    Now it will be argued that we are unusual, having ended up as academics, and that typical students even back whenever didn’t need this treatment, never mind now. It’s hard to rule this out, but I still end up feeling that more time to think has to be a good thing, and that marked coursework should be discouraged in favour of formative work – we’ve all seen the narrowness that comes from the focus on marks.

    So how to get students to graft, if not by offering marks? You paint a noble picture of being motivated by a desire for understanding, but I have to admit that this wasn’t why I worked as a student – it was more a question of not wanting to look dumb in supervisions. This is where the Oxbridge system really scores: the merciless concentration of a single expert tutor on just you and a partner for several hours a week. There’s no hiding place, and not doing the homework just isn’t an option. But at Edinburgh our groups are bigger, so students can coast along quietly or absent themselves without it being noticed.

    So this is the only true way to get high NSS scores: give the students more individualised attention from tenured academics or at least senior postdocs, and then will learn and be happy doing so. The trouble is that we spend less time with students than we might. You could say that this is laziness, or a selfish concentration on research. But the truth is that there are so many other things that steal our time, which are neither student contact nor thinking about research (from REF down to home-grown bureacratic idiocies) – so people are understandably reluctant to donate more contact hours lest research time shrink to zero. The only solution is to get the system off our backs: there are plenty of hours in the week to give students the contact they need and get research done – we just need to be able to concentrate on these priorities.

    • telescoper Says:

      Yes, one of the points I made at the meeting yesterday is that we also spend a huge amount of time marking and setting coursework that is of very little benefit to the students – and they spend a huge amount of time doing it too. I’m sure we could ditch a lot of this and concentrate more on the personal contact which is actually vital.

      Here at Cardiff we do small-group tutorials in Years 1 and 2 (usually 4 or 5 students), though many places have scrapped this I think they’re vital. On the other hand students also say it depends a lot on the tutor – some are better than others at getting students involved. Also we don’t have similar things in Years 3 and 4.

      I’m going to be teaching an MSc course next term on The Physics of the Early Universe. It will be a relatively small class (~20 students) so I might try adding some small-group activities in.

      P.S. I don’t think I was ever particularly scared of looking stupid in tutorials. I just assumed that I would!

  3. Brian Schmidt Says:

    Peter: Here Here!

    In my Cosmology class at ANU I made the assessment of class entirely from a set of research-styled problems to solve – where students have to work on substantive problems from beginning to end – and use tutorials to figure out how to do them over the course of 2-3 weeks. Most of the students never quite figured out that the problems sets were not to assess the lecture material , and many object – but I still reckon it is the best way for them to actually learn cosmology.

    Hard part is creating problem sets which they can solve that are interesting…

    This isn’t going to be so useful necessarily in first year – but it is one tool to limit the time wasted on assessment.

    With respect to John’s notes on small-group tutorials. I agree that the Oxbridge small-group tutorial provides an amazing incentive to work as one goes, but it is not scalable to 40% of the population – and still requires a crazy career-defining exam at the end. Although its is hard to argue that is is not effective at creating some of the greatest academics in the world.

    Can we figure out how to use the advantages of it, scale it up, and perhaps use some assessment along the way to remove the issues around the final exam. I think whatever we do, to get scalability, will need to use the students themselves to help each other learn, with a way of elevating the more sophisticated issues to the Prof.

    Motivating students to keep up remains a problem, peer-to-peer assessment is one way (e.g. students judge each others knowledge of material in each tutorial rank-order) and this can be augmented by input from Prof and quizzes/exams to normalise these marks, and this can form part of the student’s assessment. I personally think getting assessment in smaller bits along the way is better than a big exam at the end. But I have no data to support this view.

    • telescoper Says:

      I think it’s absolutely wonderful that you still find time to teach when you’re a VC!

      • Brian Schmidt Says:

        Sadly – I had to give up teaching this year as VC… But I am looking at how I might contribute next year at least a bit.

  4. I agree with the general point about the small group tutorials as a way of engaging with students vs. the testing/grades approach. But this style of teaching doesn’t work well for everyone either. I’m pretty sure I would struggle as I’m not very good at talking-while-learning and need a lot of time to digest things; and my experience with leading small group tutorials in Oxford was that some students thrive and others completely shut down.

    I would also worry that students from poorer backgrounds, women, minorities would feel less confident and find it hardest to adjust to this style of teaching – especially as our senior academics are still predominantly white & male. A tutor can be an excellent role model (my 2nd year tutor became my 4th yr project supervisor, then my PhD supervisor!), but can also demolish someone’s confidence and ambition with one careless comment. In that sense, more formal assessment in the form of tests & exams might be better at keeping a level playing field.

    • John Peacock Says:

      Some fair points here, and I wouldn’t suggest that there’s a universal best solution. Discussions with peers can certainly be instructive, and yes tutorials can be intimidating as much as illuminating for what the tutor says. I suppose my point was really that there’s no substitute for personal graft. You learn most when you meet a problem you can’t do at first, but keep battering away until you find a way round it. Fear of tutorial embarrassment was an effective way of making me put in the necessary hours.

      • There is no “right” way to learn: each student must find out what works for them. Likewise there is no “right” way to teach: we should be trying to offer as wide a range of activities as possible to allow students the best chance of finding what works for them.

        For this to work, though, students have to be sufficiently motivated to learn, and I don’t think giving marks is the right way of doing this.

  5. When I studied physics and astronomy in Hamburg, about a quarter of a century ago, I thought (and still think) that the assessment system was ideal.

    The final mark is based on that of the thesis (a 1-year thesis, so the first degree (and only degree before the doctorate) back then was a bit more than the equivalent of a master’s thesis, say), counted twice, and the marks in four oral exams, counted once each. The four exams were in theoretical physics, the minor (mine was astronomy), the structure of matter (one could choose two of nuclear physics, particle physics, solid-state physics, atomic physics, etc; I chose particle physics and nuclear physics), and an elective (I had advanced nuclear physics). (Hamburg was a natural place for particle physics, with DESY doing mostly basic research back then, and this jibed well with my interests, as was the case even more so with the observatory.)

    The oral exams were just half an hour, but, apart from the topic and the fact that it was covered in the corresponding lectures, there were no restrictions on the questions. The professor asked the questions, and someone else (anyone with a degree, but usually some postdoc or assistant professor in the professor’s group) took notes (and is allowed questions and comments as well, though this was rare, and was usually to clear up some misunderstanding or whatever). “Is it in the exam?” was thus not a relevant question. Most professors lectured from their own notes, but usually recommended some standard textbooks covering the same ground.
    (At least back then, all lectures were in German, but the recommended books were about half and half English and German, the latter including some translations of famous textbooks in other languages (Goldstein, Messiah, etc—they are so famous I don’t even need to mention the titles), but of course one could and some did use the originals.)

    After the exam, the professor and the note-taker would discuss the mark, then it would be told to the student (and fed into the (then paperwork) university system). I was note-taker probably 12-15 times over the years. Based on that, the marks were consistent and I essentially always agreed with the examiner. (The fact that people could—and many did—sit in on exams prevented outliers.)

    I think that one can assess a student’s knowledge quite well in half an hour. Obviously, there isn’t a set list of questions; if it is obvious from the answer to the first question that the student has this down pat, one can jump to harder questions, and so on. It is also obvious if something is simply being repeated from memory or the answer displays real understanding.

    As long as the student being examined didn’t object, one could sit in on such an examination.

    In order to be able to be examined, and to be able to start the thesis, one had to collect, for each examination (or thesis), 4–6 certificates. A typical certificate was awarded if one got at least a certain number of points on an exam and also at least a certain number of points for homework problems. These were done in tutorials (usually conducted by a professor; teaching assistants did the marking) accompanying the corresponding course. Sometimes the professor teaching the course would also have one of these tutorials. The exam lasted about a couple of hours.

    Thus, in order to be able to sit the oral exams later on, one had to demonstrate an appropriate level of understanding based on written solutions to rather complicated problems. (Typical homework assignments were perhaps half a dozen problems taking 2–3 hours to solve.) However, the marks here, for the homework and for the exam, were irrelevant for the final mark in the thesis. Once one had the certificate, which carried no mark, the marks on which it was based became irrelevant. The advantage here is that the only way to the degree is through the exams, but in order to be admitted to the exams one has to prove some real understanding; at the same time, as long as one passed and got the certificate, the mark didn’t matter, so this pressure was gone. Homework problems could be given up by 1, 2, or 3 people, so those who wanted to work in small groups could. (The exam was still there to gauge individual knowledge and sift out freeloaders.) If one didn’t get the certificate, one could try again later, taking the corresponding tutorial again.

    So, for each major lecture, there was a corresponding tutorial. There was low-level assessment in terms of marked homework problems at the weekly level and in terms of exams at the semester level. So one knew how one was doing. At the same time, since the marks didn’t matter, there was no unnecessary pressure. And cramming and then forgetting wasn’t a strategy: before the degree were the oral exams.

    There were also certificates awarded for giving a seminar talk, for labs (done in the lecture-free time, which is also when most people took the oral exams), and so on. The oral exams could be done before the thesis, after the thesis, or two before and two after, but not during. Of course, one could (and I did) get more than the minimum number of certificates required to qualify for the exams.

    Before all of this started, in the last three years of the nominal five, there was a similar programme leading to the Vordiplom, a sort of internal half-time degree, probably the equivalent of a bachelor’s degree today (but it had no significance outside the university; its sole purpose was that it was a requirement to continue on towards the final degree (and let those who didn’t have what it takes know soon enough, rather than wasting time)). Same system, but 5 oral exams, including maths (in the maths department—in addition to the “maths for physicists” lectures, which were more practically oriented) and no thesis.

  6. Penny A Gowland Says:

    Absolutely agree.. students are too stressed and cant see the big picture. It is no longer a case that ‘they wont work if we don’t make them’- they are working too hard and don’t spend any time assimilating things. Its like this from the age of 5 til the age of 22.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: