Essays and (Computational) Physics

There have been more news stories about ChatGPT and assessment in universities going around. There’s one here from The Journal and another here from The Conversation to give just two examples.

I wrote about this myself a couple of months ago in a post that included this:

I have to admit that I’ve never really understood the obsession in some parts of academia with “the student Essay” as a form of assessment. I agree that writing skills are extremely important but they’re not the only skills it is important for students to acquire during the course of a degree. Of course I’m biased because I work in Theoretical Physics, an area in which student essays play a negligible role in assessment. Our students do have to write project reports, etc, but writing about something you yourself have done seems to me to be different from writing about what other people have done. While forms of assessment in science subjects have evolved considerably over the last 50 years, other domains still seem to concentrate almost exclusively on “The Essay”.

Whatever you think about the intrinsic value of The Essay (or lack thereof) it is clear that if it is not done in isolation (and under supervision) it is extremely vulnerable to cheating.

A few people have retorted that communication skills are very important in higher education. I agree with that wholeheartedly, but it seems to me that (a) there are other ways of communicating than via formal essays and (b) there are, should be, more to academic study than  writing about things.

That said, I do think we could be doing more in some disciplines, including my own, to cultivate communication skills in general and writing skills in particular. In Theoretical Physics we certainly don’t do this as much as we should. I do have a project report in my 3rd Year computational physics module, but that is a relatively short document and the report itself counts only one-third of the marks (and the project is only 40% of the module mark).

These thoughts somehow reminded me of this. You can click on it to make it bigger if it’s difficult to read. It was the first paper (called colloquially Paper Zero) of my finals examination at the University of Cambridge way back in 1985, getting on for 40 years ago:

As you can probably infer from the little circle around number 4, I decided to write an Essay about topic 4. I’ve always been interested in detective stories so this was an easy choice for me, but I have absolutely no idea what I wrote about for three hours. Nor do I recall actually ever getting a mark for the essay, so I never really knew whether it really counted for anything. I do remember, however, that I had another 3-hour examination in the afternoon of the same day, two three-hour examinations the following day, and would have had two the day after that had I not elected to do a theory project which let me off one paper at the end and for which I got a good mark.

Anyway, to get back to the essay paper, we certainly don’t set essay examinations like that here in the the Department of Theoretical Physics at Maynooth University and I suspect they no longer do so in the Department of Physics at Cambridge either. At the time I didn’t really see the point of making students write such things under examination conditions but then we didn’t have ChatGPT way back then. No doubt it could generate a reasonable essay on any of the topics given.

I am skeptical about whether any of my 3rd year computational physicists would use ChatGPT to write their reports, but they might. But ChatGPT can write Python code too. Am I worried about that? Not greatly. I’ve asked it to write scripts for the various class exercises I’ve set so far and the code it has produced has usually failed. It will get better though….

Essays, Mills and ChatGPT

I saw an article the other day about how “contract cheating” was endangering the integrity of Ireland’s universities. This refers to the problem of students outsourcing their assignments to professional essay mills. Given the enthusiasm that university managers have for outsourcing in other contexts I’d be surprised if they see this as a problem. Indeed, their response might well be to outsource the grading of assignments in a similar fashion. It does however raise questions about academic integrity for thus of us who care about such matters.

I have to admit that I’ve never really understood the obsession in some parts of academia with “the student Essay” as a form of assessment. I agree that writing skills are extremely important but they’re not the only skills it is important for students to acquire during the course of a degree. Of course I’m biased because I work in Theoretical Physics, an area in which student essays play a negligible role in assessment. Our students do have to write project reports, etc, but writing about something you yourself have done seems to me to be different from writing about what other people have done. While forms of assessment in science subjects have evolved considerably over the last 50 years, other domains still seem to concentrate almost exclusively on “The Essay”.

Whatever you think about the intrinsic value of The Essay (or lack thereof) it is clear that if it is not done in isolation (and under supervision) it is extremely vulnerable to cheating. The article I referred to above concentrates on the corrupting influence of “Essay Mills” who will produce – for a fee – an essay on a given topic.

I believe however that this is not the biggest threat to academic integrity. There is a lot of discussion going on these days about ChatGPT, an AI system that can produce text on demand using sources on the internet. This is not great at dealing with technically complex specialist topics but can produce plausible if somewhat superficial offerings in many circumstances where something less demanding is required. Indeed, the more banal the task the better ChatGPT does. For example, here is an AI version of a university Strategic Plan which captures the vacuous nature of such documents with uncanny accuracy:

According to a pilot programme of which I am aware, the present configuration of ChatGPT scores an average of about 70% on (short) student essay tasks. It won’t be long before it gets much better than that. I predict that something like it will soon put essay mills out of business as it will be far cheaper and its use far more widespread. This is the real threat to the viability of “The Essay” in a modern university. The response will need to be quick!

Update: piece from the Irish Times.

P.S. It’s worth mentioning that AI systems can also write simple computer code to a reasonable level of proficiency, so this vulnerability also affects programming tasks.