Do you have Confidence in the Teaching Excellence Framework?

The  Teaching Excellence Framework (TEF) is, along with a number of other measures in the 2016 Higher Education and Research Bill, causing a lot of concern in academic circles (see, e.g., this piece by Stephen Curry). One of the intentions of the TEF is to use relatively simple metrics to gauge “teaching quality” in higher education institutions. On top of the fundamental questions of exactly what “teaching quality” means and how it might be measured in any reliable way, there is now another worry: the whole TEF system is to be run by people who are statistically illiterate.

To demonstrate this assertion I refer you to this excerpt from the official TEF documentation:


The highlighted “explanation” of what a confidence interval means is false. It’s not slightly misleading. It’s not poorly worded. It’s just false.

I don’t know who from HEFCE wrote the piece above, but it’s clearly someone who does not understand the basic concepts of statistics.

I can’t imagine what kind of garbled nonsense will come out of the TEF if this is the level of understanding displayed by the people running it.  That garbage will also be fed into the university league tables with potentially devastating effects on individuals, departments and institutions, so my gripe is not just about semantics – this level of statistical illiteracy could have very serious consequences for Higher Education in the UK.

Perhaps HEFCE should call in some experts in statistics to help? Oh, no. I forgot. This country has had enough of experts…






2 Responses to “Do you have Confidence in the Teaching Excellence Framework?”

  1. John Peacock Says:

    Peter: what did you expect them to do – take a Bayesian approach? The whole point of TEF and REF is that they represent a rejection of the idea that anyone can have prior knowledge regarding which universities are best at what they do. So “enough of experts” in this context means that any argument invoking priors would be deemed politically unacceptable.

    On a more optimistic note, we can at least give a little bit of praise for an acknowledgement that the results of TEF are imprecise. Nothing like that was ever seen in REF, violating the basic scientific imperative that meaningful measurements must always come with an error bar. Thus we had all this nonsense about REF league tables when it was obvious that the figures being ranked were so close together that they mostly had to be within 1sigma. And while we can sneer at TEF’s naive 1.96sigma threshold, even Bayes would find it tough to turn a 1sigma difference into a real effect.

    • telescoper Says:

      I don’t expect much, but being able to define the meaning of the concepts they’re using would seem to be a minimum requirement.

      Whether you think confidence levels are useful or not, they do not mean what is stated in the document.

      Other than that, I agree that it is an improvement that they at least acknowledge the existence of some form of measurement error, so in that sense it is a step forward.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: