Book review: The Tyranny of Metrics

The Tyranny of Metrics

Jerry Muller

Princeton University Press

240pp, £20.00

ISBN 978-0691174952

Over the past few decades, our education system has become a data-driven business as successive governments from both ends of the political spectrum have sought ways to measure and quantify the performance of pupils, teachers and schools alike.

In this thought-provoking book, Jerry Muller uses the term “metric fixation” to describe how we live in a world that has seemingly become obsessed with measuring, publicising and rewarding (or, indeed, punishing) performance. As Muller suggests, the book is fundamentally about the unintended consequences of such an approach.

In an explanation of what is known as Campbell’s or Goodhart’s law, Muller argues that even if a metric starts out as a useful measure, as soon as high stakes are attached to it, people will almost inevitably seek to game it. Gaming, he suggests, can take a variety of forms. Teaching to the test and narrowing the curriculum are used by the author as familiar examples of how this manifests itself in schools.

At the more extreme end of the spectrum, he points to how people can end up cheating or engaging in behaviour that is at odds with the ethos of their profession. While most of the examples he provides are drawn from a US context, one doesn’t have to look too far to find their English equivalents. The current issue of off-rolling - taking low-performing children off the register via informal exclusion - in secondary feels like a perfect example of this more extreme response to the pressures of performance indicators.

Muller also says that an over-reliance on metrics can lead us to disproportionately value the things that are easiest to measure. Again, it is very easy to find examples of this in our own education system. Ask any head, and they will tell you that they know it is the data that “really matters” when it comes to how their school is judged. Pupils’ Sats scores or GCSE results can be easily measured and analysed, and they therefore take on an increased value, whereas a school’s contribution to a child’s social and emotional development is harder to quantify and so this aspect of their work becomes undervalued.

These and the many other criticisms of metric fixation the author offers are well argued and will feel all too familiar to teachers and school leaders alike. They raise important questions, not just about if and how we should use data to judge school effectiveness but also about the consequences of linking teachers’ pay to the performance of their pupils.

Despite the strong criticisms Muller makes of metric fixation, this is not the polemic against the use of data that the reader might expect given the title of the book. He is clear that, used properly, measurement can be a good thing, and the last chapter is dedicated to looking at when and how metrics can be used effectively. When looking at schools, he explores how standardised tests, used in a low-stakes manner, can be useful to teachers. His argument is not that metrics are bad per se, rather that they have become overvalued and, at times, misused.

Potential readers should be aware that this is not a book just about schools and education. The author looks at how metric fixation has had a negative impact across a range of sectors, including medicine, policing and the military. What is striking is how similar these unintended consequences are. Whether it’s a surgeon deliberately avoiding operations that are unlikely to be successful in order to protect her “stats” or the police officer fudging the numbers to make it appear that crime is falling in their region, the message is consistent: high-stakes metrics inevitably pervert the behaviour of those subject to them.

Overall, the book makes a compelling case for why we should place less faith in metrics when it comes to measuring performance at a teacher and school level. However, I feel that he puts too much faith in human judgement as an alternative. While metrics are evidently flawed, we know that human judgement is also subject to all sorts of limitations and inherent bias. One only has to read Daniel Kahneman’s example of how Israeli prisoners’ chances of receiving parole fluctuated by as much as 60 per cent, depending on whether or not the judges concerned were hungry, to know that human judgement is not without its own significant flaws.

Shortly after I agreed to review this title, Ofsted’s chief inspector, Amanda Spielman, gave a speech explaining how she had recently read the book and how it was influencing her own thinking. Having now had the chance to read it myself, I think we should take this as a positive sign. My hope is that others involved in school accountability, including politicians, have the chance to consider its core message.

James Bowen is director of the NAHT Edge middle leaders’ union and a former head of an ‘outstanding’ primary. He tweets @JamesJkbowen