Where do you start when trying to turn around a struggling school? An obvious approach might be to look to the “outstanding” school down the road for answers. Analysing the policies and practices of the staff at this more successful school will surely provide a model that can be used to improve your struggling school. This sounds like a perfectly sensible thing to do. If you see something that is working, analysing what people are doing “right” will surely help you to learn the secrets of their success. But there is a catch: a phenomenon called “survivorship bias”.

The problem is that, in the situation described above, the school you are studying is not a random choice but one that has already “survived” selection. If we want to understand what truly works when it comes to school improvement, we need to be not only looking at those schools that are already “outstanding” but also studying schools that are getting it wrong - and comparing the two.

Let’s take another example. In Anders Ericsson and Robert Pool’s 2016 book Peak: secrets from the new science of expertise, Ericsson describes his decades’ worth of research into the practices of “experts”. One of his most famous studies involved top violinists. He suggested that the crucial quality these musicians had in common was that they engaged in “deliberate practice”.

This did not simply mean that they did a lot of practice; it also referred to how the practice was done - i.e., taking a systematic approach that is specifically focused on improving performance.

Deliberate mistake?

The concept of deliberate practice has since been seized on by many in education as the key to improving learning. However, in my view, there are several convincing criticisms of the theory, including those highlighting the fact that only elite experts - people who had “survived” right to the end of the practice pipeline - were studied.

The problem with this approach is that it gives only half the picture. To really know what works, we would also need to compare the experts’ performance with that of other violinists who did not use deliberate practice.

And there is another issue with only looking to the most successful for guidance: how can we be sure that they still remember what it means to be a novice accurately enough for their advice to be really useful? After all, could you explain with certainty how you first acquired the skills you are good at now? This is known as the “curse of knowledge” - and it is yet another reason why we should not rely entirely on experts to provide all the solutions.

So what does this mean for our scenario of school improvement? Well, to start with, every time we are tempted to look to examples of “outstanding” schools, which seem to embody all the features that we would like to mimic, we should be mindful that survivorship bias might come into play.

Instead of relying on these examples alone, we should also look to the schools that haven’t yet got everything right. To distil the secrets of best practice, we must analyse not only positive examples but also negative ones. We must consider not only the qualities of the end result but also the steps along the road to expertise.

Christian Bokhove is associate professor in maths education at the University of Southampton

This article originally appeared in the 1 February 2019 issue under the headline “Simply the best isn’t good enough”