As a young teacher in London, I found the work of Sir Tim Brighouse and the London Challenge profoundly inspirational for several reasons.

Firstly, I was privileged to work within a school system that was well-funded and experiencing rapid improvement. This had an undeniable impact on my understanding of the power of a high-quality schooling experience; academic and pastoral.

This dynamic and innovative environment fostered the exploration and implementation of sensible strategies aimed at raising educational standards for all pupils.

Secondly, as a middle leader tasked with driving improvements, I eagerly anticipated the annual release of the Families of Schools data.

Healthy competition and collaboration

This resource enabled meaningful comparisons between our school and others with similar contexts or within the same school “family,” offering invaluable insights into performance metrics.

The transformative impact of the London Challenge has been extensively documented. By 2024, London schools outperform all other regions in terms of attainment and progress.

While other contributing factors undoubtedly play a role, the groundwork laid in the early 2000s was instrumental in achieving this success.

Unfortunately, the original spirit of healthy competition and collaborative effort has been largely supplanted by a more reductive, one-size-fits-all approach to assessing and understanding the complexities of the school system.

While there have been undeniable advancements in professionalism, curriculum development and pedagogy, the analytical tools employed to examine and enhance the education system remain limited and, arguably, imprecise.

It was therefore a breath of fresh air to hear the secretary of state speak of the need for healthy, competitive collaboration when introducing the Children’s Wellbeing and Schools Bill.

‘A data revolution’

To achieve that, though, we need a data revolution in the system.

Earlier this year, I sought to replicate elements of the Families of Schools data from the London Challenge but adapted to align with contemporary metrics.

Using tools developed within Lighthouse, a free data analytics tool I developed during Covid, we visualised national secondary school data through a scatterplot that categorises schools based on two key metrics: Progress 8 (on the X-axis) and a three-year rolling Attainment 8 score (on the Y-axis).

This approach yielded four quadrants, with the most critical being schools exhibiting both high Progress 8 scores and consistent increases in attainment over three years.

We termed these institutions “Dynamic Schools” - those demonstrating sustained improvement in attainment alongside strong pupil progress.

These “Dynamic Schools” exist across all regions and serve diverse communities, encompassing varying levels of disadvantage, special educational needs and disabilities (SEND), and prior attainment. They include academies and maintained schools alike.

Intriguingly, many of these high-performing schools have relatively low public profiles.

Their success lies not in notoriety but in the steadfast commitment of their headteachers, leaders and educators to deliver exceptional outcomes, equipping their pupils with the tools to seize future opportunities.

Uncovering these schools is critical to policy-making that drives improvement in the sector. But the government faces critical challenges in refining education metrics and infrastructure to achieve this aim accurately.

Harness the power of new technology

While Progress 8 remains valuable and should be retained, it needs to be succeeded by a more nuanced progress measure. And before redefining metrics, it is imperative the Department for Education invests in robust data infrastructure.

A more comprehensive and accessible architecture would empower school leaders, sector experts and policymakers to access and analyse information more effectively.

By harnessing advancements in analytics and artificial intelligence, we can transform how education data informs decision-making, ultimately driving greater equity and excellence across the system.

Through innovation, collaboration and a commitment to evidence-based practice, we can recapture the pioneering spirit of the London Challenge and apply it to the evolving educational landscape.

Andrew O’Neill is headteacher of All Saints Catholic College

For the latest education news and analysis delivered every weekday morning, sign up for the Tes Daily newsletter