Who should take the blame for England’s move, in the past 20 years, towards an education system that focuses quite so relentlessly on success in the exam hall? Some people, including me, have in the past laid responsibility largely at the feet of the politicians who oversee this system.
But recently, I have begun to wonder if some academics do not also accentuate this tendency in the way they frame investigations about whether or not our education system is succeeding. This happens through the questions they ask and, crucially, the questions they do not ask.
What really got me thinking last week was a study by respected researchers at Bristol University, which purported to use test and exam data from England and Wales to show that the decision nine years ago to scrap secondary league tables in Wales had been a bad thing.
The academics argued that a “natural experiment” had taken shape in the two countries. Until Welsh devolution in 1999, England and Wales had “practically identical” education systems, the study said. In 2001, Wales dispensed with secondary league tables, but otherwise little else about Welsh education changed at that time.
It was possible to use sophisticated statistical modelling to compare secondary exam results in the two countries during 2000-08. And because league tables were the one major way in which the two systems diverged, differences in the results could be ascribed to this cause.
In other words, if results improved faster in England, league tables must be a good thing. If they improved faster in Wales they must be bad. The researchers found that results in England had improved faster - so league tables must be a good thing.
A press release followed, stating that “naming and shaming schools works”. The academics also argued that the decision to scrap league tables in Wales had “reduced school effectiveness” by two GCSE grades per pupil.
I simplify, for there are some caveats and cross-checks within the paper, as well as detailed statistical analysis. But essentially this is the study’s logic. And it is, of course, despite the technical expertise and experience on display, a simplistic nonsense.
First, the study uses only two indicators of exam results to assess the overall quality - or “effectiveness”, to use the jargon - of the secondary education systems in the two countries. But the larger mistake, it seems to me, is that it is not true to say that league tables are the only aspect of education between the two countries that may have differed over the period.
In reality, league tables have been just one element of a system of high-stakes accountability in England, which also embraces a data-driven inspection system, direct pressure from central and local government on schools to improve their results, and performance pay. All could have driven improvements in results, on both indicators.
Beyond accountability itself, policy change has been hyperactive in England over the period. Trying to isolate one factor as the cause of any results change, however sophisticated the modelling, is perilous.
But my real concern is that the researchers did not do more to investigate the non-statistical qualitative effects, positive or negative, of the focus on results which has come about because of England’s high-stakes accountability system, of which league tables are a part.
The paper does say: “Of course, teachers and schools may have broader aims than GCSE results for their pupils. These are not measured in our data so we can say nothing about the potential impact of (ceasing) publishing performance information on these broader educational outcomes, nor on any potential impact on teacher and headteacher motivation and morale.”
And yet, despite having not carried out this research, the paper’s conclusions support the unequivocal line that league tables raise school performance, implying that schools improve - not improve on certain exam measures, but improve full stop - because of the public scrutiny they bring, and that “naming and shaming” is effective.
This, I would submit - having tried to chart the side-effects of league tables and other aspects of hyper-accountability on pupils’ learning experiences in English schools for years - is just not good enough from academics.
Warwick Mansell is the author of Education by Numbers: The Tyranny of Testing (Politico’s Publishing, 2007). For a longer analysis visit http:bit.lydqWLc0.
