Gove accused of building on shaky Pisa foundations
Michael Gove has made England's apparent plunge down the international education league tables the central justification for his sweeping school reforms. But now the government's own statistics watchdog has called into question the figures at the heart of the education secretary's argument.
The UK Statistics Authority has censured the Department for Education and Sir Michael Wilshaw - appointed by Mr Gove as Ofsted chief inspector - for using uncertain, weak and "problematic" statistics to claim that England's schools have tumbled down the global rankings.
In a television interview in September, Sir Michael asserted that "our standards are falling in relation to other countries in the rest of the world", citing a significant drop in England's Programme for International Student Assessment (Pisa) ranking between 2000 and 2009.
The DfE used the same figures in a 2010 press release, claiming: "England has continued to fall in the Pisa rankings, meaning that in just nine years we have dropped from 7th to 25th in reading, 8th to 27th in mathematics and 4th to 16th in science."
Now Andrew Dilnot, UK Statistics Authority chair, has written an open letter criticising Sir Michael and the department, noting that the Organisation for Economic Co-operation and Development (OECD), which compiles Pisa, no longer views the UK results from 2000 as statistically valid because not enough schools took part.
Mr Dilnot also cites a review by the University of London's Institute of Education, which highlighted the statistical problems and the contradiction of Pisa findings by the rival Trends in International Mathematics and Science Study.
The institute's report concluded that "it may be difficult to treat an apparent decline in secondary school pupils' performance as 'a statistically robust result'", according to Mr Dilnot.
"Given these various published caveats and advice, my own view is that those commenting on data in this area should take particular care to avoid making comparisons that could be interpreted as statistically problematic, or otherwise might be misunderstood by the audience," he writes.
His verdict will be seen by critics as a blow to Mr Gove's claim that England has "plummeted in the world rankings" given that the education secretary has been so unequivocal about the figures, arguing that "these are facts from which we cannot hide".
Andreas Schleicher, the OECD official who runs Pisa, told TES that the data used by Mr Gove was "a little bit dodgy". But he added: "The more important lesson for me is that the UK has not improved in a way that we have seen other systems improving."
Mr Dilnot's letter is addressed to Labour MP David Miliband - schools minister between 2002 and 2004 - in response to a complaint by the MP about Sir Michael's use of the Pisa figures.
The statistics watchdog chair also goes on to criticise the DfE press release, writing: "Headline results for England from the Pisa study, alongside relative international rankings, were not accompanied by detailed advice or caveats to help the reader in making comparisons over time, nor were the statistical implications of an increase in the number of reporting countries in later Pisa studies noted.
"The weaknesses relating to the response-rate standard in earlier studies should not be ignored."
Mr Dilnot's letter adds: "These uncertainties and weaknesses are not just a technical footnote; they are themselves an important part of the evidence, and affect interpretation and meaning.
"League tables and the presentation of international rankings can be statistically problematic, and require clear and careful commentary alongside them. I will find a suitable opportunity to explore these matters further with the department."
Labour shadow education secretary Stephen Twigg said: "Michael Gove's department needs to be very careful in its use of official statistics, especially at a time when he has undermined the teaching profession. Running down our teachers and schools is not the best way to raise standards."
An Ofsted spokeswoman said: "The interpretation of international comparisons is complex, but the overall message from Pisa is clear. England's mean scores fell, particularly between 2000 and 2006, while other countries in the survey improved. There are also concerns in light of the latest 2009 results, where many other countries are noticeably outperforming England."
The DfE had not provided a comment at the time of going to press.