Experts have called for publicly funded and independent analysis of major international comparative education studies, to ensure that they are not misinterpreted.
Christian Bokhove, from the University of Southampton, is concerned that behind the headline-grabbing international rankings there is a mass of valuable data on schools and teachers that will either go unused or be misused or misunderstood.
He reached this conclusion after researching data on maths education collected by the Trends in International Mathematics and Science Study (Timss) and Programme for International Student Assessment (Pisa). Both are publishing their biggest editions yet - Timss this week and Pisa next Tuesday.
Dr Bokhove acknowledges that the Organisation for Economic Cooperation and Development (OECD), which runs Pisa, and the International Association for the Evaluation of Educational Achievement (IEA), which runs Timss, have their own expertise. “They have really good, large teams of analysts that are the top in the world - they really are the pick of the crop,” the mathematician told TES.
Taking interpretation too far
But, like a growing number of critics, he is concerned that the organisations sometimes over-reach themselves when interpreting and promoting findings from the studies.
He is also concerned that some Timss and Pisa findings are going to waste. “There are variables [on schools] that the OECD and IEA are recording but simply not using because there is too much data,” he said.
Dr Bokhove’s proposed solution is a systematic independent analysis of the findings. “I think it should be funded by research councils or some organisation that might be funded by the government,” he said.
“It would be smart to have a buffer…at least some actions perhaps taken to make sure that there is some kind of waiting room or some double analysis before you actually take any hard policy decisions while waving the Pisa results.”
Dr Bokhove’s proposal is being backed John Bangs, who represents teaching unions for Education International (EI) on the education working group of the OECD’s Trade Union Advisory Committee. He wants there to be a systemic analysis that would provide a “rounded 360-degree” picture by taking in the findings of Timss, Pisa and Pirls (the Progress in International Reading Literacy Study)
“There is an argument for the academic community convening a central consortium to advertise all this incredibly rich data,” Professor Bangs said.
He noted that when EI commissioned an analysis of Talis (the OECD’s Teaching and Learning International Survey) it found that “there was an enormous amount of data that the OECD hadn’t been able to analyse itself”. “There is actually a yawning gap,” he said.
Dr Bokhove is clear that it should not be governments themselves conducting the analysis because they already pay for the studies and have a stake in their findings. “It would be the butcher evaluating his own meat,” he said.
The academic admitted that setting up an alternative evaluation would be “quite a challenge”. But with concerns growing about the power of Pisa and Timss, and whether organisations such as the OECD had their own agenda to promote, he insisted that “some kind of independent intermediate analysis” was necessary to protect what had become a valuable educational resource.
@wstewarttes
