Analyse Pisa and Timss to avoid loss of valuable data

2nd December 2016 at 00:00
Cash for questions: Invest public money in interpreting comparative studies, expert says
Publicly funded research could prevent information being neglected or misinterpreted, academic argues

Experts have called for publicly funded and independent analysis of major international comparative education studies, to ensure that they are not misinterpreted.

Christian Bokhove, from the University of Southampton, is concerned that behind the headline-grabbing international rankings there is a mass of valuable data on schools and teachers that will either go unused or be misused or misunderstood.

He reached this conclusion after researching data on maths education collected by the Trends in International Mathematics and Science Study (Timss) and Programme for International Student Assessment (Pisa). Both are publishing their biggest editions yet – Timss this week and Pisa next Tuesday.

Dr Bokhove acknowledges that the Organisation for Economic Cooperation and Development (OECD), which runs Pisa, and the International Association for the Evaluation of Educational Achievement (IEA), which runs Timss, have their own expertise. “They have really good, large teams of analysts that are the top in the world – they really are the pick of the crop,” the mathematician told TES.

Taking interpretation too far

But, like a growing number of critics, he is concerned that the organisations sometimes over-reach themselves when interpreting and promoting findings from the studies.

He is also concerned that some Timss and Pisa findings are going to waste. “There are variables [on schools] that the OECD and IEA are recording but simply not using because there is too much data,” he said.

Dr Bokhove’s proposed solution is a systematic independent analysis of the findings. “I think it should be funded by research councils or some organisation that might be funded by the government,” he said.

“It would be smart to have a buffer…at least some actions perhaps taken to make sure that there is some kind of waiting room or some double analysis before you actually take any hard policy decisions while waving the Pisa results.”

Dr Bokhove’s proposal is being backed John Bangs, who represents teaching unions for Education International (EI) on the education working group of the OECD’s Trade Union Advisory Committee. He wants there to be a systemic analysis that would provide a “rounded 360-degree” picture by taking in the findings of Timss, Pisa and Pirls (the Progress in International Reading Literacy Study)

“There is an argument for the academic community convening a central consortium to advertise all this incredibly rich data,” Professor Bangs said.

He noted that when EI commissioned an analysis of Talis (the OECD’s Teaching and Learning International Survey) it found that “there was an enormous amount of data that the OECD hadn’t been able to analyse itself”. “There is actually a yawning gap,” he said.

Dr Bokhove is clear that it should not be governments themselves conducting the analysis because they already pay for the studies and have a stake in their findings. “It would be the butcher evaluating his own meat,” he said.

The academic admitted that setting up an alternative evaluation would be “quite a challenge”. But with concerns growing about the power of Pisa and Timss, and whether organisations such as the OECD had their own agenda to promote, he insisted that “some kind of independent intermediate analysis” was necessary to protect what had become a valuable educational resource.


Subscribe to get access to the content on this page.

If you are already a TES/ TESS subscriber please log in with your username or email address to get full access to our back issues, CPD library and membership plus page.

Not a subscriber? Find out more about our subscription offers.
Subscribe now
Existing subscriber?
Enter subscription number

Get Tes online and delivered to your door – for less than the price of a coffee

Save 33% off the cover price with this great subscription offer. Every copy delivered to your door by first-class post, plus full access to Tes online and the Tes app for just £1.90 per week.
Subscribers also enjoy a range of fantastic offers and benefits worth over £270:

  • Discounts off Tes Institute courses
  • Access over 200,000 articles in the Tes online archive
  • Free Tastecard membership worth £79.99
  • Discounts with Zipcar,, Virgin Wines and other partners
Order today