The academics behind Timss, the longest established global education rankings, have admitted that they are in a “David and Goliath” battle with rival Pisa, but say that they offer a unique take on school performance.
Over the next fortnight, these superpowers in international education comparisons will go head-to-head for first time in more than a decade. Findings from the latest Trends in International Mathematics and Science Study (Timss) and the Programme for International Student Assessment (Pisa) studies will be published on consecutive Tuesdays.
Both will compare the achievements of teenagers from a wide variety of different countries in maths and science.
Ina Mullis and Michael Martin, joint executive directors of Timss, used a TES interview ahead of the publication of their study next week to stress the differences between their approach and Pisa’s.
But the academics, based at Boston College in the US, also acknowledged the competition with their high-profile rival.
“It is a David and Goliath situation,” Dr Martin said. “The OECD [Organisation for Economic Cooperation and Development, which runs Pisa] is an enormous government organisation and Pisa is just a small piece of what they do. They have got a huge operation and they churn out stuff - all highly professional.
“The IEA [International Association for the Evaluation of Educational Achievement, which runs Timss] is a much smaller, essentially non-government network of participants. So it is really hard to match that kind of effort. But we do.”
The notion of competition between Timss and Pisa dismays independent experts, who say the studies work best as complementary research.
Key differences
There is plenty that is unique to each programme. Timss also tests primary pupils and Pisa looks at collaborative problem-solving. Even where they overlap on secondary maths and science, there are crucial differences.
It is not just the slightly different age groups - Pisa tests 15-year-olds in Year 11, whereas Timss examines 8th grade or Year 9 pupils. There are also important distinctions in what they test.
“[Timss] is designed to assess the material that students are learning in their school curricula in maths and science,” Dr Mullis explained about the rankings.
“Pisa is more about the skills that they think students should have to enter the workforce.
“I know it is confusing for teachers because the age groups seem close enough and the subjects seem close enough, but the kind of material that is included is a little different.”
Dr Martin said the differences could help to explain why countries could be ranked differently in the two studies.
‘Pisa and Timss are complementary and it is strange when governments seem not to view them as such’
“It is true, for example, that Finland doesn’t do quite as well on our assessments as it does on Pisa, but that may be to do with the literacy emphasis in Pisa,” he said.
“There is a lot of reading in Pisa, not just in the reading assessment but in the science assessment and the mathematics assessment. So a country that does well in reading tends to have a Pisa advantage.”
Those differences will be crucial when it comes to analysing Timss and Pisa results over the next fortnight after the biggest release of comparative education data in history.
Governments are likely to cherrypick the results that show them in the best light. But experts like Christian Bokhove argue that it is essential to understand that any differences between the studies’ headline results are not a case of one being right and the other wrong.
“They are complementary and it is strange when governments seem not to view them as such,” the University of Southampton maths educationalist, who has studied Pisa and Timss, told TES.
“I hope that with the IEA and OECD releasing these results so close together, we don’t get a culture of them being against each other and who has the most powerful announcements, because the data is not as simplistic as that. I think that is a danger.”
‘Always in competition’
Dr Martin makes no bones about the rivalry, but insists it has benefits. “We have always been in competition at the 8th grade,” he told TES. “We were there before there ever was a Pisa. We helped develop and design it way back in the beginning, so we know a bit about it.
“But, sure, the OECD likes to be in charge of things and is certainly competing heavily at the 8th grade…But it is a healthy thing, I think. Countries appreciate having two views of what goes on in schools.”
Dr Bokhove’s fear is that the rivalry will lead to homogenisation, with the two organisations competing on the same ground.
“They do seem to want to go in similar directions,” he said. “To me it feels a little bit like Apple versus Google or Apple versus Samsung,” Dr Bokhove said. “But there is a market for both - they complement each other.
“They do sometimes touch each other’s target audiences a bit and it would be a shame if that took over everything.”
But Dr Martin denied any attempt to imitate Pisa, saying: “We are doing what we always did.”
And he insisted that a crucial distinction remained: “The OECD is a big economic government-based policy organisation and it likes to tell people what they think they should do.
“We are different. We are a country-based, grassroots-based organisation and we work together to help people decide what they want to do, based on what other people are doing. We don’t issue policy directives.”
