Teaching unions want league tables and individual country rankings to be dropped from the world’s most influential international education study.
Their call comes in response to TES’s report last week on major new academic challenges over the reliability of Pisa (the Programme for International Student Assessment). Professor Svend Kreiner, a statistician from the University of Copenhagen in Denmark, claimed that an inappropriate model is used to calculate Pisa rankings, which means that they are “useless”. Dr Hugh Morrison, from Queen’s University Belfast in Northern Ireland, said that the model itself is “utterly wrong”.
The Organisation for Economic Cooperation and Development (OECD), which runs Pisa, responded by saying that “large variation in single ranking positions is likely” because of “the uncertainty that results from sample data”.
Now John Bangs, chair of the OECD Trade Union Advisory Committee’s working group on education, is calling for the organisation to put countries together in broad blocks or bands, instead of league tables based on individual rankings.
“The problem is the OECD’s insistence on presenting the Pisa results in league table form, not Pisa itself,” Mr Bangs, who works for Education International, a federation of teaching unions in more than 170 countries and territories, told TES.
“We have consistently made the case that the results could be presented in a block form or a number of blocks with the outliers identified. I think the OECD is under pressure from countries to keep the tables, but it seriously lays them open to the kind of criticism that the academics are making.”
The OECD says that it “never talks about ranks” in its reports and always specifies the range of possible variation in rankings for each country. However, numerous references are made to the “rank” of countries in its latest 2009 report.
Although ranges are published, they are not included in the main tables, where the participating countries are set out in rank order according to their individual Pisa scores. The main tables are headed “Where countries rank in...” either reading, mathematics or science.
Many education researchers would welcome a move away from a simplistic emphasis on rankings. Tim Oates, research director at Cambridge Assessment, said: “As researchers advising policymakers, the Pisa league tables are just not that interesting to us, in contrast with the far more fine-grained analytic reports which OECD produces.”
Sue Hackman, who was chief adviser on school standards at England’s Department for Education until Easter, said: “Rank order is the least interesting aspect of Pisa.”
But it is questionable whether dropping the rankings would solve the problems claimed by Professor Kreiner and Dr Morrison. Their concerns relate to a statistical model that is essential for Pisa to make comparisons between countries’ test results.
An OECD spokesperson told TES that putting nations into broader blocks or bands “would set very bad incentives”.
“You would have everybody struggling to get into a band and, as policymakers, you would probably only do something about results if you are close to the upper end of a band, because then you have a chance to make the grade to the next band,” the spokesperson said.
See feature on pages 28-32.
