The organisation behind the world’s most influential international education league tables has highlighted serious shortcomings in its methods for comparing the performance of low- and middle-income countries with that of richer nations.
The Programme for International Student Assessment (Pisa) portrayal of schooling in parts of the developing world is “problematic” and unrepresentative, according to a paper from the Organisation for Economic Cooperation and Development (OECD).
The paper warns that education in poorer countries is often hindered by language problems and lower school participation, issues which Pisa does not take into account. The paper also says that the tests - used to compile rankings and other reports - have a “lower” degree of reliability when taken in such countries. This can influence the quality of the results, it says.
John Bangs, chair of the OECD Trade Union Advisory Committee’s working group on education, said the paper highlighted an important issue. “You can’t apply the same rules to developing countries as you can do to developed countries when you are evaluating what goes on in the system,” he said.
The news came as a row over the statistical reliability of all Pisa rankings continued. As reported in TES, Professor Svend Kreiner from the University of Copenhagen claimed that the rankings were “meaningless” because they used a flawed statistical model - a criticism dismissed by Andreas Schleicher, the OECD official in charge of Pisa, as “nonsense”.
Professor Kreiner has now hit back in a letter to TES (see page 7), suggesting that Mr Schleicher’s defence does not tally with the way the survey is run.
This month’s OECD paper on Pisa in poorer nations describes the “non- representative sample of 15-year-olds” in some countries as one of Pisa’s “main challenges”.
Written by an OECD staff member, it looks at how appropriate the Pisa tests are for assessing the 40 or so low- or middle-income economies among the 82 jurisdictions that take part.
The paper notes that some of these countries have “very high” proportions of students who do not go to school. In 13 jurisdictions the rate is more than 10 per cent, rising to 22.5 per cent in Uruguay. “For Pisa, a high share of children no longer in school at the age of 15 is problematic, as the test will not capture a representative sample of 15-year-olds,” the paper says.
It warns that as Pisa only tests in schools, this problem renders the indicators it uses to measure “equity” in education “less meaningful”. The most disadvantaged students in those countries are missed by Pisa, which is instead confined to “an already relatively privileged student population”, it says.
The paper also notes that the 15-year-olds tested by Pisa will have been in school for different lengths of time in different countries because the students start school at varying ages, some repeat years and some have their schooling interrupted by “socio-economic circumstances or crisis situations”.
Problems around language are also highlighted. Pisa recommends testing students in the language they are taught in, but the paper states that “in many low-income countries, the students’ mother tongue is not the language of instruction for large minorities and sometimes even the majority of students”.
Finally, it says, in many low- and middle-income countries the majority of students are “clustered” at the “lower end of the Pisa proficiency scale”.
The paper notes that Pisa tests are geared to 15-year-olds in richer OECD nations. With fewer questions targeted at students with lower capabilities, the paper says that “reliability of measurement is, by definition, lower at the bottom”.
“This can influence the quality of results for low-performing low- and middle-income countries that have a large percentage of students performing at these levels,” it adds.
Pisa tried to address this last problem by introducing optional test booklets with easier questions in 2009, the paper says, but most countries have chosen not to use them because of a “perceived risk of stigma”.
Mr Schleicher said: “The OECD is launching a new initiative called `Pisa for Development’ that will develop and pilot adapted survey instruments and methods that better suit the contexts of developing countries, while still producing results that can be compared across all countries in Pisa.”
PISA in Low and Middle Income Countries by Simone Bloem is at bit.lyPisaLowMiddle
A fair comparison?
Some problems raised by the OECD paper could be alleviated if Pisa attempted to rank the level of education in a country, rather than the effectiveness of its schools.
But the paper states that Pisa was “constructed to evaluate the performance of school systems”.
Mr Schleicher backed this up, writing that the “goal” of Pisa was to “provide robust comparisons of the performance of education systems”.
