Andreas Schleicher defended the Programme for International Student Assessment (Pisa) in TES last month, stating that it was “nonsense” to expect “that there should be no variability in performance on individual questions between students in different countries” (“Attacks on Pisa are entirely unjustified”, Comment, 2 August).
Our conclusions are the same: Pisa reading scores are based on a nonsense model and we are happy that Mr Schleicher finally admits this. But his statement is stunning because it goes against everything in the Pisa Technical Reports. These tell us that this variability between questions (differential item functioning) is regarded as a very important issue by Pisa and that it has taken care of it by eliminating questions with this problem.
The only conclusions to be drawn from Mr Schleicher’s statement are that he does not know what has been going on or that he disagrees with the way data has been analysed.
Mr Schleicher goes on to claim that “Pisa has convincingly and conclusively shown that the design of the tests and the scaling model used to score them lead to robust measures of country performance that are not affected by the composition of the item pool” and that “the results of these analyses are documented in the Pisa Technical Reports”.
These claims are simply wrong. The reports describe in detail how Pisa has tested its scaling model, but there is no documentation of the results and therefore no discussion of differential item functioning or other model errors.
Mr Schleicher is also concerned that we did not use the complete data set for our test of Pisa’s model. Since he is worried about sample sizes, it would be extremely interesting to hear his views on the way Pisa estimates and tests its model. Instead of using the complete data set, they do this on a random subset consisting of 15,000 students from Organisation for Economic Cooperation and Development countries, half of whom have not had a chance to respond to any reading items.
Professor Svend Kreiner, University of Copenhagen
See news, page 14.
