Pisa protestations ring hollow

27th September 2013 at 01:00

Andreas Schleicher defended the Programme for International Student Assessment (Pisa) in TES last month, stating that it was "nonsense" to expect "that there should be no variability in performance on individual questions between students in different countries" ("Attacks on Pisa are entirely unjustified", Comment, 2 August).

Our conclusions are the same: Pisa reading scores are based on a nonsense model and we are happy that Mr Schleicher finally admits this. But his statement is stunning because it goes against everything in the Pisa Technical Reports. These tell us that this variability between questions (differential item functioning) is regarded as a very important issue by Pisa and that it has taken care of it by eliminating questions with this problem.

The only conclusions to be drawn from Mr Schleicher's statement are that he does not know what has been going on or that he disagrees with the way data has been analysed.

Mr Schleicher goes on to claim that "Pisa has convincingly and conclusively shown that the design of the tests and the scaling model used to score them lead to robust measures of country performance that are not affected by the composition of the item pool" and that "the results of these analyses are documented in the Pisa Technical Reports".

These claims are simply wrong. The reports describe in detail how Pisa has tested its scaling model, but there is no documentation of the results and therefore no discussion of differential item functioning or other model errors.

Mr Schleicher is also concerned that we did not use the complete data set for our test of Pisa's model. Since he is worried about sample sizes, it would be extremely interesting to hear his views on the way Pisa estimates and tests its model. Instead of using the complete data set, they do this on a random subset consisting of 15,000 students from Organisation for Economic Cooperation and Development countries, half of whom have not had a chance to respond to any reading items.

Professor Svend Kreiner, University of Copenhagen

See news, page 14.

Log-in as an existing print or digital subscriber

Forgotten your subscriber ID?


To access this content and the full TES archive, subscribe now.

View subscriber offers


Get TES online and delivered to your door – for less than the price of a coffee

Save 33% off the cover price with this great subscription offer. Every copy delivered to your door by first-class post, plus full access to TES online and the TES app for just £1.90 per week.
Subscribers also enjoy a range of fantastic offers and benefits worth over £270:

  • Discounts off TES Institute courses
  • Access over 200,000 articles in the TES online archive
  • Free Tastecard membership worth £79.99
  • Discounts with Zipcar, Buyagift.com, Virgin Wines and other partners
Order your low-cost subscription today