Pisa protestations ring hollow

27th September 2013 at 01:00

Andreas Schleicher defended the Programme for International Student Assessment (Pisa) in TES last month, stating that it was "nonsense" to expect "that there should be no variability in performance on individual questions between students in different countries" ("Attacks on Pisa are entirely unjustified", Comment, 2 August).

Our conclusions are the same: Pisa reading scores are based on a nonsense model and we are happy that Mr Schleicher finally admits this. But his statement is stunning because it goes against everything in the Pisa Technical Reports. These tell us that this variability between questions (differential item functioning) is regarded as a very important issue by Pisa and that it has taken care of it by eliminating questions with this problem.

The only conclusions to be drawn from Mr Schleicher's statement are that he does not know what has been going on or that he disagrees with the way data has been analysed.

Mr Schleicher goes on to claim that "Pisa has convincingly and conclusively shown that the design of the tests and the scaling model used to score them lead to robust measures of country performance that are not affected by the composition of the item pool" and that "the results of these analyses are documented in the Pisa Technical Reports".

These claims are simply wrong. The reports describe in detail how Pisa has tested its scaling model, but there is no documentation of the results and therefore no discussion of differential item functioning or other model errors.

Mr Schleicher is also concerned that we did not use the complete data set for our test of Pisa's model. Since he is worried about sample sizes, it would be extremely interesting to hear his views on the way Pisa estimates and tests its model. Instead of using the complete data set, they do this on a random subset consisting of 15,000 students from Organisation for Economic Cooperation and Development countries, half of whom have not had a chance to respond to any reading items.

Professor Svend Kreiner, University of Copenhagen

See news, page 14.

Subscribe to get access to the content on this page.

If you are already a Tes/ Tes Scotland subscriber please log in with your username or email address to get full access to our back issues, CPD library and membership plus page.

Not a subscriber? Find out more about our subscription offers.
Subscribe now
Existing subscriber?
Enter subscription number


The guide by your side – ensuring you are always up to date with the latest in education.

Get Tes magazine online and delivered to your door. Stay up to date with the latest research, teacher innovation and insight, plus classroom tips and techniques with a Tes magazine subscription.
With a Tes magazine subscription you get exclusive access to our CPD library. Including our New Teachers’ special for NQTS, Ed Tech, How to Get a Job, Trip Planner, Ed Biz Special and all Tes back issues.

Subscribe now