Not everyone leans the same way over Pisa

2nd August 2013 at 01:00

Ironically, your feature "Is Pisa fundamentally flawed?" (26 July) was itself so fundamentally flawed that it simultaneously misrepresented expert critics and misled non-specialist readers.

First, the Rasch model does not require questions to "be equally difficult in all participating countries". If it did, it wouldn't be a model and all countries would get the same score. Differential item functioning compares the difficulty of a question across groups only after they have been matched for ability. This is a problem not only for the Programme for International Student Assessment (Pisa) but for all comparative studies.

Second, the Rasch model describes a probabilistic relationship so it does not imply that students of the same ability will "all give a correct answer or would all give an incorrect one" to the same question. As Harvey Goldstein says, there is no conceptual error here - even if there are serious technical shortcomings.

Third, Pisa's plausible values are dismissed as if they were randomly plucked from thin air, even though the Organisation for Economic Cooperation and Development (OECD) makes it clear that they are numbers randomly drawn from a known probability distribution. This approach is entirely rational - even if in this case its validity is questionable.

I am one of Pisa's many critics but this article did little to advance our case or clarify our concerns.

George Bethell, Director, Anglia Assessment, Stowmarket, Suffolk.

Log-in as an existing print or digital subscriber

Forgotten your subscriber ID?


To access this content and the full TES archive, subscribe now.

View subscriber offers


Get TES online and delivered to your door – for less than the price of a coffee

Save 33% off the cover price with this great subscription offer. Every copy delivered to your door by first-class post, plus full access to TES online and the TES app for just £1.90 per week.
Subscribers also enjoy a range of fantastic offers and benefits worth over £270:

  • Discounts off TES Institute courses
  • Access over 200,000 articles in the TES online archive
  • Free Tastecard membership worth £79.99
  • Discounts with Zipcar,, Virgin Wines and other partners
Order your low-cost subscription today