Skip to main content

Not everyone leans the same way over Pisa

Ironically, your feature "Is Pisa fundamentally flawed?" (26 July) was itself so fundamentally flawed that it simultaneously misrepresented expert critics and misled non-specialist readers.

First, the Rasch model does not require questions to "be equally difficult in all participating countries". If it did, it wouldn't be a model and all countries would get the same score. Differential item functioning compares the difficulty of a question across groups only after they have been matched for ability. This is a problem not only for the Programme for International Student Assessment (Pisa) but for all comparative studies.

Second, the Rasch model describes a probabilistic relationship so it does not imply that students of the same ability will "all give a correct answer or would all give an incorrect one" to the same question. As Harvey Goldstein says, there is no conceptual error here - even if there are serious technical shortcomings.

Third, Pisa's plausible values are dismissed as if they were randomly plucked from thin air, even though the Organisation for Economic Cooperation and Development (OECD) makes it clear that they are numbers randomly drawn from a known probability distribution. This approach is entirely rational - even if in this case its validity is questionable.

I am one of Pisa's many critics but this article did little to advance our case or clarify our concerns.

George Bethell, Director, Anglia Assessment, Stowmarket, Suffolk.

Log in or register for FREE to continue reading.

It only takes a moment and you'll get access to more news, plus courses, jobs and teaching resources tailored to you