Pisa still has questions to answer

16th August 2013 at 01:00

In TES on 2 August, Andreas Schleicher suggests that I may be unfamiliar with advances in item response theory in the past 50 years ("Attacks on Pisa are entirely unjustified", Comment). I can assure him that this is not the case, but what concerns me is that he must know that nothing in the literature of the past 50 years addresses the central point made in my paper.

My case against the Programme for International Student Assessment (Pisa) is that all these advances simply produce more completely "static" models of the wrong type, while the measurement model I suggest is "dynamic" through and through. This confusion is another by-product of the profound conceptual error at the heart of Pisa; Pisa misconstrues the nature of ability itself.

All measurement is made against a "background" that gives meaning to the properties being measured. For example, physics textbooks often portray Einstein's general relativity as a re-construal of the background that Newton (pictured, right) took for granted (for example, he assumed absolute simultaneity, Euclidean geometry and so on).

Suppose a Pisa item were to require the examinee to find the square of 11. There is scant, if any, evidence in the literature that when one pronounces 121 the correct answer and all other answers incorrect, one is drawing on a background of mathematical practice. If one reads the psychological and psychometric literatures one has the clear impression that the question in itself is the measuring instrument. This is incorrect: without the evolving mathematical practice within which this question is embedded, the question has an infinity of answers. Without the background mathematical practice, notions like incorrect and correct lose their meaning.

This leads to a dynamic picture of psychological and educational measurement completely beyond the capabilities of Pisa's software, no matter how advanced. The examinee's response of 121 represents a contribution to the mathematical consensus that underpins the measuring instrument. An incorrect answer, for example, weakens that consensus. (Needless to say, I am not suggesting that the correct answer to any given problem is established by taking a vote among mathematicians.)

Once again, we arrive at the conclusion that no one at Pisa seems ready to face: there is no clear dividing line between the examinee being measured and the instrument carrying out the measurement. Ability is a joint property of the person and the practice of mathematics, and not simply an intrinsic property of the person. Rather, a deflationary account of ability would be to consider it to be a property of the interaction between examinee and the practice. Ability isn't something the examinee has; it's something he or she has for us.

Dr Hugh Morrison, Queen's University Belfast.

Log-in as an existing print or digital subscriber

Forgotten your subscriber ID?


To access this content and the full TES archive, subscribe now.

View subscriber offers


Get TES online and delivered to your door – for less than the price of a coffee

Save 33% off the cover price with this great subscription offer. Every copy delivered to your door by first-class post, plus full access to TES online and the TES app for just £1.90 per week.
Subscribers also enjoy a range of fantastic offers and benefits worth over £270:

  • Discounts off TES Institute courses
  • Access over 200,000 articles in the TES online archive
  • Free Tastecard membership worth £79.99
  • Discounts with Zipcar, Buyagift.com, Virgin Wines and other partners
Order your low-cost subscription today