Pisa still has questions to answer

16th August 2013 at 01:00

In TES on 2 August, Andreas Schleicher suggests that I may be unfamiliar with advances in item response theory in the past 50 years ("Attacks on Pisa are entirely unjustified", Comment). I can assure him that this is not the case, but what concerns me is that he must know that nothing in the literature of the past 50 years addresses the central point made in my paper.

My case against the Programme for International Student Assessment (Pisa) is that all these advances simply produce more completely "static" models of the wrong type, while the measurement model I suggest is "dynamic" through and through. This confusion is another by-product of the profound conceptual error at the heart of Pisa; Pisa misconstrues the nature of ability itself.

All measurement is made against a "background" that gives meaning to the properties being measured. For example, physics textbooks often portray Einstein's general relativity as a re-construal of the background that Newton (pictured, right) took for granted (for example, he assumed absolute simultaneity, Euclidean geometry and so on).

Suppose a Pisa item were to require the examinee to find the square of 11. There is scant, if any, evidence in the literature that when one pronounces 121 the correct answer and all other answers incorrect, one is drawing on a background of mathematical practice. If one reads the psychological and psychometric literatures one has the clear impression that the question in itself is the measuring instrument. This is incorrect: without the evolving mathematical practice within which this question is embedded, the question has an infinity of answers. Without the background mathematical practice, notions like incorrect and correct lose their meaning.

This leads to a dynamic picture of psychological and educational measurement completely beyond the capabilities of Pisa's software, no matter how advanced. The examinee's response of 121 represents a contribution to the mathematical consensus that underpins the measuring instrument. An incorrect answer, for example, weakens that consensus. (Needless to say, I am not suggesting that the correct answer to any given problem is established by taking a vote among mathematicians.)

Once again, we arrive at the conclusion that no one at Pisa seems ready to face: there is no clear dividing line between the examinee being measured and the instrument carrying out the measurement. Ability is a joint property of the person and the practice of mathematics, and not simply an intrinsic property of the person. Rather, a deflationary account of ability would be to consider it to be a property of the interaction between examinee and the practice. Ability isn't something the examinee has; it's something he or she has for us.

Dr Hugh Morrison, Queen's University Belfast.

Subscribe to get access to the content on this page.

If you are already a Tes/ Tes Scotland subscriber please log in with your username or email address to get full access to our back issues, CPD library and membership plus page.

Not a subscriber? Find out more about our subscription offers.
Subscribe now
Existing subscriber?
Enter subscription number

Comments

The guide by your side – ensuring you are always up to date with the latest in education.

Get Tes magazine online and delivered to your door. Stay up to date with the latest research, teacher innovation and insight, plus classroom tips and techniques with a Tes magazine subscription.
With a Tes magazine subscription you get exclusive access to our CPD library. Including our New Teachers’ special for NQTS, Ed Tech, How to Get a Job, Trip Planner, Ed Biz Special and all Tes back issues.

Subscribe now