Pisa comparisons ‘not useful’, globally respected education academic says

Dylan Wiliam says that international Pisa rankings are bedevilled by missing data and distorted by private tuition
26th April 2018, 11:11am

Share

Pisa comparisons ‘not useful’, globally respected education academic says

https://www.tes.com/magazine/archive/pisa-comparisons-not-useful-globally-respected-education-academic-says
Thumbnail

International comparisons of education systems are not useful because they are plagued by missing data and the private tuition of students, and because policies cannot be simply transplanted to other countries, a world-respected education academic has said.

Speaking last week at the Best in Class summit organised by the Sutton Trust in New York, Dylan Wiliam, emeritus professor of the UCL Institute of Education, poured criticism on international league tables like the Organisation for Economic Co-operation and Development’s Programme for International Student Assessment (Pisa).

Appearing on a panel that included the boss of Pisa, Andreas Schleicher, Professor Wiliam said: “Like everybody else in this room, I am really interested in how we raise levels of achievement in general and to close the achievement gap.

“Probably unlike most people in this room, I am not particularly convinced about the usefulness of international comparisons for directing this effort in a particular country for a variety of reasons.”

He then went on to list what he considered to be a number of major flaws with international comparisons, including:

  • Missing data - “We know that a number of students were not sampled in Shanghai for example, so we can’t be sure that those are representative of all the students living in Shanghai,” he said.
  • Private tuition giving a misleading impression - “Singapore is clearly very successful,” Professor Wiliam said. But he added: “I happen to think it’s mostly private tuition. We know that in [South] Korea for example if you take out the effects of tuition, the performance of kids in Korea could well be below that of the United States.”

Professor Wiliam also pointed out that some systems celebrated for their high performance actually appear to be stagnating. “Ontario’s results have basically been flat for 20 years,” he said. “The reason that’s quite important is because if what they were doing in Ontario was good educationally, then as children have been exposed to one, two, three, four, five years of this system, the results should be going up, and they’re not. And we see results in Finland actually going down.”

He said that, even if problems with Pisa could be overcome, he was still “sceptical” about the value of international comparisons because education policies cannot easily be transplanted to other countries.

“Even if we know why those countries are high performing, could we implement that in the UK or England? The answer is probably not.

“I’m very sceptical about even if we can figure out good research from this, whether these would provide us with solutions.”

After the summit, Professor Wiliam told Tes: “The problem is that people claim to know why particular countries get particular results, and there are far too many variables, and far too few data points, for this to be possible.”

You’ve reached your limit of free articles this month

Register for free to read more

You can read two more articles on Tes for free this month if you register using the button below.

Alternatively, you can subscribe for just £1 per month for the next three months and get:

  • Unlimited access to all Tes magazine content
  • Exclusive subscriber-only articles 
  • Email newsletters

Already registered? Log in

You’ve reached your limit of free articles this month

Subscribe to read more

You can subscribe for just £1 per month for the next three months and get:

  • Unlimited access to all Tes magazine content
  • Exclusive subscriber-only articles 
  • Email newsletters

This is 0 of 1

Now only £1 a month for 3 months

Subscribe for just £1 per month for the next 3 months to get unlimited access to all Tes magazine content. Or register to get 2 articles free per month.

Already registered? Log in

This is 0 of 1

Now only £1 a month for 3 months

Subscribe for just £1 per month for the next 3 months to get unlimited access to all Tes magazine content.