What is Pisa and why do we take part?

10th January 2016 at 09:26
Pisa
What is the Programme for International Student Assessment (Pisa) and why do we participate?

The Programme for International Student Assessment (Pisa) is a triennial international survey evaluating education systems worldwide by testing the skills and knowledge of 15-year-old students.

Since 2000, students from randomly selected schools representing more than 80 economies have participated in the assessment testing the key subjects: maths, reading and science.

In 2012, around 510,000 students in 65 economies took part in the Pisa 2012 assessment representing approximately 28 million 15-year-olds globally.

You can view the Pisa 2012 results here and see the Pisa results for all subjects for all years, here.

Seventy-two economies took part in the most recent assessment in 2015, of which science was the main focus.

The first set of results from the 2015 round will be published on 6 December 2016 at 11:00 (Central European Time).

The Pisa tests are not directly linked to the school curriculum. Their aim is to assess to what extent students at the end of compulsory education, can apply their knowledge to real-life situations and be equipped for full participation in society. The information collected through background questionnaires are also completed and the information collected from these is intended to provide context to aid analysts in interpreting the results.

As Pisa is an ongoing survey, those countries participating in successive surveys can compare their students' performance over time and draw on the results to assess the impact of education policy decisions.

Pisa is run by the Organisation for Economic Cooperation and Development (OECD), an intergovernmental economic organisation with 35 member countries, founded in 1961, with a headquarters in Paris.

View the results from Pisa's 2015 round of testing here on 6 December 2016, and read our expert analysis of the results. 

Key findings from 2012

Maths

 

All students, mean score

Ranking

UK

494

26/65

England

496

 

Northern Ireland

487

 

Wales

469

 

Scotland

 

 

 

Reading

 

All students, mean score

Ranking

UK

499

23/65

England

500

 

Northern Ireland

498

 

Wales

480

 

Scotland

 

 

 

Science

 

All students, mean score

Ranking

UK

514

21/65

England

516

 

Northern Ireland

507

 

Wales

491

 

Scotland

 

 

'Meaningless' comparisons between countries

One of the surprising facts about the Programme for International Student Assessment (Pisa) is that not all participating 15-year-olds answer the same questions.

In Pisa 2006, for example, about half the participating pupils were not asked any questions on reading, and only one in ten were tested on all 28 of them. Similarly half were not tested at all on maths, even though full rankings were produced for both subjects. Science, the main focus of Pisa that year, was the only subject that all participating students were tested on.

The basic reason for this is straightforward – if all pupils took all the questions the tests would be too long. However Pisa still assigned reading scores to 15-year-olds who did not answer any reading questions, with the same true for maths, so that there was a full set of data to calculate country scores and rankings.

The OECD says Pisa is a “system-level assessment” rather than a measurement of individual achievement. So it says there is nothing wrong with calculating “plausible valuables” for pupils that were never asked particular questions.    

To work out what these values should be, scores from pupils who did answer the questions are fed into a statistical “scaling model”. Up to and including Pisa 2012 the scaling model it chose was the “Rasch model” – a choice that turned out to be the subject of huge academic controversy.

As TES revealed in 2013 Professor Svend Kreiner, a Danish expert in Rasch argued that it was a completely unsuitable model for Pisa and would not work, because the questions used were of different levels of difficulty in different countries. As a result, he said, Pisa’s comparisons between countries were "meaningless" and “useless”. 

Dr Hugh Morrison, a mathematician that at Queen's University Belfast in Northern Ireland, went further  and argued that the Rasch itself was "utterly wrong" and rendered Pisa rankings “valueless".

At the time the OECD stuck to its guns and robustly rejected the criticism. 

But in a TES interview last month Andreas Schleicher, OECD education director, took a very different stance when asked about academic criticisms of Pisa.

“The ones that are constant are ones that have helped shape PISA, a lot, and our thinking as well,” he said. TES then brought up the criticisms of Rasch and asked whether the model was still being used, Schleicher responded: “No. we have now changed to a…That’s a good example where actually over last few cycles, we started in 2009 to modify the Rasch model and then in 2012 we used a 2-parameter variant and now in 2015 we have used the full 3-parameter model that those [critics] were recommending.”

According a subsequent interview with Pisa lead analyst Miyako Ikeda, Pisa has not actually abandoned Rasch altogether. But it has moved away from it – a change she says was brought in from 2015 rather than 2009 or 2012. 

Pisa now uses a mix of Rasch and the two parameter model, depending on the question in what Ikeda describes as a “hybrid” solution. But she says there are no plans to introduce a three parameter model – which would also take into account how guessable questions are, as well as their level of difficulty and discrimination – because that would make the test too long.

The differences between Ikeda’s and Schleicher’s accounts are worth highlighting, not to be facetious but because they illustrate how difficult it can be to fully understand the highly technical, but crucial, details even for those at the heart of Pisa. 

Want to keep up with the latest education news and opinion? Follow TES on Twitter and like TES on Facebook

Comments

Get TES online and delivered to your door – for less than the price of a coffee

Save 33% off the cover price with this great subscription offer. Every copy delivered to your door by first-class post, plus full access to TES online and the TES app for just £1.90 per week.
 
Subscribers also enjoy a range of fantastic offers and benefits worth over £270:

  • Discounts off TES Institute courses
  • Access over 200,000 articles in the TES online archive
  • Free Tastecard membership worth £79.99
  • Discounts with Zipcar, Buyagift.com, Virgin Wines and other partners
Order your low-cost subscription today