The UK’s Pisa data reporting was not “sufficiently transparent” about sources of bias and the limitations of the data, the Office for Statistics Regulation said today.
The watchdog was responding to a complaint from UCL professor John Jerrim calling for an independent review into the data used in the Programme for International Student Assessment in the UK.
Professor Jerrim wrote to the regulator last month alleging the data had ”serious flaws” because his analysis showed that low-achieving students were underrepresented in the England and Wales samples, while he found a number of anomalies, including a high number of ineligible students, in the Scottish data.
Background: ‘Investigate serious flaws in England’s Pisa data’
Related: Pisa: Low-achieving students ‘systematically excluded’
Explainer: What is the Pisa test and what does it measure?
He also highlighted that England and Northern Ireland had conducted a non-response bias analysis - but was not reported publicly.
In the letter published today, Ed Humpherson, director general for regulation of the Office for Statistics Regulations, said: “The technical standards set out by the OECD ensure a high level of quality in the results.
“The national reports make it clear whether differences are statistically significant or not. However, we agree with you that the reports were not sufficiently transparent about the limitations of the data and the potential sources of bias.”
He also added: “Where a non-response bias study is required, the results of it should be published in full.
“In addition, any changes to the PISA data collection, including to the timing, should be clearly explained along with any potential impacts on the results.”
Mr Humpherson said in the letter that his team had held “constructive conversations” with the relevant statisticians in the four nations of the UK.
He added: “Each of the producer teams recognises that reporting information about the quality of the statistics is important for users to understand the impact of limitations and sources of bias on their use of these data, and each has committed to publish more information about quality for future rounds of Pisa.”
In the letter to the Pisa report producers in England, Wales and Northern Ireland, the watchdog encouraged each producer to publish information, such as any non-response bias analysis, in full.
And in its communication with the Scottish Government, which compiles the Pisa Scotland report, one of the recommendations was to explain better why pupils’ withdrawal rates had increased and give more transparency around ineligibility of students.
In both letters, the regulator also added a number of good practice points that the review found in the work that went into producing the Pisa reports.
Professor Jerrim said that on the whole he was satisfied with the review.
“They have made some reasonable points: there are some elements of good practice,[…] but there are some things that clearly need to be done better,” he told Tes.
But he added: “There is one point I disagree with…about how the OECD’s criteria make sure that the data is high quality. Actually […] for Wales there is a lot of bias even if you do meet the criteria. Outside of that I am generally quite happy.”
A spokesperson for the Department for Education said: “We welcome that the OSR found a range of areas of good practice in our reporting, and we will consider and take on board their recommendations.”
A spokesperson for National Foundation for Educational Research said: “At NFER, we are extremely supportive of transparency in research and look forward to reading the OSR’s response”.