Policy - Does autonomous always mean better? Not necessarily..

7th February 2014 at 00:00
'More detailed' analysis of Pisa data shows no significant effect

An influential finding from the world's best-known education study showing a link between greater school freedom and better results is wrong, according to new research.

The Programme for International Student Assessment (Pisa) study has concluded that when principals are given more independence over spending, their schools achieve better academic results, as long as those results are published. "Autonomy matters when combined with accountability," the study first advised in 2010. The recommendation has influenced education ministers and was repeated in the latest Pisa report in December.

But an academic from an international exam board has carried out a more sophisticated analysis, concluding that Pisa got it wrong. Tom Benton, principal research officer for Cambridge Assessment, found no statistically significant association between the amount of freedom given to schools over how to spend their budgets and their academic results. "With schools in developed nations there doesn't seem to be a relationship," he told TES. "I find the (Pisa) evidence quite unconvincing."

The news will be a blow to politicians including Michael Gove, England's education secretary, who has used the finding to help justify his controversial academies expansion policy and his free schools programme, both of which are designed to give school leaders more freedom.

The finding follows significant challenges last year to the credibility of Pisa, which is helping to shape education policy in a growing number of countries. Academics claimed that the country rankings produced by Pisa were "useless" and "meaningless" because the statistical model used to calculate them was either unsuitable or "utterly wrong". A further study cast doubt on the quality of Pisa data collected through school questionnaires.

The importance of freedom for schools has been a particularly influential Pisa finding. In January 2012, Mr Gove said Pisa had "clearly" shown that "the greater the amount of autonomy at school level, with headteachers and principals free to determine how pupils are taught and how budgets are spent, the greater the potential there has been for all-round improvement".

However, Dr Benton notes in his report that when Pisa 2009 (published in 2010) looked at individual school autonomy, it did not find any significant link between curriculum freedom and students' results.

Pisa did find that "within countries where schools are held to account for their results through posting achievement data publicly, schools that enjoy greater autonomy in resource allocation tend to do better than those with less autonomy".

But Dr Benton looked at the same data that Pisa based this claim on and found no such link. He examined the figures for state and independent schools separately and concluded that the relationship did not exist in either sector.

The researcher believes he has reached a different conclusion because his analysis is "more fine-grained". When considering other factors that might explain individual school performance, Dr Benton took into account the fact that the difference between achievement in private and state schools varied "significantly" between countries - a fact that the Pisa calculation glossed over.

He also included the significant variation of other background factors, such as gender and whether the school in question was in the state or private sector - another point Pisa had not accounted for. Each of those factors alone would account for his differing conclusion, Dr Benton told TES.

Tim Oates, group director of assessment research and development at Cambridge Assessment, who advised Mr Gove on the curriculum, said that governments should follow the example of Germany, which conducted detailed analysis into Pisa findings. "There is not much point in people bashing (each other) over the head over the topline Pisa findings... it doesn't really work," he said.

Andreas Schleicher, deputy education director of the Organisation for Economic Cooperation and Development, which runs Pisa, stood by its analysis. "It makes no sense to separate out public and private schools from an analysis of the impact of autonomy and accountability on pupil performance... since that is one of the main expressions of school autonomy in many countries," he said. "Even then the nature of public and private schools varies considerably from one country to another, and retaining that diversity is important in analysis such as this."

A spokesperson for England's Department for Education did not comment on the new research, but restated the department's belief in the importance of school autonomy.

Log-in as an existing print or digital subscriber

Forgotten your subscriber ID?


To access this content and the full TES archive, subscribe now.

View subscriber offers


Get TES online and delivered to your door – for less than the price of a coffee

Save 33% off the cover price with this great subscription offer. Every copy delivered to your door by first-class post, plus full access to TES online and the TES app for just £1.90 per week.
Subscribers also enjoy a range of fantastic offers and benefits worth over £270:

  • Discounts off TES Institute courses
  • Access over 200,000 articles in the TES online archive
  • Free Tastecard membership worth £79.99
  • Discounts with Zipcar, Buyagift.com, Virgin Wines and other partners
Order your low-cost subscription today