Are the Prime Minister's warm words for specialist schools really justified? Fran Abrams reports on the use - and abuse - of educational statistics.
Congratulating the stalwarts of the Technology Colleges Trust at their 10th-anniversary celebration in Birmingham, the Prime Minister was understandably anxious to highlight the remarkable successes of specialist schools.
"So many of you have brilliant stories to tell," he enthused. "I've seen it in my own Sedgefield constituency, at the Sedgefield specialist sports school. It is transformed in terms of ethos, atmosphere, facilities, and achievement. And it is teaching exactly the same intake as before. The difference is they're getting better results."
This was remarkable indeed - especially as the speech was made two years ago, just a few months after the school had been awarded its specialist status. And even more so as the one set of GCSE results it had received since its elevation had actually been slightly worse than the previous year's.
To be fair, results at Mr Blair's local sports college have now started to improve. But as the Statistics Commission - an independent body set up by the Government to ensure the figures it releases are fair - begins a review of the way educational performance is measured and presented, it might consider making an assessment of Tony Blair's ability to interpret exam results.
The technology colleges speech was not an isolated case. Last September, at the opening of the Bexley business academy, Mr Blair had another opportunity to praise specialist schools. This time he produced a list, citing huge rises in standards at a number of institutions.
The Sir John Cass language college, he said, had seen a huge rise in the number of pupils gaining five or more A*-C grades at GCSE. Between 1995 and 2003 the number had risen from 8 per cent to 80 per cent.
Again, impressive figures, and an achievement for which the London school certainly deserves recognition. But it only received its specialist status in 1999 - by which time it had already achieved some impressive rises in exam scores.
So its specialist status was certainly not the only reason for its success.
The St Marylebone school of performing arts London was also singled out for praise. The school's GCSE score had risen from 33 per cent in 1994 to 93 per cent in 2003, Mr Blair pointed out. Yet the London school only became a performing arts college in 1998, by which time its exam score had already risen to 74 per cent.
The revelation that a politician has used statistics in a less than scrupulous manner can hardly come as a shock, of course. But Martin Rogers of the Education Network, observing these two speeches, confesses his reaction was more one of amusement than anger.
"I don't have great expectations of politicians' speeches," he says. "But I think when they get to the point of actually saying things which aren't true, as Mr Blair did in Birmingham, they should come round and collect themselves."
The Statistics Commission may not bother itself with Mr Blair's speeches as it embarks on its review. It has plenty of more weighty matter to discuss.
In general, it will look at whether there is useful or interesting information held by schools which is not available to the public, and at whether the information which is available is presented in a fair and comprehensible manner. In particular, it will look at some major areas of concern, two of which have been raised by academics.
The first of these is the issue of how standards are maintained over time.
Professor Peter Tymms, of Durham university, has submitted a paper to the commission which suggests the Government's claims of continued success for its National Literacy Strategy may be due to a lack of adequate, long-term monitoring.
The TES reported last December that a three-year study commissioned by the Qualifications and Curriculum Authority had found that rises in 11-year-olds' reading test scores were due not to rising standards but to the fact that the pass mark had been set too low.
The Statistics commission is considering an analysis by Professor Tymms, published by the British Educational Research Journal this autumn, which provides further evidence that this may be the case.
In August the schools standards minister, David Miliband, praised primary schools for record results in English at key stage 2, in which 77 per cent of pupils reached level 4.
"Pupils and parents can be confident that their primary schools are still getting better and better," he said.
But Professor Tymms's paper disputes this. He has analysed a range of long-term studies, all of which seem to show reading scores have not improved in the way the Government claims. At the root of the issue, he says, is the absence of any body which can ensure standards are maintained over time. The job used to be done by the assessment of performance unit until it was abolished by the last government in 1990.
The problem arises not so much through direct political interference as through a more subtle pressure on all the agencies involved, he believes.
"The Government sets so much store by rising standards that if it was in any way to admit these huge improvements in primary schools were not well founded it would be a major political problem," he says.
"The fundamental problem is that we are using statutory tests to monitor standards over time. There should be a different system, such as the one Scotland has in its assessment of achievement programme, which uses a small sample and administers the same tests year in and year out."
Another area of concern which will be covered by the Statistics commission's review will be the Government's method of measuring the value-added performance of schools. This is done simply by comparing pupils' results with their previous attainment when they entered their schools. But many observers feel this does not give a true picture.
Professor Harvey Goldstein, chair of the Royal Statistical Society's education study group, has already raised the issue with the commission.
Such a basic measure fails to take into account important factors such as social deprivation, he argues. And there is evidence that such factors can severely inhibit schools' abilities to make progress with their pupils.
It also fails to take into account "statistical uncertainty" - the fact that a small school's results will be far more likely to fluctuate than those of a larger one.
This issue has been raised repeatedly with the Department for Education and Skills, not just by Professor Goldstein but also by the Secondary Heads Association. Its spokesman on value-added league tables, Tony Neal of the De Aston school in Lincolnshire, says the result is that parents are not seeing the true picture.
"The tables are intended to provide parents with information about the effectiveness of schools," he says. "At present they are encouraging parents to draw conclusions which are simply not valid. So, in effect, they are providing disinformation."
A DfES spokeswoman said the department welcomed the commission's review but defended its record on the use of statistics. Performance over time was monitored effectively by the Qualifications and Curriculum Authority, she said. On the subject of value added, it was developing a more sophisticated measure of exam results.
She said: "We announced our intention to develop a more sophisticated, value-added methodology which takes account of factors beyond previous attainment. Professor Goldstein was part of the group that helped us develop our prototype model, which we will be trialling with some 400 schools this autumn."
The DfES argues that its value-added measure is not intended to be a fully comprehensive means of measuring school performance, but simply one narrow indicator to be taken alongside others. And on that point the Commission agrees with it.
But it has issued a paper saying that it believes the figures should be presented differently. And while the Commission deliberates about how the Government presents the results of its education policies, it seems likely the department will continue to face hard questions.
The things they say...
"Sir John Cass language college in Tower Hamlets, east London, has transformed its results from just 8 per cent in 1995 to 80 per cent this year. It was the most improved school in the country last year. It is one of the few secondary schools in Europe that teaches Mandarin Chinese.
Again, its success is due to the outstanding leadership of its headteacher, Haydon Evans, and the marvellous support of its sponsors HSBC and the Sir John Cass Foundation."
Tony Blair praises the excellent results of a school which achieved specialist status in 1999; September 2003.
"Specialist schools are also playing a vital part. You enhance diversity and choice, building centres of excellence school by school in return for additional investment. You are at the forefront of school improvement, for only schools with good leadership, and a credible strategy for raising standards, can gain and maintain specialist status."
Tony Blair praises the specialist schools for leading the way in raising standards, before naming a school in his own constituency where exam results had just fallen. Technology Colleges Trust 10th anniversary conference, Birmingham, November 2002.
"At 11 a record 77 per cent of pupils achieved the expected level in English. Last year I described the primary results as a platform rather than a plateau of achievement. This year primary schools have built on the platform and achieved the best-ever results in English and maths.
"Pupils and parents can be confident that their primary schools are getting better and better."
The schools standards minister, David Miliband, praises schools for record results at key stage 2, August 2004. Several studies have now revealed the rise in reading standards could be much less than ministers claim.