More than half of specialist schools are failing to meet GCSE targets in their chosen subjects, David Bell, the chief inspector, said this week.
Improvement has slowed in some specialist subjects during the past three years and has started to decline in others, he said.
Mr Bell said that being a specialist school does make a difference.
Compared with other schools, specialists do well against a range of indicators. He said that "targeted use of funding and being part of an optimistic network of like-minded schools contribute to an impetus and climate for improvement".
But the latest Ofsted evaluation of the specialist school programme warned of significant variations in performance and singled out arts and language colleges for particular criticism.
Ofsted figures showed the number of arts college pupils who gained good GCSE grades in drama and music is lower than the national average. The report is based on visits to 52 schools and a survey of 521 schools which have had specialist status since at least September 2000. There are more than 2,000 specialist schools and ministers hope all secondaries will become specialist by 2008.
Overall, specialist schools have improved 60 per cent faster than other secondaries since 1998 and the quality of teaching is higher in specialist schools than their non-specialist neighbours. Language colleges have been the slowest to improve and technology colleges the fastest.
Inspectors found a "major turn-around" in specialist schools' work with the wider community, especially primary schools, since Ofsted's last evaluation of the programme in 2001.
But the report said that not enough has been done to raise achievement in specialist subjects despite weaknesses being identified four years ago.
The latest report found that in 2003, a smaller proportion language college pupils gained A*-C grades in French, German and Spanish than did in 2001.
The number of arts college pupils gaining good grades in drama fell from 70 per cent in 2001 to 63 per cent in 2003. During the same period the national average dropped from 68 to 65 per cent.
Overall, fewer than half of the schools hit their targets for the number of pupils gaining GCSE grade C or better in each specialism.
Ofsted said few schools could adequately explain their failure.
Mr Bell said: "Specialist schools must ensure that the drive for improvement is maintained. The variations in performance between specialist schools must be addressed to ensure that all types of specialist schools are consistently of the same high standard in all areas of teaching and learning."
Sir Cyril Taylor, Specialist Schools Trust chairman, said Ofsted's findings were based on a small number of schools with only 100 language and 50 arts colleges included in the survey.
He said: "Ofsted is not comparing like with like. All language college students are expected to study a language, but only about a third of pupils in other schools do so."
Other weaknesses identified by the report included:
* schools need to do more to create links with other secondaries and business;
* not enough support for students at risk of disaffection;
* uneven provision for gifted and talented pupils with one in five language, technology and arts colleges unsatisfactory;
* arts and sports colleges need to improve support for special-needs pupils.