The heads of both education inspection services, David Bell and David Sherlock, raised concerns that too many specialist colleges were failing.
Mr Bell, Ofsted's chief inspector, said: "We continue to be concerned about specialist colleges, where the inadequacy rate is above 20 per cent.
Colleges are doing a caring job, but not focusing as much as they can on the education they should be offering to students with disabilities and learning difficulties."
The Adult Learning Inspectorate, meanwhile, judged eight out of 18 - 44 per cent - of the specialist colleges it inspected to have inadequate leadership and management.
Mr Sherlock, its chief inspector, said in his annual report: "Staff are too often warm, caring and deeply committed to the interests of their learners.
"These very qualities often hinder them effectively supporting progress to open employment. They are too easily satisfied with little progress, rather than encouraging more."