The Further Education Funding Council's quality assessment committee says there is a strong case for greater external intervention and special support for under-achieving colleges.
A report from the committee includes an analysis of FEFC inspections of 134 colleges and observations of more than 9,000 lessons. Its recommendations come in the wake of highlighted failures - Bilston, Halton, Matthew Boulton and the Wirral - and the naming and shaming of another four last week - Ealing, West Cumbria, Workington and the Isle of Wight.
The report says: "There is irrefutable evidence that a few colleges consistently fail to provide an adequate quality of education for their students. In such cases, there is a poor record of student retention and achievement and little evidence of focused action to improve matters."
In addition to its call for more intervention and support, the committee urges far greater collaboration with other partners in the post-16 sectors to share the burden.
The committee's call was echoed by Jim Donaldson, FEFC chief inspector, at a national conference in London, Raising Standards in Further Education.
More local learning partnerships with councils, the voluntary sector and employers were needed if the country was to hit its training and education targets by 2002.
Despite considerable gains in further education - 65 per cent of lessons observed by his teams were awarded a grade 1 or 2 last year - there was still some way to go. "There is still significant progress to be made if further education is to achieve the challenging learning targets that have been set.
The targets' aim is a rise from 74 per cent to 85 per cent of 19-year-olds with GCSE-equivalent qualifications within four years.
The hoped-for rise in the number of 21-year-olds with A-level equivalent over the same period is from 52 to 60 per cent.
Both Mr Donaldson in his speech and the FEFC committee in its report pointed to deep-seated problems at management level.
The committee calls for better training for managers and for all college governors. Teaching staff also need more training and support, the committee says.
Mr Donaldson warned that the wider problems of increasing participation in education and training by excluded groups such as ethnic minorities and the unemployed could not succeed without the colleges.
"If we don't embrace inclusive learning, who will?"