Sir Andrew Foster, former head of the Audit Commission, whose review on the role of FE was published last week, told Ruth Kelly, the Education Secretary, that private-sector training organisations should take over the worst colleges.
"Time should be called on those institutions that have relentlessly failed their communities," Sir Andrew said.
He recommended that management be given a year to turn the college round.
"If they fail re-inspection, control should go to someone else - another college, voluntary organisation or private provider," he said.
But Mr Haysom said such a handover was most unlikely. "In truth," he said, "I see that as a bit of a red herring. To my knowledge, there has only been one college that has failed re-inspection - and re-inspection comes within a year."
He said the bigger issue was not the 4 per cent of colleges judged inadequate overall but patches of poor provision across the sector. "Even in very successful colleges, there can be provision that is not good enough. I want to work with principals to drive up quality across the board."
Inspections by the Office for Standards in Education and the Adult Learning Inspectorate last year showed that half of all colleges had some inadequate courses. Even here, however, college leaders argue that persistently poor provision has already been removed and that the Foster review adds nothing to this debate.
Julian Gravatt, director of finance at the Association of Colleges, said:
"We are talking in most cases about very small problems such as an underperforming construction department. When this occurs, these departments are often closed or contracted out - usually at the request of the college managements themselves."
Mr Haysom agreed that colleges had made improvements over the past two years, with success rates rising to 71 per cent, an increase of 10 percentage points.