Baroness Blackstone, the minister for further and higher education added that the situation had called for new powers of intervention.
Speaking in defence of the Learning and Skills Bill, which is going through its committee stage in the upper house, she described much publicised cases of mismanagement and failure such as Halton and Gwent Tertiary College as "the tip of the iceberg".
She said: "At far too many of our colleges we have found low standards and the improper use of public funds. In dealing with the backlog of failure we have been hampered by the present statutory provisions."
The Government's groundbreaking plans for post-16 education and training emerged unscathed from their first examination by the House of Lords.
But, in an eight-hour debate, Conservative and Liberal Democrat members challenged clauses giving the proposed Learning and Skills Council powers to intervene in colleges' affairs.
The Bill allows te LSC to appoint up to two new governors to the board of any further education institution, powers described by Liberal Democrat Baroness Sharp of Guildford as "draconian" and an "unreasonable interference with the autonomy of colleges".
But Baroness Blackstone said the powers would not be used frequently or lightly and only as a last resort. She said: "We need to applaud excellence, but we also have to deal swiftly with failure. No one should doubt our determination to do so."
Opposition peers also criticised the Bill's distinction between pre and post 19 education. It states that facilities up to the age of 19 are "proper" if they are of "a quantity sufficient to meet the reasonable needs of individuals and of a quality adequate to meet those needs". But after the age of 19 facilities need only be "reasonable".
Lib-Dem Lord Tope described the phrasing of the clauses as "weasel words". He suggested the Government should aim to implement the Kennedy report's recommendations that adults should be entitled to an education up to level three, the equivalent of A-levels.