Baroness Helena Kennedy QC - made a working peer in the post-election honours list - makes the claim in a new report aimed at improving ways of bringing new students into colleges.
But she said there was widespread evidence of "wonderful work" in FE colleges and beyond, and called on principals to emulate the best.
The 55-page guide has been produced in the wake of Ms Kennedy's report on widening participation in further education, published in July.
She said: "The fact is that good practice in widening participation is patchy. Strategies . . . are unevenly developed nationally, regionally and within individual colleges and other provision.
"The aim of this guide is to provide a resource for those responsible for widening participation in colleges and other organisations."
The report, sent to colleges this week, argues that local partnerships are vital to reach groups such as the unemployed or ex-offenders, and emphasises the use of a college's own information about students to help reach under-represented groups. And it gives advice on setting targets for reaching groups who are poorly served.
The main part of the report is taken up with 36 case studies of colleges which have been successful at attracting more people.
The Further Education Funding Council plans a series of pilot schemes to test the use of local partnerships, as well as developing a new "pathway" to help ease adults into colleges.