Alan Tuckett, director of NIACE, the adult education watchdog, said adult education was been squeezed both by local authority cuts and by the freeze on cash for further education expansion.
He said half of Britain's local authorities had cut adult education in the past five years, and many were axing their support altogether.
This week alone he has protested at Newcastle's decision to cut adult education, and further proposed cuts in Bury and Kent.
And he warned that adult education would be the first to suffer as a result of the loss of expansion funding from the Further Education Funding Council.
He said: "If you get cuts in both areas, you get erosion of two areas of the curriculum.
"If you take Pounds 100 million out of the system, the reality is you lose the adults the FEFC widening participation committee is trying to reach.
"The fastest growth has been in part-time adult study. If the cut in demand-led funding hits, the effect on local availability will be catastrophic.
"People who do not live near a college want help with literacy. Their prospects are horrendous."
He warned local authorities that it was a statutory requirement to provide adequate non-vocational adult education.
And he called for better definitions of council's obligations to stop the sector from becoming a target for cuts.
Senior figures in NIACE have long been campaigning for a clarification of adequate adult education. They point to wide variations between areas with little or no local authority funded courses, and areas of full provision.
They are concerned that adult education is too easy to cut when councils are faced with stark choices to defend schools.
Mr Tuckett said: "People are caught in a vice. If you have to spend about 85 per cent of your budget on schools, and you do not want to cut them, you have few choices."