Ruth Kelly this week put the blame on colleges for their axing of an estimated 200,000 adult places.
The Education Secretary said further education had been given pound;1 billion of extra cash in recent years and individual colleges took responsibility for how it is spent.
She said it was "absolutely right" for the Government to set out its priorities, but colleges were free to make their own decisions on which courses to provide.
Speaking at the launch of Adult Learners' Week in London, she said colleges were "not obliged" to follow the priorities set by the Learning and Skills Council.
"Individual colleges will always make decisions about which courses they want to fund, what courses they feel they don't need to subsidise quite as much, and how they meet the learning priorities.
"Just because the Learning and Skills Council may decide that it's going to fund different priorities doesn't mean that a college necessarily stops funding a particular course."
FE Focus revealed last month that a crisis looms in adult learning because of lower than expected funding. And in an FE Focus survey last week, 30 per cent of colleges said they were getting less cash next year while 77 per cent were getting less than expected.
Ms Kelly rejected claims that colleges faced a massive funding shortfall.
She also suggested they could charge higher course fees if they needed more money.
"About a hundred million pounds of fees are currently waived by the FE sector. They might think very carefully about whether they want to do that in future," she said.
But Dr John Brennan, AoC chief executive, said imposing or increasing fees was to "ignore the reality" facing adult learners.
"Many of the individuals affected by the cuts are low-earners, which is why colleges so often waive their fees. For ministers to say that colleges must ask them to pay the full costs is to ignore the reality of many people's lives.
"For example, most of our students are women, many on low wages and with childcare costs."
He accepted that the Government had increased funding since 2001, but added: "At the same time it has made even more demands, with a whole new set of targets for young people and adults.
"The courses colleges are cutting are not only leisure provision, but include those which thousands of individuals use to improve their job prospects."
Ms Kelly attended the launch after digesting a 30-page briefing note prepared by her advisers about cuts in courses.
A source said that the note had originally been 18 pages long, but was almost doubled in size after FE Focus revealed the depth of the funding crisis last week.
Rob Wye, LSC Director of Strategy and Communications said: "Colleges are feeling the funding pressure in part because of their success in driving up rates of participation, retention and achievement among students.
"We now have the highest ever participation of 16 to 18-year-olds in education and training, and funding these rising numbers is putting a big strain on the FE budget, leaving less for other areas.
"Funding is also being squeezed because we are asking colleges to concentrate core funding on four key priority areas that we have agreed with the Government to deliver the Skills Strategy."" David Sherlock, head of the Adult Learning Inspectorate, urged the Government to ensure adults don't lose out to younger learners.
He said: "Adult learning, in all its guises, plays a vital role not only in improving skills and employability, but also in meeting the government's agenda for social inclusion, community cohesion and citizenship.
"There are some tough choices to be made, but we can't afford for adults to get the thin end of the wedge."
The four priorities are education and training for the 16-18 age group, young people taking apprenticeships, people who need to improve their basic skills of literacy and numeracy, and people undertaking their first Level 2 qualification.