The older people are, the less likely they are to sign up for a Government-recognised course, says the National Institute of Adult Continuing Education.
The institute, which advises ministers on post-19 education, has been calling on the Department for Education and Skills to protect courses which fall outside the Government's targets for basic skills and vocational training, fearing they will be squeezed out under current funding arrangements.
The survey will add weight to the argument that adults, especially those above retirement age, are finding little to suit their needs in the current climate.
The research shows an overall decline in the number of students over the age of 19, and the drop is particularly dramatic among the over-65s.
More than half of people over 65 say they have not participated in education since leaving school.
Only 17 per cent of people aged 65-74 have any recent experience as "learners", says NIACE, compared with 20 per cent last year. The figure is 8 per cent for people over 75, compared with 10 per cent last year.
In contrast, 80 per cent of 17 to 19-year-olds and 61 per cent of 20 to 24-year-olds are current or recent learners.
The survey also found that people's intentions towards learning also differ dramatically with age.
The report says: "Seventy-three per cent of 17 to 19-year-olds say that they are likely to take up learning in the next three years, compared with only 41 per cent of those aged 45 to 54. For adults aged 55 and over, the drop in future intentions to learn is even more severe."
NIACE has met Ivan Lewis, the adult skills minister, to point out its concern over courses which fall outside Government targets.
Alan Tuckett, director of NIACE, says he believes Mr Lewis is committed to learning "for its own sake".
He said: "I think he is very much committed, but what we want is to make sure this area of learning, which is not about the economy but is nevertheless important, has some space to exist.
"Otherwise there is a danger that you diminish the quality of our civilisation in the pursuit of rather narrow goals which, while important, are only part of the wider picture."
NIACE wants the DfES to protect learning which falls outside government targets by earmarking 3 per cent of colleges' budgets for other types of adult courses, including those aimed at leisure activities such as art, which are popular with older students.
But he says the Government could also be "missing a trick" by not widening participation among the over-55s in vocational courses. The possible impending pensions crisis could lead to many people continuing to work after retirement age, requiring re-skilling.
Employers have already been complaining that older apprentices are needed in industry to fill existing skills gaps.
The DfES has made clear in its plans for the skills strategy that fees for some non-targeted programmes could be increased from the present level of 25 per cent of course costs.
The strategy is expected to focus on adults without level 2 qualifications (equivalent to GCSE grade C or above), people under 30 taking level 3 (A-level equivalent) and adults taking level 3 qualifications in skills shortage areas.