Basic skills teaching has been named as the "biggest challenge" for further education in Ofsted's annual report.
The document, which covers the last 12 months of inspections, said colleges were continuing to improve, but literacy, numeracy and English for speakers of other languages were the weakest areas for all providers in adult education.
Melanie Hunt, Ofsted's director of learning and skills, said: "The initial assessment of learners when they first join a programme at a college or training provider needs to be better at identifying their literacy and numeracy needs. There needs to be a sharper focus, something which the good providers have anyway."
Basic skills also needed to be integrated more closely with a student's main learning so it is seen as relevant and useful, she said.
The challenge mostly involves improving provision, which is "merely" satisfactory. Just one of the 72 basic skills programmes inspected was inadequate, with 34 rated satisfactory.
But the report said improvement was particularly urgent given that the job market in the recession was placing an increasing importance on good literacy and numeracy skills.
A shortage of staff suitably qualified to teach Skills for Life courses was delaying progress, while employers and adults were reluctant to address basic skills issues.
"Adults in employment require particularly high levels of personal support and encouragement to tackle their basic skills needs effectively," the report said.
Ofsted also signalled that light-touch inspections had relied too much on paperwork and promised to focus more on what was happening "on the front line".
A medium-sized college might now expect four subjects to be inspected, even if it has received a top rating, although inspections will be less frequent.
This year's report showed a decline in the proportion of good and excellent colleges, from 71 per cent to 63 per cent, but inspectors said that this may just reflect a different range of institutions inspected.
Improvement over four years was marked, with the numbers earning top grades rising from 48 per cent, while the inadequate provision halved.
"They've been building this ability to be critical of their own performance, to self-assess and to drive their own improvement," Ms Hunt said. "When you look at the ability of providers to improve themselves, colleges are the strongest part of the system."
The Association of Colleges said the report provided clear evidence of the huge efforts made by colleges across very diverse provision, while the University and College Union warned that proposed funding cuts to FE risked undoing the good work.
The prison education system received its best-ever ratings - though it still trailed behind some other parts of FE - and Askham Grange, a women's open prison near York, became the first penal institution to be rated outstanding for education.
Work-based training providers fared worse in the ratings, with just 5 per cent graded outstanding and 37 per cent rated good.
But the Association of Learning Providers said Ofsted had inspected a larger proportion than before of new, unproven organisations offering Train to Gain provision, and called for the programme to be restricted to providers with a track record.