Courses in English for speakers of other languages (Esol) have improved steadily since 2005, but there was still too little provision rated "good" or "outstanding", an Ofsted report said.
Colleges fared better, with half of their Esol teaching earning one of the top two ratings. But among adult and community learning providers, the proportion rated good or outstanding dropped to just one-fifth.
While the number of top-rated courses had not risen, providers had become better at eliminating poor classes, with only one college course inspected being rated less than satisfactory since 2005.
Christine Gilbert, chief inspector of schools, said: "While it's encouraging to see a general picture of improvement, it is time that standards were raised so that the overwhelming majority of Esol provision is good or better.
"We must equip learners with the very best English skills to help them have the confidence to make a positive contribution to the community."
The report, which looked at 114 colleges and 30 other providers, also found that institutions were now mostly offering externally accredited Skills for Life Esol courses instead of internal programmes.
But introducing fees for all migrants who are not receiving tax credits or benefits had caused a drop in the numbers enrolling, according to the report.
Initial data from the Learning and Skills Council (LSC) showed there had been a fall of 14 per cent from autumn 2006 to autumn 2007, suggesting that about 70,000 fewer people were taking classes.
Paula Whittle, principal of Ealing, Hammersmith and West London College, the country's biggest Esol provider, said: "Due to the effects of the new LSC Esol funding structure, we are finding it increasingly difficult to meet the needs of our learners.
"As a result, not only is the quality of life of these people affected, but the London economy and the UK as a whole are missing out on what is essentially a vital skills resource."
The report recommended that the Government should monitor the effect of fees and whether they were causing some students to stay away from college courses.
It also said colleges and other providers needed to ensure more Esol teachers had specialist training, since a quarter had no English language teaching qualification.