Serious concerns have been raised in a report, commissioned by the Learning and Skills Council, calling for finance to be better-directed towards such students.
The LSC and associated organisations need a national strategy to provide a service which is "high quality, learner-centred and cost-effective," it says.
The LSC should also aim for "a common funding approach across the whole of the post-16 learning and skills sector".
The report says: "There is tension between target-led provision and meeting the needs of learners with learning difficulties andor disabilities.
"In particular, there should be a designated individual at a senior level whose role it is to provide the necessary operational oversight.
"The fact that the LSC distributes money through five different funding avenues - school sixth forms, work-based learning, adult education, specialist colleges and further education - is not conducive to helping achieving quality nationally and consistently," it says.
The report reveals that 579,000 of the seven million students funded by the LSC say they have learning difficulties or disabilities.
Between 2003 and 2004, the LSC's bill for the 579,000 students came to Pounds 1.3 billion and the largest group of 382,000 was in FE.
But the report says too much money is misdirected, with funds being spent on non-educational activity. This is believed to include cash to pay for accommodation for some disabled students because there is no suitable college near their house.
There is a need for "systemic changes across the whole sector, enabling more flexible, equitable and cost-effective targeting of funds".
The report was compiled by a steering group, chaired by Peter Little, a trustee of Skill, the National Bureau for Students with Disabilities. He said: "Disabled people rightly have enhanced expectations in post-compulsory education and training, and we hope this review can point the way to meeting these within available resources."