Ofsted inspectors said Scope, which supports people with cerebral palsy and other disabilities, had been too slow to make Beaumont college properly accessible to its 78 disabled students.
The residential college in Lancaster for 16 to 25-year-olds, which aims to help disabled people become independent, does not meet the national care standards or Special Educational Needs Disability Act legal requirements.
Social care inspectors have criticised Beaumont's facilities for years, but improvements have been delayed due to lack of funds. There are too few lifts, inaccessible sinks and kitchenettes, and doors too narrow for wheelchairs, the Commission for Social Care Inspection has reported.
Ofsted said: "Improvements to the environment have accelerated recently but accommodation continues to be inadequate."
Scope spokesman Barry Hugill denied the charity had forfeited the moral authority to criticise others for failing disabled people.
He said: "It's a standard Seventies building with all the problems you get with that. It's not just us. Half the schools and colleges in the country are like this. You don't get changes overnight."
Leadership of the college was rated inadequate, but Mr Hugill said a new principal would turn it around. "There will not be another report like this," he said. However, education at the college was judged satisfactory by Ofsted and student achievement good.
Colleges have come under increased scrutiny over access since the introduction of the Disability Discrimination Act in September. Sasha Callaghan, equality officer for lecturers' union Natfhe, said 40 colleges were breaking the law.
But Ofsted's criticism will be particularly embarrassing to Scope in the year that it launched a campaign against disability discrimination, called Time to Get Equal. It has criticised the three-quarters of businesses that fail to provide proper access for disabled people.
And, on access to education, the charity says: "Inclusive education is... a matter of human rights."