Much work-related provision is also being offered on a stereotyped gender basis, with boys being steered into engineering and motor maintenance, and girls into hairdressing.
John Brown, qualifications and curriculum adviser for inclusion, said the findings, from an unpublished QCA report, suggested that vocational courses remained a "second-best option" in many schools.
The report has major implications for the Government, which is seeking to make a new vocational diploma qualification a first choice for pupils of all abilities within three years.
Mr Brown told The TES about the findings after highlighting the issue at a seminar at the Education Show in Birmingham.
He said: "Some schools are only offering vocational provision at key stage 4 to pupils with learning difficulties. What sort of messages does that send out about vocational education, and about learning difficulties?"
The QCA visited 100 secondaries in 2004-05 as part of a monitoring report on provision for special needs pupils, covering General National Vocational Qualifications and other work-related courses.
Mr Brown said more than 20 had offered this only to pupils with special needs. He said: "Vocational provision needs to be planned into a school's whole curriculum, and not just bolted on for some pupils, because you do not know what else to do with them."
He said some secondaries took a very different approach. For example, some girls' schools were offering engineering courses for all. The findings come three months after Ofsted warned - following a KS4 survey in 130 secondaries -that some schools did not see the relevance of work-related learning for high-achievers.
Some 100,000 14 to 16-year-olds are offered vocational courses in colleges under the Government's increased flexibility programme. This has been praised by inspectors, although reports have said that some schools are not taking it seriously.
Alison Wolf, professor of public sector management at King's college, London, said that current vocational courses were rightly not prioritised by many schools because they were not valued by employers.
New vocational or "specialised" diplomas - to be available from 2008 - will offer pupils the chance to combine GCSEs and A-levels with work-related learning in areas such as construction, health and social care and media.
Ruth Kelly, the Education Secretary, said last year that she expected more than half of teenagers eventually to take the diplomas. The QCA findings suggest this may require a culture change in schools.
David Blunkett, the former education secretary who achieved academic success despite his blindness, said: "What we should avoid at all costs is the presumption that craft skills are for the less able".
He said this was both "deeply patronising" and untrue.
Instead, he said, he favoured a broad, "Leonardo da Vinci" approach to learning, where every pupil would be encouraged to study science, languages, technology and arts.
Mr Blunkett said that it had been assumed in his early life that he would go into a job such as piano tuning or clerical work, and he had therefore been encouraged to learn typing. This, though, had helped him in going on to get a degree.