Pupils with the most complex needs make good progress in all types of schools where there are high-quality, experienced teachers and heads who are committed to their success, he said.
Ofsted criticised mainstream schools which rely too heavily on teaching assistants to look after pupils with special needs. Statements of need are not working, he said. Pupils with behavioural, emotional and social difficulties are most likely to miss out on the support they need.
The report, Inclusion: Does it matter where pupils are taught?, found little difference between the standard of education offered in mainstream and special schools, although mainstream schools with specialist provision were most likely to achieve the best outcomes for pupils. Pupil-referral units were the least successful.
Mr Smith said: "Pupils with even the most severe and complex needs can make outstanding progress in all types of settings. The inclusion debate has for too long focused on whether children with learning difficulties should be educated in special schools or mainstream schools rather than the quality of the education and support they receive."
Ofsted's report is the third in two months to criticise support for pupils with SEN.
A report by the Commons education select committee, published last week, said the statementing system was "not fit for purpose".
A study carried out by Cambridge university for the National Union of Teachers, published in May, said that teachers in mainstream schools were regularly doing dangerous and unpleasant tasks such as cleaning tracheotomy tubes and changing nappies.
Ofsted's report said initial training in SEN should be improved for all teachers. It also called on the Government to focus more on improving the results of the bottom quarter of pupils, and set out what progress it expects schools to achieve for special needs pupils.
Andrew Adonis, schools minister, said: "We are pleased to see that a key finding of this report is that children with SEN are able to make outstanding progress in all types of settings. We have always been clear that inclusion is about the quality of children's education ... whether that is a mainstream or a special school."