Inexperienced swimming teachers have been excluding some children with behavioural problems on the grounds of health and safety, a study by school inspectors found.
In other schools, the needs of black ethnic minority children were not taken into account. This group has less experience of swimming and makes the least progress, Ofsted said.
The best schools made more effort to include all pupils, for example by making special provisions for Muslim children. One local authority installed "modesty panels" in changing rooms and ran single-sex swimming lessons for Muslim girls.
According to government targets, all pupils are supposed to be able to swim 25 metres by the time they leave primary school.
Ofsted inspected lessons in 30 schools that had received funding to improve their links with swimming clubs. They found that around three-quarters of pupils were able to attain the 25 metre target.
One of the biggest problems in these schools was time pressure, with almost a third not allowing enough time for the activity, Ofsted said.
In some cases this problem was exacerbated by travel time. Some pupils were travelling for 70 minutes to swim for 30.
Other schools felt they could not offer more time because it would cut into literacy and numeracy time, Ofsted found.
Schools were encouraged to liaise with mosques to take account of different cultural concerns about the sport.
Family swimming sessions were also found to be effective in breaking down cultural barriers.
Miriam Rosen, Ofsted's director of education, said pupils were overwhelmingly positive about the swimming programmes they were offered.
But she said: "More needs to be done to address the barriers that prevent all pupils participating in swimming activities in order to build up self-esteem, keep fit and healthy, and to meet swimming standards."
Inspectors recommended that schools should monitor children's progress more closely and provide more opportunities for talented swimmers.