An inquiry by the Government's Social Exclusion Unit has found that half of pupils in some rural schools miss out on after-school sport and other activities because of inadequate transport.
The Central Council for Physical Recreation - an umbrella group which includes sports governing bodies and teacher unions among its members - said that poorer pupils were more likely to be excluded from activities.
Preliminary research by the council shows that, in some areas, nine out of 10 pupils rely on the school bus service to get home.
A council spokesperson said: "The school bus is often the only way home for those most at risk of exclusion. As such there is a real danger that the Lottery investment of pound;750 million in school sports facilities will only benefit those children with access to private cars.
"We have record levels of obesity among young people and that will only continue unless something is done to tackle this problem."
The Government has promised all pupils a right to two hours of PE per week, but this is made up of timetabled and after-school sport.
A Department for Education and Skills spokesman admitted that disadvantaged pupils were more likely to miss after-school activities in areas where transport is poor. He said the department was bidding for funds from the Treasury in this summer's spending round to tackle the problem.