Primary schools could be overlooking the benefits of PE in helping children's intellectual development in their drive to push up test results, according to the Youth Sports Trust.
It is campaigning against the Government's plans to allow primary schools to slim down the national curriculum to concentrate on the 3Rs.
Schools will have the freedom to relax non-core subjects and the trust says many are already using it as an excuse to cut back on PE, art and music.
Trust chief executive Sue Campbell said: "Physical activity is a great vehicle for developing literacy and numeracy in early years. But if you almost remove PE from the curriculum you're in danger of removing one of the most powerful influences on children's lives.".
A review compiled for the trust of research into links between sport and early-years learning suggests a consensus that physical activity boosts young children's cognitive development and academic achievement.
The review, by Ben Tan of Loughborough University's department of PE and sports science, concludes PE helps children develop language and motor skills and understand concepts such as space and time.
The trust says it is seeing evidence that some schools are cutting PE - particularly those in inner-city areas which have tough literacy and numeracy targets to hit.
In Tower Hamlets, east London, PE curriculum adviser Bob Bellow said early indications were that some schools were cutting back, with swimming one of the main casualties.
That could affect ethnic-minority girls in particular, he said. "In some inner-city boroughs, sport doesn't happen outside the curriculum. You don't see many Bengali girls in swimming pools at the weekend."
Platform, page 13 Martin Rowson, page 14