The slimming down of the curriculum is a dangerous measure which threatens the long-term health of the nation, according to the Central Council for Physical Recreation.
The council claims it will leave children less fit than ever and will squeeze out swimming lessons altogether.
The Government's decision to suspend the primary school curriculum for PE and some other subjects has also been attacked by the Physical Education Association, which says the subject will be pushed to the margins.
"This will be seen as a reduction in PE by teachers," said Nigel Hook, from the CCPR. "It is not in the long-term interests of Britain, which requires a fit, healthy workforce. Primary school is the most critical time for developing a love of exercise."
Research has shown that British children are becoming less active, he said, while the incidence of pupils who have heart disease is rising.
The CCPR is particularly concerned about the future of swimming. "There is nothing more important in primary PE than swimming. The chances of teaching every child to swim will be negligible. Mums and dads should be going barmy about this," said Mr Hook.
Meanwhile, the PEA argues that British pupils already take less exercise than most European children at only 90 minutes a week. With the CCPR, it is campaigning for a two-hour minimum.
The Central Council is writing to schools warning them that cutting PE still further is a dangerous move.