Campaigners warn that poor facilities and an overcrowded school curriculum are restricting children's physical activity and having a detrimental effect on their health.
The Office for Standards in Education revealed this week that one in five schools has sports facilities that are not up to scratch.
The double blow comes at a sensitive time for a government which has promised more than pound;1 billion for PE and school sport in England.
Ministers recently threw their weight behind London's bid for the 2012 Olympics and the country has been basking in the afterglow of England's Rugby World Cup triumph.
But the amount of time young people in England spend on physical activity decreased by 11 minutes to eight hours 12 minutes per week between 1999 and 2002, Mr Caborn said.
Falling activity levels followed the Government's promise in 2000 to raise significantly year on year the average time spent on sport and physical activity by those aged from five to 16.
That target was replaced in 2002 with a promise to increase to 75 per cent the proportion of pupils taking part in at least two hours of high-quality PE and school sport.
International figures show that the UK is one of the fattest countries in Europe. One in 10 six-year-olds and one in six 15-year-olds is obese and children are getting bigger faster than anywhere else in the developed world.
Sue Campbell, chief executive of the Youth Sports Trust and a government adviser on school sport, said that falling activity levels are a result of changes in children's liefstyles. "Evidence shows that pupils are less active outside school and spend more time sitting at computers," she said.
Ofsted found that the quality of PE lessons had improved in almost two-thirds of schools inspected.
Colin Moynihan, shadow sports minister, said: "I worry that the Government will not reach its targets. Until all the promised sports facilities are built and all schools have time for sport and PE, we will continue to raise expectations without delivering."