Progress in PE is not as fast as it could be given the extra investment, the Qualifications and Curriculum Authority said.
Many secondary schools are cutting lunch breaks - a popular time for sport - in an effort to contain badly-behaved teenagers, it said.
Ministers have committedpound;459 million over three years to transform PE and school sport, on top of pound;686m already being spent on better facilities.
They believe getting more children involved in sport will help cut obesity levels, improve concentration in class and develop the next generation of sports stars.
The Government has set a target of ensuring 75 per cent of pupils spend at least two hours each week on high-quality PE and school sport by 2006.
But two reports from the QCA said that standards were not as high as they could be and that secondary schools found it very difficult to meet the target for 15 and 16-year-olds.
"There is significant progress to be made if schools are to reach the target by 2006," its review of the PE curriculum and qualifications says.
"The time to get involved in school sport is becoming more limited in many secondary schools. This is due to the trend to shorten lunch hours, often to contain poor behaviour."
A separate assessment from its PE team, setting out issues for the future of the subject, said PE was not showing the improvements evident in many other subjects.