Improvements in the way physical education is taught could be undermined by cutbacks in the number of hours teacher-training colleges dedicate to the subject.
Inspection agency Estyn fears progress made through the PE and school sport (PESS) initiative may be eroded by Wales's training colleges.
The Assembly government-funded scheme was launched four years ago with the aim of raising primary PE standards and improving transition to secondary by sharing innovative ideas between clusters of schools.
In its annual inspection of PESS, Estyn said it was achieving this goal and primary teachers were "becoming even more confident and competent" at teaching PE. But the inspectors stressed that fewer hours, particularly on postgraduate certificate in education courses, "posed a threat to further progress and has the potential to undermine the progress made so far".
According to Estyn, five colleges have cut the time spent on training specialist PE teachers on their BA education courses since 2000, because of financial pressures. But these students still spent 80 to 200 hours on PE last year.
In addition, half of Wales's training colleges are converting sports facilities to other uses.
At PGCE level, three colleges - Trinity College in Carmarthen and the University of Wales in Bangor and Aberystwyth - allocated 15 hours or less to the subject in 2005-6. Estyn said this was not enough time to cover one or more of the four compulsory elements of primary PE - dance, games, gymnastics and swimming.
"The inadequate time allocated by institutions to PE places additional burdens on teachers in their early careers," said the report.
Estyn praised PE lecturers in all colleges, saying they had been providing high-quality training, but the reduction in hours was a "cause for concern".
Dr Janet Pritchard, head of Bangor's school of education, said: "There's only so much we can do on the initial teacher-training programme. We always stress 'initial' because we can't do it all.
"This underlines the importance of the induction year and the continuing professional development of teachers."
The Sports Council for Wales, which manages PESS, does provide CPD to help teachers who feel that they need more training.
Up to January 2006, 5,300 teachers took up these courses, but Sports Council chairman Philip Carling warned that this provision could be affected by future financial constraints. The Assembly government has agreed to fund PESS until 2008.
He said: "We support the recommendation that initial teacher-training institutions should ensure that trainees, once qualified, are readily equipped and confident to teach PE .
"Once the PESS project reaches the end of its funding, there will be limited support for CPD in PE."
In Merthyr Tydfil, co-ordinator Bev Symonds organises CPD for teachers. She said: "They're coming out of college without enough expertise and confidence.
"Teachers particularly lack confident in gymnastics because it's such a specialist activity, and a lot of them worry about safety issues."
Despite this, Ms Symonds said teachers were pleased with the impact PESS has made. More pupils have taken part in outdoor activities and dance sessions, and an emphasis on leadership has seen pupils from all key stages working with each other and getting involved in after-school clubs.
There are 63 PESS clusters, known as development centres, across Wales.
According to Estyn, a higher proportion of PE classes at key stages 1 and 2 in these schools were good than in the schools not taking part.
It has also made an impact on transition, with more teachers visiting each other's schools and sharing expertise.