Mr Raeburn was commenting on Scotland's Sporting Future - Towards the Goal, the Government's pre-election update on school and youth sport published last week. His remarks follow a decision by Scottish Borders that may end primary school swimming instruction and close the pool at Berwickshire High.
Elsewhere Stirling is proposing to withdraw its staff member from the Fir Park ski centre at Tillicoultry. The slope is designed to teach young people to ski, often as part of a Standard grade physical education course or Scotvec module.
In a foreword to the report, Raymond Robertson, the Education Minister, paints a different picture and highlights "the great strides" on the range of initiatives developed by the Government, adding: "I genuinely believe that we are well on the way to delivering a sea change in sport and sports participation in this country".
Mr Robertson emphasises the Government's commitment and its backing "for the first time for a fully co-ordinated and cohesive strategy for youth sport".
The strategy covers schools, the club structure, coaching, physical activity and equality issues. Mr Robertson also highlights the contribution of lottery cash and plans for a Scottish Institute of Sport to develop talent.
The minister is expected to make an announcement shortly about the Sports Mark award scheme under which secondary schools will be eligible for Pounds 1,000 in recognition of their achievements.
The Scottish Office has also promised guidance designed to encourage authorities to make maximum use of sports facilities out of school hours.
But Mr Raeburn, whose association represents around a dozen school-based sports, spoke of "frustration" at the lack of support for teachers. The association has argued for help to encourage extracurricular sports as part of an extended curriculum.
He accepted progress had been made in some areas but commented: "Maybe a proper look at what is happening right now around the country would suggest that we are going backwards. How are we to do swimming at Scottish school level if children are not getting taught in primary?" Mr Raeburn says core funding is needed to finance a nationwide programme of activities. But the Scottish Office report points out that Pounds 250,000 has been set aside at the Scottish Sports Council to develop the youth strategy, Pounds 560,000 is directed at the team sport initiative, and another Pounds 400,000 at the Sportsmatch scheme, many of whose projects benefit young people.