Music teachers are isolated and lack training, according to Ofsted, which this week warned that too many music lessons are inadequate - particularly in secondaries.
The watchdog's report says that music was outstanding or good in around half of the primaries and secondaries it looked at. But overall, the quality and range of music lessons was too inconsistent.
It said part of the problem was that some teachers did not understand musical progress was only made when all the separate components of music came together.
"Helpful continuing professional development and challenge were rare. Developments in music education had gone unnoticed or been disregarded. In the primary schools visited, the subject leaders were frequently not given enough time to monitor and support the work of their colleagues."
Inspectors also reported a "major concern" about the amount of time given to music and found examples of primaries stopping music lessons in Year 6 to allow pupils to spend more time on English and maths.
"While there is no statutory requirement for subjects to be taught in every year, stopping music severely curtailed pupils' progress," the report said.
"They need continued development to make the most of their opportunities when they join the secondary school. Pupils in these primary schools were also very disappointed that they did not have music lessons."
Ofsted found that in schools where music provision was outstanding, it helped improve pupils' learning across all subjects, boosted their self-esteem and re-engaged them.
Local authority music services had contributed significantly to broadening provision by offering pupils instrumental tuition and the opportunity to perform in ensembles.
The report said now was a "very positive time for music education" because of considerable Government funding. But the money needed to be better targeted.
Chris Keates, NASUWT general secretary, said: "The decline in music provision has its roots in the previous governments' vandalism of excellent local authority music services due to changes in education funding."
She also blamed league tables, which were leading to schools marginalising subjects like music.