Strong leadership and a clear emphasis on improving classroom learning are among the key factors in combating indiscipline, teachers have told education inspectors.
In a major report on discipline issued on Wednesday, HMIE says that relationships and behaviour posed problems for just over half of secondaries and more than a quarter of primaries. These also tended to be schools where behaviour management was handled inconsistently.
A small minority of schools - one in 12 secondaries and one in 30 primaries -were judged to have "important weaknesses" in pupil relationships which were affecting the learning of others, according to inspection evidence.
The general conclusions drawn by HMIE on the implementation of the Scottish Executive's Better Behaviour-Better Learning strategy are tentative at this stage, two years after its inception. Education authorities could boast "progress on a number of fronts", while schools had started "some encouraging initiatives".
The inspectorate confirms evidence from other sources that secondary schools represent the main challenge. There were "considerable variations" in the perceptions of secondary teachers about the management of behaviour: around a third were concerned about how their schools dealt with it and nearly a fifth felt there was "insufficient mutual respect" between teachers and pupils.
Inspectors report: "In almost all secondary schools where there was strong leadership focused on improving effective learning and teaching, the proportion of teachers who believed that indiscipline was dealt with effectively was higher than the national average. This included schools in socially disadvantaged communities."
The report continues: "In contrast, in almost all secondary schools where leadership was not clearly focused on improving pupils' learning, teachers' views of the quality of managing pupil discipline were below, and often well below, the national average. These included schools which served communities with low or moderate levels of deprivation."
The contrast with primary schools was stark. "Overall, the responses of primary staff showed a marked confidence in their own, their colleagues' and their managers' capacity to manage behaviour," the report states.
Graham Donaldson, head of the inspectorate, acknowledged the progress made in sustaining good behaviour, but said that there were "heightened challenges" for teachers and others because of social changes around schools and the inclusion of difficult pupils.
Serious indiscipline was "neither inevitable nor endemic", Mr Donaldson said. But major breakdowns were occurring in a small minority of schools.
The result was a "major loss of learning for pupils and significant distress for staff".
Mr Donaldson urged schools to move quickly to deal with such situations but added: "Clear-sighted and resilient leadership is also required to ensure that short-term actions taken in such circumstances are not a substitute for the development of longer-term strategies to promote positive behaviour and better learning for all pupils."
Authorities should monitor the impact of initiatives on learning more rigorously, inspectors say.
The report gives notice that HMIE will continue to monitor how the Executive's discipline action plan is being implemented. The next step for inspectors is to evaluate how well strategies for improving behaviour lead to improved learning. This will be the focus of the next HMIE investigation in two years' time.