Educational research is under the spotlight. Following the publication this summer of two critical reports, researchers are now having to justify their work against accusations of irrelevancy, bias and poor value for money.

As someone who has spent much of his career in the field of school effectiveness, I find it ironic that this should be happening at a time when - as I explain in my new book, The Road to Improvement - our research appears to be having a greater influence on government policy than ever before.

I have worked in school effectiveness research since the mid-1970s, when I joined Professor Michael Rutter’s three-year study of secondary schools. Fifteen Thousand Hours proved to be a seminal study, establishing, against the prevailing consensus of the time, that individual schools can make a difference and that all children can learn.

Over the next 20 years, successive studies have built on these findings by identifying which factors lead some schools to perform better than others with a similar pupil intake and by drawing attention to the limits of school influence.

It has taken central government 20 years to absorb the lessons of Fifteen Thousand Hours and a further 10 years to absorb those of School Matters, which studied 50 primary schools over four years. Had the policy-makers of the day acted upon those lessons - for example, about the importance of considering intake in judging a school and measuring progress as well as attainment, the negative effects of unbalanced intakes in secondary schools, the benefits of school self-evaluation, whole-school policies and parental involvement - many of the “tough” actions now deemed necessary might have been avoided.

Most importantly, the justification for improvement policies would have come from evidence arising out of independent research carried out with the active support of teachers, rather than from “top-down” political diktats. We will never know for certain how much could have been achieved and how much has been lost, but I suspect that the recent clamour for early retirements and the current low level of professional morale might have been avoided.

The relationship between government and educational researchers is bound to be complex. Politicians and their officials tend to want clear, unambiguous findings that they can use to justify particular policies. Researchers, because they seldom follow a “party line” and because their data often produces paradoxical findings, are too often dismissed as unhelpful.

But this is short-sighted and can lead to expensive policy mistakes - as recent history so vividly illustrates.

Attitudes, behaviours and relationships in schools jostle for position in an education system created in a somewhat haphazard matter over many years. Independent research offers a way of focusing on a segment of this complexity. By systematically collecting, analysing and interpreting information about particular aspects of life in schools, researchers - free of distorting power relationships over those who work there - can add to the understanding of educational processes and outcomes. Theories of how schools work effectively can be formulated on the basis of this information and used to help schools improve.

This type of research is difficult and contentious and the assumptions, methods and motivations of researchers need to be challenged constantly. But without research, who can provide objective evidence of what actually happens and suggest the most likely explanations of why?

As with medical research, “breakthroughs” are rare. But careful, cumulative work leads to greater understanding. This is why policy-makers need to invest in research programmes. In England, we currently spend on research less than one-fifth of 1 per cent of the education bill - far less than in other comparable fields or in other developed countries.

Equally, we researchers need to find more effective ways of communicating our findings both to government and to practitioners. If schools are to keep improving, teachers and heads need to be aware of the latest research. Yet, other than perusing The TES, few teachers have time to study research findings. The increasing numbers who study for higher degrees and specialist diplomas, however, certainly do so. Moreover, the recent emphasis given to research by the Teacher Training Agency should help alert practising teachers to the availability of research evidence.

I hope that I am not mistaken in detecting a recent change in Government’s attitude to the teaching profession. The promotion of Estelle Morris puts a former practitioner in a key position. Moreover, David Blunkett’s speech on the penultimate day of the Labour party conference celebrated the work of teachers and went some way to repairing the damage of the earlier “anti-teacher” rhetoric.

The statement that “naming and shaming” would not be repeated is a further sign that the punitive approach is being dropped. Thank goodness. The haemorrhaging of teachers and heads from our schools and the fall in applications for teacher training are grim reminders of the consequences of negative approaches. I trust that this rapprochement will now extend to educational researchers and that our work will be recognised for the vital contribution it has to make in the search for improvement.

Let us hope that all who work in the education service will now celebrate school improvement and that at least one of the lessons of school effectiveness research - that praise works more effectively than punishment - may finally have been learned.

Peter Mortimore is director of the Institute of Education, University of London, and vice-president of the British Educational Research Association.