Professor Gray told a conference of researchers at London University's Institute of Education that schools encounter difficulties in maintaining improvement rates beyond three to four years.
The pattern emerging from school improvement research is that most schools appear to reach a plateau after an initial gain in pupil performance. A similar pattern can be discerned from changes in GCSE pass rates, he said.
In recent years, results have remained relatively static, following rapid improvement in results between 1987 and 1991.
The results of an unpublished survey by Professor Gray suggest that schools have not concentrated on achieving substantial improvements in the quality of teaching and learning. Teachers who were asked to list measures that had been introduced in their schools did not tend to report substantial changes in teaching and learning.
Schools are to be required from next year to set targets for improvements in national curriculum tests in English, maths and science at seven, 11 and 14.
Schools will also be required to set targets for improved pass rates at GCSE.
However, Professor Gray suggested that research indicates that the scope for improvement by schools is probably fairly limited. Research suggests that only between 10 and 15 per cent of the variance in performance can be directly ascribed to the effectiveness of the individual school.
In attempting to assist schools, researchers need to monitor the changes in the value-added scores over several years. School improvement is probably more difficult than is officially recognised, he said.
The main reasons why some schools fail to improve tend to be: lack of will or effort; lack of knowledge about how to bring about improvement; lack of appropriate personnel or resources.
At the same conference, Professor Michael Barber, head of the Department for Education and Employment's standards and effectiveness unit, emphasised the part that has to played by local education authorities in setting school improvement targets.
Local authorities will be required to have a system for monitoring schools. They will be given available data on performance and will be involved in data interpretation training for staff.
The Government is expected to publish national benchmarks in the next couple of weeks as a guide to schools on setting targets.