Big is best, but does not come cheap
Despite being a statistician I appreciate the value of properly conducted qualitative research, which attempts to discover the "why" of what is happening, preferably in league with quantitative methods which may uncover the "what". But both types of research need to be properly planned and conducted. Poor research helps neither the education system, nor the cause of educational research itself.
At the National Foundation for Educational Research (NFER) we are addressing the question: "How exactly do you set about demonstrating that an educational initiative of some kind has had a quantifiable impact, in terms of measurable change in a desirable outcome?" It is not enough to measure something before and after an intervention takes place, since we cannot be certain that the observed change would not have occurred in any case. A common solution is to use a "control group" of schools and pupils not experiencing the intervention, in order to compare them with the "experimental group". There are often problems, however, in selecting the control group, persuading them to take part and ensuring they are not "contaminated" by the intervention the others are getting.
The sample sizes are also crucial. To make definitive statements about national trends, or the influence of some educational technique, requires large, nationally-representative samples of schools and pupils. In simple terms, if you want to make your study twice as accurate, you need about four times as many pupils.
Some current and much-publicised research seems to ignore the need for nationally representative samples - for example, by generalising from five schools in one area to the country as a whole.
It is also important to ensure that the data is collected "longitudinally" - that it refers to the same individuals measured at more than one time. Otherwise differences between individuals may obscure the effect you are trying to measure. Once you have collected such data, it will be possible to analyse it using a sophisticated statistical technique, such as multilevel modelling. These methods are important because it is essential to take into account as many of the factors which may affect the results as possible.
It is also essential to clarify the objectives of the intervention being studied and of the research study itself. An initiative may improve attitudes, and if such an improvement is its objective then a study which concentrates on measuring attitude changes is sufficient. If, however, the aim of the intervention is to improve actual attainment in a certain area, then this is what should be measured, with the appropriate instruments.
Our goal should be to make each study as simple as possible, but no simpler. Otherwise we may fail to find what we are looking for, and have wasted our time and effort.
I have nothing against small-scale, locally-based research, if it is well-planned and professionally conducted, and the limitations of its results are recognised. But to gain understanding of national trends and relationships requires large-scale studies containing nationally representative samples of schools and pupils. Such large studies need to be cafefully planned and conducted, to a sensible timetable and with proper statistical analysis.
These do not come cheap, and some sponsors of research may be startled to discover just what is needed to find out what they want to know. But if we are really serious about using research to discover important truths about education which will influence policy, then we need to start being honest about the resources which are needed.
Ian Schagen is head of statistics at the National Foundation for Educational Research, but the opinions expressed are entirely his own