POLICY-MAKERS exhibit a worrying willingness to believe the results of educational research. This is not a problem with the kind of research that sets out to collect information as evidence, but it is a problem where the success of educational approaches is claimed through experimentation.

Too often we learn in the media of how some catchy research, feeding vitamins, daily exercise, parental involvement, or whatever, has led to enhanced performance as monitored by carefully designed tests. There is reason to suspect some at least of these studies may be fundamentally flawed.

The very act of showing an interest in something can generate measurable positive results. The classic study, for the Western Electric Company by Harvard University, was reported three-quarters of a century ago, in 1923.

The focus of study was the way individuals worked and different work practices were subject to trials under controlled conditions. A sample of the skilled craftsmen and assembly personnel were isolated and observed in specially constructed laboratories alongside their usual workplaces.

What shook the investigators was that management reported marked improvement among the rest of the workforce engaged in similar work but not selected as part of the experimental group. The researchers found this phenomenon held true throughout a decade-long study. If you show an interest, you get improvement.

The discovery passed into the folklore of science as the Hawthorne effect, from the name of the works. The effect pops up in a range of disciplines. Cows treated to music in the dairy produced more milk. But only for a while. As the tale goes, the farmer found milk yields slipping back to normal so he abandoned his experiment and stopped the music. The yield rose again, only to fall back.

Those researchers who must get it right take great pains to quantify and discount the Hawthorne effect. Medical researchers faced with a similar complication assume at least 30 to 35 per cent of any improvement from a treatment will be due to the placebo effect where attention, not medication, seems to be the curative factor.

An elegant study was carried out in her classroom by Margaret Reilly, then a postgraduate student at Strathclyde University. The investigation set out to show whether drill and practice, mediated by the classroom computer, would improve an important aspect of children’s mental arithmetic, their recall and accurate use of “times tables”.

In the straightforward design, the children’s accuracy and speed were measured before and after the introduction of the computer support. Not unexpectedly, average accuracy rose from around

60 per cent to just over 80 per cent, and the time reduced from a mean of six minutes to less than three minutes.

Group one who already performed consistently well improved from 88 per cent to 96 per cent. Group three started at an average of 47 per cent and crept up to 67 per cent. So far, a clear success for the approach would seem to be indicated. The results upheld the hypothesis that drill and practice would help.

But, and you knew there would be a but, the drill and practice were not begun until after the first six rounds of testing. By then the improving test scores had just about reached the plateau, nearing the 80 per cent. All the success pre-dated the experiment.

It was, as I say, an elegant piece of work, but not one that it is easy to broadcast. It does call into question the reliability of all those studies where “before and after” comparisons are made and used as the basis of the justification, or proof.

Without doubt the study needs to be repeated, and by others, but this report should cause a ripple of concern. The methodology and the results of many widely accepted studies may need to be put under scrutiny. Some will pass the test and so justify their implementation. But which? We owe it to a generation of children to check, and to check carefully.

Robin Frame is in the primary education department, faculty of education, Strathclyde University. Margaret Reilly’s study, Turning the Tables on Computers, is available in the library, Jordanhill Campus, Strathclyde University.