Evidence-based policy-making is very much in favour. The Economic and Social Research Council and the Department for Education and Skills have both recently funded research centres devoted to systematic research reviews, in the social sciences generally and education specifically.

This recognition that policy must be based on sound evidence of what works is welcome, but does the Government properly understand what it means to take account of research evidence?

The recent White Paper justified the introduction of specialist schools with the claim that specialist schools have “proven success in raising standards”.

The claim for “proven success” arises from a study carried out for the Technology Colleges Trust (see www.tctrust.org.uk ) and published last summer. It carried out two sets of “value-added” analyses; one charts progress from KS3 to GCSE and one from KS2 to GCSE.

In fact, neither analysis was a true value-added one since they were carried out not using data on individual pupils, but taking the average KS3 or KS2 scores for each school and comparing these with the average GCSE scores.

It is known that such aggregate level “pseudo” value-added analyses may be misleading when comparing types of schools, yet the report does not stress the potential flaws in its approach.

The report compares KS2 and GCSE performance and claims that “irrespective of intake differences, specialist schools had 53 per cent A*-C passes at GCSE compared to 43 per cent for the remainder”. The problem is that the average KS2 score is based on a different set of pupils from the GCSE score, because of pupil mobility. It may well be the case that the specialist schools in the study acquired high-achieving pupils from nearby comprehensives, thus boosting their pseudo value-added scores.

For these reasons the results of the analysis are unsafe and should not be used to draw conclusions about the relative merits of different kinds of schools. The “evidence” for the so-called “proven success” of specialists does not stand up to close examination.

This study of specialist schools was not published in a peer-reviewed journal nor, it seems, subject to peer review via seminars or conferences. This is not necessarily a criticism of the authors of the report, since many research results become public through reports to sponsors. It does, however, raise an important issue for those who wish to use the findings, which may be controversial and open to technical criticism.

The role of peer review is to air such issues. It is reasonable to assume that, had the report been reviewed, its weaknesses would almost certainly have been raised and even perhaps accepted by the authors.

It is not clear whether the authors of the White Paper sought views on the adequacy of the research before using it. Some within the DFES would also have spotted flaws in the report and warned against taking it at face value.

But the research supported government policy. It is hard to escape the conclusion that this is what drove the decision to use it as “evidence”.

If the Government is serious about the use of evidence in policy formation it surely needs to avoid such opportunistic use of research. At the very least it should expect to use research that has been through a validation process, typically by stringent peer review and where possible by means of replication. Indeed, it could be argued that government should require some kind of review process before any results are regarded as “evidence”.

This case does not augur very well for the future. If they wish to engage people in serious rational debates, those responsible for formulating policy will need to exercise considerably more respect for soundly-based evidence than has so far been apparent.

Harvey Goldstein is professor of education at the Institute of Education, London. For his detailed critique of the White Paper see www.ioe.ac.ukhgpersonalspecialistschools.