New Labour’s enthusiasm for diversity and choice in secondary education is creating resentment and resistance in old Labour. Equally important is the consequential renewal of energy fermenting in what one might call old Conservatism. The Conservative party repents its abolition of the grammar schools and aspires to a more selective system. Initially this was done by stealth and attrition of the comprehensive principle. Now a return to selection is open policy.

New Conservatism rather half-heartedly accepts comprehensive schools and therefore opposes the restoration of the grammar schools. The urgent but difficult task of raising the quality of education for all is not, on this view, to be achieved through a return to selection but through diversity. John Major, an old Conservative, demands selection, apparently against new Conservative Gillian Shephard, to put clear water between himself and New Labour.

This mirrors the tension between old and new Labour, which was vividly illustrated in the conflict at the 1995 Labour party conference between David Blunkett and Tony Blair, representing education policies committed to diversity and choice, and Roy Hattersley, vigorously opposing dangerous ideas which he believes betray the comprehensives.

Perspectives currently fall into three camps, which I call the selectionists, the traditionalists and the evolutionists.

The selectionists are (with significant exceptions) from the political Right. Asserting that comprehensives have failed, they believe in the 11-plus and grammar schools. Both will probably reappear, but in new guises, for the selective principle, like the comprehensive principle, can take a variety of forms.

The traditionalists see the comprehensive school as under threat from all sides, including from within the Labour party itself, and counter-claim that comprehensive schools are succeeding and so must be protected from change.

The evolutionists claim, as I do in my Demos pamphlet, The Mosaic of Learning, not that the comprehensives have failed but that in their present form they are not good enough and that the solution does not require a re-introduction of selection. Evolutionists are open-minded about the forms schools take, provided the comprehensive principle is not infringed.

The distinction between a comprehensive school and the comprehensive principle is important. Definitions of both by politicians are curiously rare, and have been since Tony Crosland’s The Future of Socialism (1956) or Harold Wilson’s attempt in 1964 to sell comprehensives to a suspicious electorate as “grammar schools for all”. Central to the comprehensive principle is opposition to selection by general ability into schools with a restricted ability range. The early comprehensives overlooked the impact of curriculum but sociologists soon documented how the earlier institutional differentiation of grammar and secondary modern schools reproduced itself within the comprehensives to perpetuate curricular inequalities.

The national curriculum, a new Conservative reform, means that comprehensive schools are now effectively “interchangeable” on the aspect of schooling which is potentially so divisive - who has access to which kinds of knowledge. The battle for equality of curricular access has, through the continuing evolution of the comprehensive principle, been largely won. Opposition to curricular inequity is now inherent in the comprehensive principle.

For some traditionalists, the comprehensive school is ideally about “mix” or balance - co-educational, with a full range of ability, social class, cultural, ethnic and religious background. For others, it is ideally a neighbourhood or community school. In practice, neither ideal is fully realised: a school’s natural locality rarely has the desired pupil mix and “bussing” to achieve one undermines the neighbourhood ideal.

As ideals are inevitably compromised, the traditionalists’ hope that comprehensive schools should be interchangeable is doomed. This has led to a conviction that it is dangerous to allow significant differences between comprehensives, for this encourages demands for parental choice, which entails popular schools choosing parents, or selection by the back door.

Obsessed by equity in everything, and a concern that it should be to no child’s advantage or disadvantage to attend one comprehensive rather than another, the traditionalists continue to want comprehensives to be virtually identical. In their view, local education authorities should devise a secondary school system which approximates to interchangeability, for example by adjusting catchment areas. Talk of parental choice, diversity and self-managing schools is heresy to the traditionalists, Since the 1988 Act, the idea of parental choice of school has gained ground. Schools with a curricular and philosophical specialisation have been developed, alongside religious schools, in countries as different as the United States, Denmark and Australia. Choice is popular and does not necessarily conflict with the comprehensive principle as defined above.

The 1994 Organisation for Economic Co-Operation and Development international review, School: a matter of choice, distinguishes between “demand-led” or “competitive” choice and “supply-led” or “pluralistic” choice. In the former (new Conservative) option, choice means encouraging schools to compete with one another along narrow criteria (test results and league tables). The aim of the latter (new Labour) is to increase the range of schools from which choices might be made.

The OECD report is clear that competitive choice tends to create frustration when it is not accompanied by pluralistic choice. It concludes: “The only way in which more choices can be satisfied is if preferences are more evenly spread. Schools that differ by pedagogical style or subject balance are more likely to be chosen evenly than if they differ by social or academic status. Demand pressures alone are rarely enough to create such diversity. Initiative to diversify educational supply may therefore be needed . . . Under a uniform model of schooling, choice is more likely to reinforce educational hierarchies than to improve educational opportunities or overall quality.”

The standard, look-alike comprehensive - designed to match an idealised “balanced intake” or “neighbourhood” version - is in decline, especially in urban areas where there are enough secondary schools to make diversity and choice feasible. Diversity is a means of eluding both the return to selection and the hierarchies of schools based narrowly on league tables.

New Labour should actively encourage more diversity than the Conservatives ever did, but this extension of specialised schools must be protected by the comprehensive principle’s two features: the “negative” aspect of banning selection by ability, which requires control over admissions, and the “positive” aspect of preserving students’ curricular entitlement, safeguarded by some limited national curriculum.

As evolutionists displace traditionalists, they design policies to attract most parents and so isolate the selectionists, whose policy of “back to the sec mods” will, I hope, get its just electoral deserts. The problem is the emergence of selectionists within the upper reaches of the Labour party - including those who advocate some form of selection by ability or send their own children to selective schools whilst claiming adherence to a comprehensive principle.

To end the confusion, these crypto-selectionists should come clean and declare themselves as evolutionists or selectionists. Within all political parties there are evolutionists who favour the process of progressively clarifying the comprehensive principle and devising a variety of schools which conform to it. They will respond pragmatically to parental desire for choice by creating diversity without inequities that damage the individual or society.

Traditionalists and selectionists prefer to fight a bloody educational battle among themselves. For the evolutionists of new Labour and new Conservatism victory will not be easily won, though the long-term future of comprehensive secondary schools probably lies in their hands.

David Hargreaves is professor of education at the University of Cambridge