Skip to main content

What comes first, evaluation or the policy?

No country has been more reluctant than Britain to hold itself up to external scrutiny, says Maurice Kogan. This Government is keen on evaluation. Britain is the prime example of the evaluative state which has produced a whole zoo of evaluators, inspectors and auditors. But no country has been more reluctant to hold itself up to external scrutiny.

The most benign form of scrutiny is a policy review conducted by the Organisation for Economic Co-operation and Development. Since the early 1960s, some countries have asked for and received two or three such reviews. England and Wales have been reviewed once, in 1975, when shamed into it by OECD prodding.

The resulting report was interesting and stringent, but was pinned down to a narrow set of questions by ministers and officials who took no steps to allow the main interest groups and critics of the system to get a hearing - quite different from the reviews mounted in other countries. If ministers are so proud of their reforms, why don't they get them looked at by international experts?

What is an OECD examination? The OECD is a body in which member states come together to identify economic, social and educational problems and trends, and to provide an opportunity for their analysis and the search for common solutions. Its role is therefore in effect that of a consultant, and its power is persuasive.

It is governed by a body consisting of ambassadors especially appointed by each of its member countries (the UK representative is Keith MacInnes). Its membership includes western Europe, Australia, New Zealand, Japan, Canada, Mexico and the United States. Observer status has been granted to four central and eastern European countries and Korea. An informal dialogue has been established with the Dynamic Non-Member Economies, such as Hong Kong, Malaysia, Singapore, Taiwan, Thailand and some Latin American countries (Argentina, Brazil, Chile).

In 1961 an OECD meeting in Washington decided that countries should undertake comprehensive policy and planning surveys of their education systems. There have since been 39 reviews of its member countries and three reviews of associate or observer countries.

The content of the reports has changed. At one time, they were mainly concerned with education's contribution towards economic development.

Later they concerned questions of access and equality. More recently there has been an emphasis on quality and relevance. The reports thus respond to the general movement of educational policy. A fascinating question is how far this movement is the result of activities such as OECD examinations and how far it represents similarities in the development in educational systems following some kind of shared evolutionary pattern.

The process varies slightly in each case. A member country asks for a review, or occasionally is prodded into asking for one. The motives in asking for a review are mixed. Some countries want their policy-making to be critically assessed. That was the case in the four reviews in which I have participated. Some want external legitimation of what they intend to do anyway. Others regard it as demonstrating membership of the international community.

There is negotiation on the scope of the review, the membership of the examining team and how costs are shared between the OECD and the country. The scope of the review can be comprehensive - and in the past mainly has been - or it can deal with one major sector, as it has done recently in reports on youth education in Denmark or higher education in Austria, the Czech Republic and Finland.

Countries first produce a self-evaluative report. The best are based not simply upon official materials and judgments, but incorporate the views of interest groups and of researchers. The OECD examiners offer a critique of the first draft. The rapporteur of the examiners and the officer responsible for the reviews then visits. The examiners meet ministry officials, usually the minister, and major interest group representatives and also visit some key institutions. This enables the team to get information together and to sharpen up the examiners' questions in advance of their visit.

The visit takes place over a period of between seven and 14 days. The examiners number three or four, always from different countries and with a balance of expertise. The programme is carefully worked out by the country being evaluated with the chairman and OECD officer. The best programmes have a good geographical range of visits and cover all sorts of perspectives. The Finnish higher education examination team met ministry officials from several departments, a minister, senior members of a range of universities, polytechnics and other research institutions, research councils and their equivalents, employers, students, journalists and members of parliamentary committees.

The visit has to strike a good balance between extensive and intensive work. The major institutions must have an opportunity to make statements at which all or most of the examiners are present. But it is also important that one or two of the examiners should be able to meet with individuals.

It is essential for the examiners to have time for reading and sustained discussion so that material is analysed as the examination progresses. In the recent Finnish examination, a summary of the first draft of the examiners' report was produced by the time they left Finland.

The draft report goes back to the ministry for correction of errors of fact or of interpretation and to the OECD for critique and checking. A version is finally agreed.

There is then an examination meeting, usually in Paris, conducted by the OECD education committee. The examiners' report contains questions and recommendations for discussion. A degree of stage management ensures that the questions are relevant and the answers given authoritative. The national authorities should be led by a minister and preferably consist not only of ministry officials, but also of some of the other key players from the country.

The reports of the examiners and the country's national education authorities are eventually published. Some countries organise their dissemination through press conferences, seminars and translation into local languages. The national authorities are invited to deliver a follow-up report to the education committee within two years.

Several years ago the OECD experimented with a "thematic review"on comprehensive schools in several countries. The exercise is now being repeated to study higher education in four or five countries.

In the past, OECD examinations have been criticised for their "social democratic" bias; the emphasis was thought to be too much on equality and not enough on efficiency. Effectiveness and efficiency are now important focuses of reviews. Attention is also been given to the efficacy of the follow-up process. The latest procedures ensure a more interactive process, although since the procedure is voluntary there is no commitment to follow it up.

The British Government is remarkably immune to critique. The National Audit Office could do more to investigate Department for Education efficiency (the cost of "reform" - consultancies and all those quangos). But we could do with a really good bash on main policies in public and international view. The shift to markets, the emasculation of local government, the imposition of selectivity and managerialism in universities would do for starters.

Professor Maurice Kogan has been an examiner on four OECD reviews since 1969 and was the author of an OECD report: OECD Education Policies in Perspective: an appraisal of OECD country educational policy reviews (1979).

Log in or register for FREE to continue reading.

It only takes a moment and you'll get access to more news, plus courses, jobs and teaching resources tailored to you