You started your annual lecture, “OFSTED and its critics”, with a brilliant opening remark: “Those of you who have travelled long distances to be here tonight will probably, given my title, be wondering whether you’ll make the last train home.” Your critics are indeed legion.

The problem with inspections is that teachers and headteachers have no sound reason to believe that the judgments made are valid, or value for money, or tell them anything accurate about the school that they didn’t already know.

The first step is to ask whether or not the judgments are reliable, that is, consistent from one inspector to another. If different inspectors would render different judgments when observing the same class, then the whole edifice crumbles - for which inspector is to be believed? Can you demonstrate that any pair of your inspectors would arrive at acceptably similar judgments? And can you demonstrate that the classes they observe are representative of ordinary lessons? When are you going to publish evidence that these minimal requirements, these basic standards for evidence, are being met? Better late than never.

Let us assume, generously, that you were able to demonstrate that all your inspectors were highly likely to agree with each other. This kind of reliability is no guarantee of validity. The inspectors might all be expressing the same prejudice.

Validity (which you unfortunately confused with reliability) can be established in a number of ways. One standard procedure would be to see if your inspectors’ judgments matched other evidence. I have offered again and again to make available value-added data on hundreds of schools (anonymously), so that this measure could be compared with inspectors’ judgments. Publicly you always say “we must talk”, but you take no action. Are you afraid of what the data might show? A study could be done for about the cost of one inspection.

In summary, there are well established, simple procedures for giving judgments credibility. OFSTED has signally failed to attend to these procedures. Its standards can only be judged to be those of a failing organisation, since it has not adopted even the most elementary procedures for establishing that it can do the job it claims to do.

You made the following points in defence of OFSTED judgments: * “Judgments” on a failing school are corroborated by a second HMI inspection.

This is no more satisfactory than other internal self-investigations.

* The approach taken for recording evidence and judgments allows easy checking for internal consistency which we do through monitoring.

Well, this is news. You haven’t published any evidence of internal consistency, have you? And if your inspectors so consistently agree, why did we have this statement in a report on inspections? “The majority of registered inspectors were able to make appropriate decisions about conflicting evidence” (Independent inspections of secondary schools 1993-1994 a progress report, HMI, 1994, page 10). This clearly implies that the majority of registered inspectors had to reconcile “conflicting evidence”.

* The core judgments on attainment draw heavily from hard evidence and quantitative data.

In that case, why do your inspectors spend about 70 per cent of their time in schools sitting in classrooms - a very expensive activity? And with regard to this “quantitative data”, I do not believe schools are given sufficient description of exactly how the quantitative data is interpreted. What little evidence we have is that it is used in a very rough-and-ready way.

Furthermore, your statisticians seem to be adopting the generally low standards of your organisation. Commendably they are adopting a recommendation we made to OFSTED some years ago and using a method of examination analysis called relative ratings. Unfortunately they fail properly to acknowledge the original work in Scotland, and they use a simplification without acknowledging this. In our Year 11 Information System (YELLIS), we compute relative ratings properly. You can take that as a benchmark for OFSTED to set as a target in your action plan.

* If anything, inspectors err on the side of caution and are certainly very wary of making any judgments they cannot back up with evidence.

Wary they may be, accountable they are not.

* Very few judgments are challenged by schools.

This is largely due to fear of making things worse and to the lack of any independent procedure whereby schools can defend themselves.

It is always wise to be particularly nice to people who can close you down, or humiliate you in public, and whose judgments cannot be questioned.

I would refer you to the case of Breeze Hill, which wished to take OFSTED to court but was advised by the legal profession that the judgments could never be challenged.

* The great majority of heads and governors are content with the management, evidence base and findings of the inspection.

Perhaps you would refer us to the independent, confidential survey on which you base this statement?

* Some heads are concerned when the findings are not critical enough, as they see it, of various weaknesses in the school.

Heads who cannot deal with various weaknesses in their school could offer professional advice on how the mechanisms and resources that they have available could be improved. It is on such advice, negotiated with teachers’ associations, that changes should be made.

No one favours children’s life chances being jeopardised by inadequate teachers but we also favour of proper, professional personnel practices that sustain and motivate an effective teaching force.

The new framework encourages inspectors to look at trends in the school’s performance, not just a snapshot. Trends, like snapshots, need to be based on good evidence.

In summary, OFSTED should be evaluated. It cannot be taken as valid simply on the basis of assertions of its current leader, even such a charming one as yourself. Will you use your influence to set up an evaluation to set a world-class-standard?

Schools that are similar should be paired and then, by tossing a coin, one school in each pair should be subject to a regime of OFSTED-type inspections and the other schools should instead receive funds equivalent to the costs of inspections. This will all occur in the current context of published indicators such as those in the school performance tables, soon to be enhanced with value-added indicators.

You may say such an experiment is impossible but one on these lines is already being conducted in the Netherlands, with feedback to hospitals. It would be good if England could lead the way in social science instead of being so dreadfully backward as to continue to support your antiquated inspection system. A society which gets its social science wrong will have social troubles as surely as an industry that gets its physical science wrong will have production problems.

Schools must be accountable for outcomes but those outcomes must be credibly, reliably, efficiently, validly and fairly assessed.

Carol Taylor Fitz-Gibbon is professor of education and director of the curriculum, evaluation and management centre at the University of Newcastle-upon-Tyne