Many tests are like this. In the quest for objectivity and consistency, the skills they set out to validate have become so distilled into measurable bits that their essence has vanished. Passing proves only that you can pass the test.
Everyone in education must now hope and pray that the forthcoming National Professional Qualification for Headship does not go the same way. If it does, we will end up with senior teachers, apparently qualified for headship, who have merely demonstrated each of the practical competencies on a list. The whole will be somewhat less than the sum of the parts.
To be fair, the Teacher Training Agency has expressed awareness of this danger. Anthea Millett, its chief executive, has spoken and written of her determination that the qualification should be about leadership, and all concerned with it are robustly of the view that leadership can be taught and assessed. "We are not going to fight shy of the idea that you can train and assess leadership," said Frankie Sulke, head of policy at the agency.
This is why every candidate, including the whizziest on the fast track, will have to study a core module which encompasses strategic direction, and a clear long-term vision of how to improve teaching and learning in the school. There will be other modules, on the practical aspects of administration and school efficiency - whether prospective heads cover some or any of these will be discussed at their initial assessment.
As Ms Sulke said, the system has to be flexible enough to accommodate a variety of approaches. "There isn't just one model of leadership - we don't want 24,000 hero-innovators for example."
And, she added, it is important to keep sight of what you might call the practical extensions of leadership - "the ability to communicate the vision and translate it intopractice".
It is clear that the credibility and quality of the new qualificiation rests on the expectation that this multi-faceted quality called leadership can be broken down into well-defined and measurable parts. Where, for instance, might a tick-box approach - so convenient for consistency - fit in? An academic close to the debate said to me that, "One of the reasons why such approaches have fallen into disrepute is because the things that go into the boxes are the things that are easy to assess."
All of this, and much more, is being discussed at the training agency. The main forum for the debate is a development group, made up of the managers of the regional centres that will between them assess the needs of candidates, train them, and award the qualification. This group, according to Ms Sulke, "contains a real wealth of different kinds of experience from higher education, local authorities, schools, the private sector and business".
No one is saying anything for the record about the proceedings at these meetings. I understand, though that the range of opinions and experience makes for lively debate. "They're actually designing the materials as they go - 20 people round table," another person close to the group told me. "It's a very healthy debate, but so democratic that it's often slow andfrustrating. "
So, when the qualification is up and running next September, what, for aspirant heads, will be the route to qualified headteacher status?
The formal gatekeeper is the system of assessment centres. Teachers from the grant-maintained and private sectors can approach the centres directly, as can any other teachers who intend to pay their own fees. A teacher in a local authority school who wants a share of government funding (which presumably will be the majority of applicants) will need first to be approved by his or her authority. Some teachers and advisers have expressed reservations about this, because it is a common perception in the profession that some local authorities have their own, not always objectively based, ideas about which of their favourite sons and daughters are fitted for headship.
Ms Sulke pointed out, however, that early consultation showed teachers decidedly against putting selection in the hands of governors and heads of individual schools, so the involvement of local authorities naturally followed. She gave the assurance that "There will be clear procedures for authorities to follow. We are confident that they will fulfil their obligations fairly." There may well be an appeals procedure for rejected and disgruntled candidates.
It is also planned that there will be an additional early filter in the form of some self-assessment materials to help candidates decide whether or not they are ready to apply.
Once accepted, candidates will be helped to assess their training needs. Some "fast-track" candidates will have to do only the core module. Others will have to do more. The training agency sees the accurate identification of candidates for training as crucial to the purpose and credibility of the new qualification. If the training is to mean anything at all, it must attract not only those applicants who could be good heads next week, but those who feel that they have things to learn. As Ms Sulke said, "It's about preparation as well as about letting through those who are already prepared." At the same time, though, the course must filter out those who are simply not suited to the job. Fulfilling these two purposes is undoubtedly presenting yet another formidable challenge to the planners.
There is no doubt that the task of creating this new qualification from scratch against a deadline is turning out to be intellectually stimulating.I sensed this from Ms Sulke and from various participants and observers of the process, one of whom who told me "It is very very exciting because absolutely nothing is being lifted down from the shelves - everything is being created afresh."
This excitement stems from a feeling that because the quality of headship is, by every known piece of evidence, central to the quality of teaching and learning in a school, then this opportunity to improve the quality of headship has to be seized and made to work better than any existing programmes. As Ms Sulke said: "We are not just reorganising educational management training in this country, but improving it".
The biggest test, undoubtedly, is going to be whether the more elusive ingredients of headship really can be pinned down for training and assessment, whether the medal can be for dancing as well as for just doing the steps.