With this year’s exam results ready to be posted, Paul Black commends an effort to analyse flaws in the system.

This book is “written for anyone concerned about the public examinations for GCSE, AS and A-level” and so is addressed to a wide audience, from pupils and their parents to educational experts. The author writes from the perspective of one inside the public examination process, drawing on extensive experience both as an examiner and as an officer of an examination board.

Most of the text is devoted to explaining such issues as how exam boards are run, how their work is subject to external scrutiny, who is responsible for designing and setting papers, the marking procedures, the awarding of grades and appeals procedures. In this mode the book is thorough and effective - for example, in explaining the critical importance of formulating a mark scheme and modifying it in the light of evidence of pupil responses.

This information, which is extensive and clearly presented, is a strong point of the book. However, where the book is critical and evaluative, the results are more uneven. On several contentious issues, such as the use of calculators in examinations or whether enhanced rates of success indicate that standards are rising or falling, there are clear and cogent arguments that explore public misunderstandings of the issues involved. The argument is particularly strong in examining the prospect of having a single national examining board, pointing out that to ensure parity of marking across a team large enough to examine simultaneously all GCSE English candidates in the country would be a logistical nightmare.

What does emerge across the chapters is the many problems raised by modular examinations. At a mundane level are examples of trivial errors - notably the school that, intending to record the completion of six modules by each of 20 candidates, presented instead records implying the completion of two modules by each of 60 candidates. More technically difficult are, for example, issues about deciding overall grades for a set of candidates among whom the same module has been examined at different times, with each occasion producing different pass marks and passing rates. Complications abound when considering appeals, disqualifications, the timing and conduct of borderline reviews and the logistics of the national examination timetables. This book could make a valuable contribution in drawing attention to the need to think through these issues more carefully.

The text is, however, weak on some of the key issues underlying examining policy. The discussion on how examination boards are governed and their finances lacks both a broader policy perspective and important detail, most of the published work on “subject pairs” analysis to compare standards between subjects is ignored, and there is no focused treatment of the validity or reliability of exam results. The author correctly emphasises that any result is an imperfect measure, but fails to discuss most of the sources of imperfection.

This matters: what should concern any group setting an examination is to decide, given the variability of any pupil’s response from one question to another, what numbers of questions are needed to achieve adequate reliability in the overall result. To be fair, such consideration is not to the fore in policy discussions, but this absence illustrates the lack of sophistication in the design of our public examinations.

More generally, one can only wonder at the compliance of the public in a system where it is not possible to tell candidates the probability of their result being incorrect by (say) one grade, which could be surprisingly high, and where there is no serious attempt to generate data to inform the provision of such information.

The book moves from neutral reporting to personal opinions about policy issues, which is welcome, but the separation between the two modes could have been made more carefully and explicitly. Notable in this respect is the closing chapter, which is mainly devoted to informed speculation about predictable problems in operating the proposed new ASA-level system. Some of the predictions are alarming, but these are matched by a plea for flexibility in evolution: here the author points, neatly and appropriately, to ways in which the problems raised by the tiering system have been alleviated by a willingness to adapt as lessons are learnt through practice.

In summary, this book’s

pattern of strengths and weaknesses reflects both the career perspective of its author and, more fundamentally, the strengths and weaknesses of our examining boards. Nevertheless, this is a significant effort to improve the public understanding of examinations.

The author has rightly judged that both public policy and public confidence are marred by lamentable ignorance, and he is to be praised for trying to help tackle this problem.

Paul Black is emeritus professor of

science education in the School of Education, King’s College London