How can it be possible for an exam board to award A grades to GCSE students who achieve only 45 per cent on key papers?
Edexcel's decision this year to lower grade boundaries on the country's most popular maths exam, revealed in The TES, captured national headlines.
The subtext of the coverage was familiar.
Here was yet more evidence of falling standards, suggested the critics. No wonder our universities and employers are struggling to find decent mathematicians, when young people can be deemed high-flyers on the basis of such flimsy evidence.
Is this charge fair? Readers should now beware. As ever, where exams are concerned, this is anything but simple. This summer, the lower boundary for an A grade on the two written papers for Edexcel's linear maths GCSE, higher tier, were set at 45 marks out of 100. The corresponding figure for the third paper, coursework, was set at 37 marks out of 48.
The exam is taken by more than 300,000 pupils. Although Edexcel's figures were the lowest of England's three boards overall, pupils taking GCSEs through the OCR board needed just 44 marks on one exam paper, and 51 on the other, to score an A, while, as with Edexcel, 37 marks out of 48 were needed on coursework for that grade. Pupils needed only 13 out of 100 to get a C grade on one Edexcel paper.
Across all three boards, the higher papers are designed so that questions become progressively harder. On these papers, a quarter of the problems are of C-grade difficulty, a quarter are set at B-grade, a quarter at A grade and a quarter at A*.
The justification for this is that pupils' maths abilities vary so much that questions have to be set at different levels so that a clear picture emerges of how strong the candidate is.
This makes it more difficult to decide whether grade boundaries have been set too low.
The boards say that it is misleading to quote a figure of 45 per cent needed to achieve an A, because A-grade students are not supposed to be able to even attempt all of the questions.
A quarter of the problems are targeted at A* pupils, they remind us. So it would be fairer to say that, to achieve an A, a student needed to score 45 marks from the questions they are expected by the examiners to be able to answer: those targeted at C, B and A candidates.
This amounts to 45 marks out of 75, or a more reasonable 60 per cent of the marks "available" to them.
So, if we take into account this technical information, is it fair to say that the boundaries were set too low and that the exam has been "dumbed down"? Putting aside questions about whether all this complexity is necessary, the boards still need to answer some difficult questions about the way they set grade boundaries.
Let us look more closely at that 45 per cent figure.
Given that a quarter of the questions were targeted at C-grade students, and a quarter at B-grade, a pupil could score 50 per cent, easily enough to get an A, without getting a single A-grade question right.
No wonder such experienced outside observers as Roger Porkess, the designer of the first modular maths A-level, branded the 45 per cent figure as "ludicrous".
A further conclusion is inescapable. Either standards really have fallen, or the questions were too hard in the first place.
In other words, questions which the board originally calculated a B-grade student should have been able to answer were actually set at A-grade difficulty level. Boards are notoriously reluctant to talk about papers being harder or easier from one year to the next, but this is close to the official line coming from Edexcel.
It also appears to tally with the experience of schools and pupils. One teacher from a leading private school where most students usually achieve top grades, said: "Some of the questions were harder than you would find at AS-level."
But is it enough to say simply that the papers were too hard, and that therefore the board, having made a mistake in its question-setting, was right to lower the grade boundaries?
This is too simple an explanation, according to at least one experienced voice. Barbara Ball, professional officer of the Association of Teachers of Mathematics with 30 years' experience as a maths teacher, said that it was right that papers should be harder now than they have been recently, and that Edexcel should not have responded by setting such low grade boundaries.
The justification for her position is this. Two years ago, the Qualifications and Curriculum Authority reacted to claims that modern exams are mechanistic, offering pupils little opportunity to demonstrate their ability to think for themselves.
Accordingly, it changed the exams so that 15 per cent of the marks on the written papers were assigned to "unstructured", problem-solving questions.
A reasonable person might suggest that this would make the new-style papers, introduced in 2003, harder than their predecessors. If the boards simply reacted by lowering grade boundaries, what would have been the point of introducing harder questions?
A report to schools from Edexcel's chief examiner lends weight to this view.
Some papers had become more demanding, the report said, because of the introduction of the unstructured questions. This year, the board had compensated "through the awarding process" (in other words, in the setting of grade boundaries). From next year, it said, the board would address these questions urgently, with "accessibility" given a priority in future papers.
So there we have it. Regulators make a well-intentioned attempt to raise standards, and examiners respond by lowering grade boundaries.
Admittedly, this is evidence from only one board, and it could be argued that in reality, Edexcel simply made a mistake by making the paper too hard, and should have said so in its report.
Yet anyone trying to put a more favourable gloss on the situation will be disturbed to read pupils' comments on a popular exam chat site. Messages tabled immediately after this year's exams see students complaining about how difficult the papers were.
But several responses are posted from those who sat the corresponding papers last year. Don't worry, they said, yes, the papers were hard but grade boundaries will be lowered.
And, of course, this is just what happened - as this year's students thankfully observed in August. One student, who reported getting an A* this year, told the site: "Seeing as I thout (sic) I was gona get a C it wasn't a bad grade!!"
Another said: "I missed out several questions. I got A*." Another said: "I was told I would not pass maths and I ended up (with) an A in maths."
To cynics, the situation is compounded by the existence of rival boards.
Tony Gardiner, reader in mathematics at Birmingham university, who has led calls for more emphasis on problem-solving, said he could not envisage a situation where standards could be raised in the current market-based exam system. No board wants to be in a position of setting exams where it is harder to get a top grade than with a competitor, for fear of losing market share.
Some will dispute the market's role in exam-marking. But Edexcel's apparent need to reassure schools that next year, papers will be more "accessible", hardly inspires confidence.
Whatever your take on a complicated picture, there seems little doubt that something is seriously wrong with maths GCSEs. Even the boards admit this.
Edexcel also conceded that this year it was possible to achieve an A grade without having mastered key areas of mathematical knowledge such as geometry or algebra.
Earlier this year, Professor Adrian Smith, who led an 18-month Government inquiry into the "crisis" facing the subject, backed employers' concerns that GCSEs were virtually worthless as guides to pupils' achievements.
Pressed on the Edexcel grade boundary story, Mike Tomlinson, who has proposed a diploma to replace GCSEs and A-levels, said he wanted a system whereby pupils had to master all elements of the subject to achieve a pass.
The problem is that employers and universities are going to have to wait years for the changes he proposes - making more unwanted headlines inevitable.
Teacher magazine 12