A new approach to assessment that replaces marking with a concept more akin to eye tests could help to secure the future of coursework and science practicals, according to experts.
Faced with evidence that teachers were cheating and over-marking, exam regulator Ofqual last week decided to detach practical work from new science A-levels. Critics claimed that the change would sound the “death knell for UK science education” and would even damage the economy.
But two assessment experts believe they have a solution that would not only rule out cheating but would also improve teaching quality and dramatically increase the reliability of exam judgements.
The simple but radical idea from Alastair Pollitt, one of Ofqual’s external advisers, and Matt Wingfield, chair of the e-Assessment Association, rejects actual marking. Instead, teachers would view pairs of videos of practical work from other schools over the internet and make a simple judgement about which one of the two is best.
A computer program would then use information from these initial judgements to select more closely matched pairs of science practicals for teachers to compare, with the process continuing to establish a rank order of thousands of candidates.
“The practical activities would need to be designed appropriately so that you could collect the evidence - that is all,” said Dr Pollitt, from Cambridge Exam Research. “Then it is easy. Think of the professional development in that too; every teacher is going to see work from around the country. That has got to be good.”
The method, known as adaptive comparative judgement (ACJ), uses the same concept as eye tests, where pairs of lenses are compared until the optimum one is found. England’s biggest school exam board, AQA, has commissioned research into the system and Sweden’s national education agency is running trials.
Mr Wingfield’s company, Tag Assessment, developed the algorithm that allows the technique to be applied on a large scale to quickly achieve finely grained collective judgements. He said that on a reliability scale of 0-1, with 1 signifying a perfect system, scores for traditional marking were typically between 0.5 and 0.6, but ACJ reached 0.93 to 0.98.
ACJ has yet to be adopted on a large scale by any country, but England could be an ideal candidate for a system that gets rid of marking. The quality of exam marking has become a major issue for many of the country’s schools and Ofqual has already warned that more stretching extended questions in new GCSEs could mean a further “shift.away from reliability”.
Last week, the difficulty of accurately marking science practical work was raised by Ofqual’s chief regulator Glenys Stacey. “Where you are assessing a physical competence, it is generally accepted that it is hugely difficult to fine-grade it,” she said.
That was one reason why the watchdog decided that practicals should no longer contribute to overall science A-level grades. Instead, students will receive a separate passfail mark for their practical work. Ofqual says the new checks will boost the quality of practical work.
But the problems around assessing practicals could be countered by ACJ because a finely graded rank order can be achieved through a large number of straightforward judgements, its proponents claim. Grade boundaries could then be set at certain points on the rank order.
“The beauty of this system is, because we are encouraging these judgements to be made by groups of people, we get away from the potential bias, intentional or not, of individuals,” said Mr Wingfield.
The computer program could also instantly identify any teachers whose judgements were out of line with the majority, potentially flagging up the need for more training.
In England, the system could be up and running by 2015, in time for the introduction of the new science A-levels, Dr Pollitt said.
An Ofqual spokesperson said: “Adaptive comparative judgement can have a place in assessment. However, we do not believe it is a suitable model to use when assessing practical skills in the sciences.”
