Evaluation - Giving fair marks across the board

14th February 2014 at 00:00
Radical chess-based assessment approach lessens bias, study says

Seven hours into a marking marathon and here, finally, is the last essay. But your eyes are glazing over, the post-caffeine slump has set in and the student in question is intensely irritating - can you really guarantee him a fair deal?

Educators everywhere regularly face scenarios like this, but a new assessment system has revolutionary promise: by applying principles from opticians' tests and chess tournaments, it could virtually eliminate inconsistency.

A trial of "adaptive comparative judgement" (ACJ) in five Scottish further education colleges has resulted in "exceptionally high" reliability, with marking "significantly" more consistent than under traditional methods.

The basic concept is simple: rather than teachers marking individually with a prescriptive checklist, they take two anonymised essays and decide intuitively which one is better. This process is repeated many times by a team of markers. The concept is similar to an eye test, where pairs of lenses are compared until the optimum one is found.

ACJ employs the "Swiss tournament" method from chess: rather than a knockout system, with the inferior essay dropping out instantly, all essays continue to be assessed after the first round. But they are paired with essays of similar quality in a fine-tuning process that ensures reliable rankings.

Research has consistently shown that traditional marking is hugely susceptible to human frailties. "The problem is that, typically, you mark in silos - you can't help but be subjective," said Matt Wingfield, managing director of London-based Tag Assessment, which developed the algorithm that allows the new technique to be applied on a large scale. "With ACJ, it doesn't take any longer to mark assessments, but you end up with significantly higher consistency and reliability."

Reliability is measured on a scale from 0 to 1, with 1 signifying a perfect system. Scores for traditional marking are typically between 0.5 and 0.6 but ACJ reaches 0.93 to 0.98, said Mr Wingfield, who chairs the e-Assessment Association.

The trial, evaluated by Alastair Pollitt of Cambridge Exam Research, looked at 72 handwritten essays and shorter answers from the former Angus, Aberdeen, Dundee, Ayr, and Banff and Buchan colleges. It produced a consistency rating of 0.96.

Australia, Sweden and Singapore have taken ACJ furthest, with the Swedish education agency running trials in science, design and technology, English and Swedish. The University of Edinburgh and the University of Limerick in Ireland have also adopted ACJ, with the latter using it for student teachers' work.

There was still widespread reluctance to employ radically different forms of marking, Mr Wingfield said, despite general acceptance that existing systems were flawed. And some organisations feared that ACJ would be misrepresented by the media as an educational version of "Hot or Not", the precursor to Facebook that asked users to decide which of two women was more attractive.

Last month Mr Wingfield spoke at an event in Stirling held by the Emporium of Dangerous Ideas, which proposes radical ways to improve education. "What ACJ does is employ the teacher to use their professional judgement to make decisions, and that shouldn't be a dangerous concept - but you get the impression from society that it is," he said.

The report, for the College Development Network, finds that ACJ could be an ideal addition to the wider range of assessment approaches encouraged by Curriculum for Excellence: for example, it would allow reliable comparisons of essays, videos, performances and posters. It recommends a bigger trial involving colleges and schools, and the College Development Network is appealing for volunteers.

The Scottish Qualifications Authority ran a small ACJ pilot in 2011, but the results have not been published. Martyn Ware, head of assessment, development and delivery at the body, said: "The results of the trial were encouraging and we are now reviewing a number of possible applications of the technique to support the assessment of our qualifications."

Read the report at bit.lyACJreport.

Log-in as an existing print or digital subscriber

Forgotten your subscriber ID?


To access this content and the full TES archive, subscribe now.

View subscriber offers


Get TES online and delivered to your door – for less than the price of a coffee

Save 33% off the cover price with this great subscription offer. Every copy delivered to your door by first-class post, plus full access to TES online and the TES app for just £1.90 per week.
Subscribers also enjoy a range of fantastic offers and benefits worth over £270:

  • Discounts off TES Institute courses
  • Access over 200,000 articles in the TES online archive
  • Free Tastecard membership worth £79.99
  • Discounts with Zipcar, Buyagift.com, Virgin Wines and other partners
Order your low-cost subscription today