It may be too early to use artificial intelligence to assess the writing ability of 10- and 11-year-olds, because it could “struggle to recognise skills in creativity, reader-based prose and persuasiveness”, a new research paper has said.

The report from Ofqual looked at assessment of writing at the end of primary school in countries around the world, including some that use AI in a process known as “automatic essay scoring”.

Human marking can be costly and unreliable, the report says, while computer marking potentially reduces the need for human markers.

It has been used in several countries for assessing technical writing skills in multiple-choice and short-response items but “automatic marking of extended responses reflects a greater challenge”, the paper says.

Quick link: ‘How to retain examiners? Show them the money’

Quick link: The exams system doesn’t put children at its heart

Quick link: Marking consistency varies across subjects, finds Ofqual

This is because extended-response questions do not lend themselves to ‘right or wrong’ answers, although trials have been conducted for writing tasks in Australia “with some apparent success” largely reliant on “mathematically based textual features” such as vocabulary, sentence length and complexity.

“The ability of [automated] systems to target deeper compositional type skills has been called into question,” the report notes.

“[It] may struggle to recognise skills in creativity, reader-based prose and persuasiveness…Concerns over validity may be too great for some at present.”

Automated marking could, however, be used as a monitor for, rather than a replacement for, human markers, the report says.

As well as looking at prospects for the use of AI, the research found that England is an outlier in using teacher assessment to judge pupils’ writing in its key stage 2 Sats.

Comparing assessment across 15 countries - or parts of countries - Ofqual found: “Unlike the current preference for teacher assessment in England, the majority of other jurisdictions currently assess writing via an external test.”

The paper found most used an ‘extended response’ test requiring a response of at least one paragraph in length.

Most countries adopted a “best-fit”-level approach where assessment decisions were made according to “fairly holistic descriptions of attainment”, whereas England used “pupil can” statements, showing specific statements of attainment.

“This report does not attempt to arrive at any conclusions as to which assessment method is ‘best’, because such conclusions would largely depend upon the purpose and uses of particular assessments within individual contexts,” the research paper states.