Markers' personal preferences affect KS2 grades

22nd September 2006 at 01:00
A large proportion of the differences in national test results in English are due to inconsistency between examiners rather than pupil ability, research shows.

An analysis using this year's writing test for 11-year-olds reveals that on every element of the marking scheme, pupils' ability accounted for less than 70 per cent of the variations between their scores.

On one element, handwriting, marker inconsistency accounts for more than half of the differences in marks, with actual performance only explaining 47 per cent.

The National Foundation for Educational Research study said: "Even closed response questions that would be expected to show complete consistency can display higher than expected levels of variation between graders. In writing the analysis shows the difficulty in achieving objective measurement of ability."

The study found that some of the marking differences on the writing paper cancelled each other out. But overall, inconsistency still accounted for more than a fifth of the variation in scores.

Markers were more likely to consistently give generous or harsh marks when faced with pupils with poor spelling ability. It also showed that the inconsistency of marking increased slightly if the candidate was a girl, had used extra paper or had written in paragraphs.

The researchers write: "Different graders may prefer different styles of writing or attach greater weight to different elements of a pupil's answer."

But they believe that these differences will not affect school comparisons as they are likely to cancel each other out when looking at large groups of pupils.

The reading paper, with multiple choice and short response questions and requiring less marker judgment, stood up better to the analysis. Pupil ability explained nearly 97 per cent of the differences.

The study involved 49 test scripts each independently evaluated by nine experienced examiners last autumn.

Kathleen Tattersall, chair of the Institute of Educational Assessors, said examiners had been trained two months before the research began. But in reality it would have come immediately before they marked tests. Their performance would also have been monitored closely leading to greater consistency.

"Exploring the importance of graders in determining pupils' examination results using cross-classified multilevel modelling," Tom Benton. To be published soon at brs.leeds.ac.ukbeiwwwel.htm

Subscribe to get access to the content on this page.

If you are already a Tes/ Tes Scotland subscriber please log in with your username or email address to get full access to our back issues, CPD library and membership plus page.

Not a subscriber? Find out more about our subscription offers.
Subscribe now
Existing subscriber?
Enter subscription number

Comments

The guide by your side – ensuring you are always up to date with the latest in education.

Get Tes magazine online and delivered to your door. Stay up to date with the latest research, teacher innovation and insight, plus classroom tips and techniques with a Tes magazine subscription.
With a Tes magazine subscription you get exclusive access to our CPD library. Including our New Teachers’ special for NQTS, Ed Tech, How to Get a Job, Trip Planner, Ed Biz Special and all Tes back issues.

Subscribe now