Exams: Ofqual admits to 'shying away' from talking about test result reliability

18th September 2009 at 01:00
Fearful of negative headlines, exams regulator hired PR consultants to find alternative terminology

Original paper headline: Error: not a word in Ofqual's vocabulary

The exams regulator has admitted to "shying away" from talking about test result reliability for fear of negative headlines and has revealed it paid communications consultants to provide alternatives to the word "error".

Ofqual describes one of its key roles as "encouraging debate about important topics, such as standards of exams and qualifications".

But in a speech yesterday, Dennis Opposs, Ofqual head of standards, said the regulator had been reluctant to openly discuss the reliability of exam results.

"Why have we previously shied away from communicating much about reliability in public?" he asked an assessment conference in Australia. "Well it is a complex idea that is hard to explain. We have worried in the past that negative news stories about it could damage public confidence.

"On the other hand shouldn't assessment organisations be transparent and communicate with the public about measurement inaccuracy?"

Mr Opposs went on to reveal that Ofqual had hired the "communications messaging consultant" Blue Rubicon to "produce a narrative for Ofqual staff to use when speaking about reliability" and to help it come up with a substitute for the word "error".

"It was necessary to choose an alternative term to `error' as this was too closely associated with culpability, and because it had an unhelpfully subtle word grammar (see box)," Mr Opposs told the International Association for Educational Assessment conference in Brisbane.

Variation was judged to be the best alternative to error, beating rivals such as variance, uncertainty, discrepancy, inconsistency and clash.

"Clash is probably not close enough in meaning to unreliability and also has the potential to provide incendiary headlines," Mr Opposs said.

Reliability of test marking has become a big issue in recent months. Research by Ofqual's predecessor, the Qualifications and Curriculum Authority, published in March, suggested that in 2007 only 55 per cent of pupils taking the key stage 3 English writing test would have been awarded the correct level (grade).

Later that month Isabel Nisbet, Ofqual acting chief executive, was criticised for claiming the public did not believe that that level of "variability" in marking was a scandal because they watched TV programmes such as Strictly Come Dancing, which highlighted variations in experts' judgments.

Sticks and stones

"Researchers reported that (error) had some negative impacts when used with the public. In particular, the common meaning of that word, in contrast to its technical meaning, reinforced an inclination to treat unreliability as necessarily implying culpability.

Further, the word grammar of `error' tends to cause the issue of inherency, agency and culpability to be further muddied. For example, to speak of `an error' seems to imply a single event, for which some person or thing must be responsible. In contrast, the slightly less common in public parlance, more `technical' use of the word `error' lessens the necessary connection with culpability.

This degree of syntactical subtlety and potential for ambiguity suggests that this is not an ideal word to use centrally in an important public communication campaign."

Speech by Dennis Opposs, Ofqual head of standards, to International Association for Educational Assessment conference in Brisbane.

Subscribe to get access to the content on this page.

If you are already a Tes/ Tes Scotland subscriber please log in with your username or email address to get full access to our back issues, CPD library and membership plus page.

Not a subscriber? Find out more about our subscription offers.
Subscribe now
Existing subscriber?
Enter subscription number

Comments

The guide by your side – ensuring you are always up to date with the latest in education.

Get Tes magazine online and delivered to your door. Stay up to date with the latest research, teacher innovation and insight, plus classroom tips and techniques with a Tes magazine subscription.
With a Tes magazine subscription you get exclusive access to our CPD library. Including our New Teachers’ special for NQTS, Ed Tech, How to Get a Job, Trip Planner, Ed Biz Special and all Tes back issues.

Subscribe now