Exams: Ofqual admits to 'shying away' from talking about test result reliability

18th September 2009 at 01:00
Fearful of negative headlines, exams regulator hired PR consultants to find alternative terminology

Original paper headline: Error: not a word in Ofqual's vocabulary

The exams regulator has admitted to "shying away" from talking about test result reliability for fear of negative headlines and has revealed it paid communications consultants to provide alternatives to the word "error".

Ofqual describes one of its key roles as "encouraging debate about important topics, such as standards of exams and qualifications".

But in a speech yesterday, Dennis Opposs, Ofqual head of standards, said the regulator had been reluctant to openly discuss the reliability of exam results.

"Why have we previously shied away from communicating much about reliability in public?" he asked an assessment conference in Australia. "Well it is a complex idea that is hard to explain. We have worried in the past that negative news stories about it could damage public confidence.

"On the other hand shouldn't assessment organisations be transparent and communicate with the public about measurement inaccuracy?"

Mr Opposs went on to reveal that Ofqual had hired the "communications messaging consultant" Blue Rubicon to "produce a narrative for Ofqual staff to use when speaking about reliability" and to help it come up with a substitute for the word "error".

"It was necessary to choose an alternative term to `error' as this was too closely associated with culpability, and because it had an unhelpfully subtle word grammar (see box)," Mr Opposs told the International Association for Educational Assessment conference in Brisbane.

Variation was judged to be the best alternative to error, beating rivals such as variance, uncertainty, discrepancy, inconsistency and clash.

"Clash is probably not close enough in meaning to unreliability and also has the potential to provide incendiary headlines," Mr Opposs said.

Reliability of test marking has become a big issue in recent months. Research by Ofqual's predecessor, the Qualifications and Curriculum Authority, published in March, suggested that in 2007 only 55 per cent of pupils taking the key stage 3 English writing test would have been awarded the correct level (grade).

Later that month Isabel Nisbet, Ofqual acting chief executive, was criticised for claiming the public did not believe that that level of "variability" in marking was a scandal because they watched TV programmes such as Strictly Come Dancing, which highlighted variations in experts' judgments.

Sticks and stones

"Researchers reported that (error) had some negative impacts when used with the public. In particular, the common meaning of that word, in contrast to its technical meaning, reinforced an inclination to treat unreliability as necessarily implying culpability.

Further, the word grammar of `error' tends to cause the issue of inherency, agency and culpability to be further muddied. For example, to speak of `an error' seems to imply a single event, for which some person or thing must be responsible. In contrast, the slightly less common in public parlance, more `technical' use of the word `error' lessens the necessary connection with culpability.

This degree of syntactical subtlety and potential for ambiguity suggests that this is not an ideal word to use centrally in an important public communication campaign."

Speech by Dennis Opposs, Ofqual head of standards, to International Association for Educational Assessment conference in Brisbane.

Log-in as an existing print or digital subscriber

Forgotten your subscriber ID?


To access this content and the full TES archive, subscribe now.

View subscriber offers


Get TES online and delivered to your door – for less than the price of a coffee

Save 33% off the cover price with this great subscription offer. Every copy delivered to your door by first-class post, plus full access to TES online and the TES app for just £1.90 per week.
Subscribers also enjoy a range of fantastic offers and benefits worth over £270:

  • Discounts off TES Institute courses
  • Access over 200,000 articles in the TES online archive
  • Free Tastecard membership worth £79.99
  • Discounts with Zipcar, Buyagift.com, Virgin Wines and other partners
Order your low-cost subscription today