The warning that could have prevented chaos

1st August 2008 at 01:00
A TES investigation has found that the Government had evidence dating back to 2004 to suggest something was seriously amiss with the tests

The government was advised nearly four years ago to take a "hands-on" role in managing national curriculum testing, The TES can reveal.

An independent report on the last crisis to face the Sats regime - in 2004 - criticised civil servants for not having more "ownership and accountability" regarding the testing process.

The findings will increase the pressure on Ed Balls, who has maintained that the Qualifications and Curriculum Authority (QCA) should bear principal responsibility for the fiasco.

The Schools Secretary has insisted that his department has no more than an arm's length relationship with the marking process. But critics, led by the Conservatives, say Mr Balls should apologise and take his share of the blame for the mess.

The 2004 report, which followed an inquiry led by Mike Beasley, a QCA board member, was commissioned after similar problems hindered that year's key stage 3 English tests. Its conclusions echo much of this year's debacle.

The report also made other prescient recommendations, including changes to the way marking quality is checked, although these changes have never been implemented.

As schools broke up for the summer last week, tens of thousands of results were still outstanding. Furious heads were left asking who would accept accountability for the shambles, much of which has been blamed on the American firm ETS Europe, which was awarded the contract for overseeing the marking of the national tests.

Similarly, 2004 saw some schools receiving their KS3 English test papers late, as well as tardy delivery of results to others, while some schools had papers returned that had not been checked properly.

The Beasley report recommended that "key partner" organisations that bore responsibility for the tests, including the Department for Education and Skills (DfES), as it then was, should establish a team of officials who would "accept corporate responsibility" for test delivery.

It added: "The review team also heard evidence which suggested that the DfES might usefully have adopted a more `hands-on' role, acting as part of the team helping to deliver the 2004 KS3 English test, and taking their share of ownership and accountability for it."

It is understood that this never happened. Instead, said a former QCA employee, teams of consultants were sent into the National Assessment Agency (NAA) to try to rectify the problems of 2004.

"At times, it seemed that there were more consultants there than there were employees from the NAA," the employee said.

The Beasley report also recommended that "borderline checks" or second marking be carried out on the papers of all pupils whose test results were found, on first marking, to lie either within three marks above or below a national curriculum level threshold.

Until 2007, borderline checks happened only with papers up to three marks below a national curriculum level. This led to claims that they would inflate scores because marks could only improve on second marking, but never fall.

This year, the QCA changed the process, but not in the way suggested by Beasley. Instead, it simply scrapped borderline checks. This has been unpopular in schools - teachers say they now have to do the double- checking themselves.

Finally, the report in 2004 also suggested that schools should submit teacher assessment judgements of their pupils' achievement before results were issued. These would then be compared with actual test results so that any anomalies could be investigated before the scores were sent to schools. This change never took effect.

But the Government did have a role in the procurement of ETS Europe to run this year's marking. It was one of at least nine organisations that appear to have had some involvement in the events leading up to this summer's marking fiasco.

Six bodies, including the former DfES, the QCA, the NAA and the consultants PricewaterhouseCoopers, helped select ETS Europe for a pound;156 million contract to mark the tests from this year until 2012. The procurement process went through two reviews by the Office of Government Commerce, an independent office of the Treasury, according to QCA board minutes from 2006.

Linklaters, a law firm, offered legal advice on the procurement. ETS then subcontracted part of its work administering marking to the postal service UPS, for script delivery, and to the call centre company Interaction, which ran a telephone helpdesk for schools and markers from two centres in the Republic of Ireland.

Professor Alan Smithers, of Buckingham University, said: "All these different bodies, rather than making things easier, seemed to get in each other's way, with responsibilities shared around. So no one seems to have had ownership of the process of ensuring that the papers got to schools on time and were marked accurately."

The OCR exam board's letter to the NAA (see panel below) also argues that having different bodies responsible for writing the tests and then setting pass marks blurs responsibility.

A Department for Children, Schools and Families spokeswoman said: "The delivery of tests is an independent function of the QCA to ensure there is no possibility of inappropriate departmental influence over the validity or reliability of the tests or their results. The department's approach is entirely consistent with a general approach to policy delivery, whereby the department takes a strategic role in setting clear remits and success measures and monitoring delivery against them, rather than getting more directly involved in hands-on operational delivery."


There are striking similarities between the 2004 report into that year's key stage 3 Sats - its recommendations were never implemented - and the problems that have emerged with this year's national tests


The report on key stage 3 English said: "The test operations process was plagued with myriad issues and errors. While each issue and error would have been manageable, the combination of so many caused the failure."

The problems it identified included:

- Shortage of markers.

- Late recruitment of markers.

- "Inadequate" systems for tracking marking.

- Late decision by Pearson, then the contractor, to change the website through which schools received results.

- Flawed software for checking "borderline" scripts.


Ken Boston, chief executive of the Qualifications and Curriculum Authority, said last month: "I apologise for the failure of our contractor to have key stage test results available on July 8, and I share their frustration at this quite unacceptable outcome."

Problems identified this year include:

- Marker recruitment and retention.

- Markers being given wrong. information about training.

- Delays getting papers to markers.

- Unmarked scripts being returned to schools.

- Inadequate call centre capacity.

- Slow and unpredictable data feeds.


Anthony Staneff says he is one of several staff to have warned Ed Balls about this year's Sats as long as three months ago.

Mr Staneff, an advanced skills maths teacher at St Aidan's High in Harrogate, wrote to the Schools Secretary on April 26 to inform him of the "shambles" building at ETS Europe. "I get the feeling that if some issues are not addressed, then this year's national test marking process will be flawed and could potentially be another problem for yourself and the Government," he wrote.

A week before the tests, he wrote, "no markers have a clue about what schools they are marking, where they will be marking, some don't even have a clue about whether they will be marking at all". ETS, he wrote, did not respond to calls or complaints, many markers were resigning and the number of scripts allocated to those that remained was rising as a result.

Mr Staneff himself resigned as a key stage 3 maths marker in mid-May. He said he received no response from Mr Balls' department for another six weeks. He eventually met the Schools Secretary on June 6.

This is not the first time questions have emerged over when Mr Balls was first told that the Sats were going wrong. Last week he told a newspaper that he was first made aware that marking was "not on track" by the Qualifications and Curriculum Authority (QCA) at the start of July. He admitted he was told about the state of the marking by Ken Boston, the authority's chief executive, on June 2.

"I feel frustrated," said Mr Staneff. "I think that the man in charge should have been taking a closer look at what's been going on."

A spokeswoman for the Department for Children Schools and Families said that ministers had sought assurances from the QCA as soon as they were made aware of problems. They met many times in May and June. Only on July 1 did the QCA tell ministers that results would be late.

Meanwhile, it has emerged that one of England's biggest exam boards ended its 14-year involvement in the development of Sats in 2006 because it believed the tests lacked credibility with many teachers.

OCR pulled out of bidding to write the English and science KS3 tests, saying they "threatened the reputation of any organisation involved in their delivery".

Greg Watson, the chief executive, wrote to the National Assessment Agency warning that "the strong political drivers behind the testing regime lead many teachers and members of the public to question their value".

Warwick Mansell and Helen Ward

Photograph: Lorne CampbellGuzelian.

Log-in as an existing print or digital subscriber

Forgotten your subscriber ID?


To access this content and the full TES archive, subscribe now.

View subscriber offers


Get TES online and delivered to your door – for less than the price of a coffee

Save 33% off the cover price with this great subscription offer. Every copy delivered to your door by first-class post, plus full access to TES online and the TES app for just £1.90 per week.
Subscribers also enjoy a range of fantastic offers and benefits worth over £270:

  • Discounts off TES Institute courses
  • Access over 200,000 articles in the TES online archive
  • Free Tastecard membership worth £79.99
  • Discounts with Zipcar,, Virgin Wines and other partners
Order your low-cost subscription today