THE EDUCATION committee found that what developed into “an extremely significant problem in data management” should have been anticipated during the implementation of Higher Still. The problem was compounded by the SQA’s own procedures, which were often changed without warning.

Another factor was the decision which allowed schools to wait until the end of June before submitting results from two of the three Higher Still course units.

This meant two-thirds of the data could not technically be considered missing until after that point.

It was at this stage, the committee considers, that the problems really began to pile up. Software for the new awards processing system (APS) was often arriving on the day it was required. It was not fully tested so electronic data went missing and management information from APS which could have identified the missing data and allowed corrective action to be taken was inadequate.

Early problems with registering exam candidates accurately and the consequent late recruitment of markers made matters worse. “Near panic” had set in by early August, according to the report.

It continued: “The attempts by SQA staff to overcome this deficiency by taking a number of well-intentioned but disparate approaches to resolve missing data problems resulted in a loss of control of data management. A subsequent effect was the burden imposed on centres for repeated requests from various SQA officers for resubmission of data.

“This late loading of work and the processing of the associated data by an already overstretched and exhausted operations staff led to even mor room for error including the non-processing of some external paper marks. This was compounded further in that some candidates’ names appeared on more than one attendance record for the same paper. This particular flaw created spurious missing data which was already available on the parallel attendance records.”

The report commends the “valiant” efforts by staff to track down missing documents, including searches of the shelves in the SQA’s procedures hall. But the basic lack of high quality management information meant nobody had any idea of the true extent of missing information. This led to the decision to issue certificates as planned on August 9 based on “an honest but self evidently incorrect belief that the number of resolved cases was much lower than was subsequently determined.”

The committee recommends a full testing of the APS system before next year’s exams. It notes that, while only three per cent of candidates were affected, “the potential scenario was much worse.” If the late arrival of the APS software had been a few weeks later “there would have been no results processing at all.”

The report says the maxim in all software developments should be : keep it simple. But it adds that despite the management of such projects being regarded as extraordinarily difficult, responsibility was given to David Elliot, the former director of awards, who had no IT background. “In the circumstances, senior management had to accept significant criticism,” the report states.

The committee recommends that the SQA should ensure it has sufficient senior staff who are both qualified and experienced in educational measurement, data management and the administration of public examinations and national certification.

Leader, page 18