Disaster waiting to happen . . .
The chief executive's written report to the education committee inquiry
Ron Tuck, chief executive of the SQA, who resigned on August 12
Data management and missing data were the causes of the problems, Mr Tuck has repeatedly insisted. "By early June, we became aware that there was a large volume of data outstanding. In the case of internal course assessments, we believed initially that this was because operations staff had not made the normal reminder calls to centres (due to the increasing pressure they were under). We therefore began to contact centres.
"We then started to become aware that many centres had already submitted data to us which for some reason was not being processed and therefore appeared to be 'missing'.
"Over the next few weeks, we investigated a number of possible reasons for data going missing, each of which yielded only partial explanations. These were:
* failure of electronic transfer of data between centres and SQA. Some examples of this were found, for example, some centres using old Phoenix software, one or two cases of faulty discs, and a failed SEEMIS transfer on May 3;
* the existence of duplicate entries creating a false impression of missing results. This factor accounted for a certain number of apparently missing results;
* submission of data out of sequence. Lateness of entries combined with non-user-friendly error reports meant that results were sometimes received for entries which had not been processed;
* paper reports being filed without being processed. Searches were ordered on three occasions but yielded only a relatively small amount of missing data. I now understand that these searches may not have been carried out thoroughly.
* errors by data punch bureaux going undetected by SQA staff. This potential cause came to light relatively late and I am unable to estimate its significance."
Mr Tuck says it became clear from April onwards there were difficulties in recruiting enough markers. Another 1,000 on top of the normal 6,000 were needed, recruited from schools, colleges and universities. They had to have had three years teaching courses, unless courses were new.
(In evidence to the lifelong learning committee, David Miller, SQA chairman, confirmed 88 per cent of markers had marked previously. Invitations to mark went out over a month late because of software problems.) Mr Tuck states 90,000 scripts (out of a total of more than three million) were caught up in late marking. "The marking process itself, including the usual quality checks, was carried out as normal."
He concludes: "The lateness of marking added to the difficulties of data management, in that it meant that the normal strict sequence of procedures could not always be followed."
Mr Tuck notes it would cost an extra pound;3 million to bring markers' fees up to more acceptable levels after "informal feedback suggested that there was a growing reluctance to take on the duties".
THE DIRECTOR OF AWARDS' WRITTEN REPORT TO THE EDUCATION COMMITTEE INQUIRY
David Elliot, director of awards at the SQA, who resigned at the end of August
In my view the problems with the accuracy and reporting of examination results in 2000 arose from the following causes: l the timescale from the date of SQA's creation in April 1997 to the start of Higher Still courses in June 2000 was very demanding. Full implementation in year one without phasing was demanding, although it seemed achievable;
* creating the new organisational structure and culture absorbed substantial amounts of management time in the first two years;
* new administrative processes had to be developed. In some cases, this was preceded by extensive consultation with stakeholders on policy. Developing policies and procedures which met the needs of both secondary schools and further education was demanding and time-consuming (but necessary);
* the reduction from four directors to three in April 1999 was perhaps premature."
More generally, Mr Elliot says new software had to be specified and written for business processes not yet in place. The new exams were more difficult to administer because of the internally assessed units and the exams started and ended later in the year. This cut processing time and put more pressure on the marking. "In mid-June there were 65,000 scripts unallocated, far less marked. This caused major problems."
Mr Elliot believes there was no problem with the quality of the marking - only the timescale by which it was completed.
The SQA's operations unit - the key department - was less prepared for Higher Still than other parts of the organisation in terms of management, staff briefing, structure and staffing levels.
Mr Elliot says while core software was delivered in time, data management tools were sometimes cumbersome and time-consuming. Late marking meant that operations staff were inputting marks data when they should have been free to concentrate on other tasks such as "query resolution".
"Under pressure arising from the new qualifications and new software, not all staff consistently maintained good practice in data processing. As data problems were arising from a range of factors, this was not immediately identified."
Mr Elliot blames the lack of preparedness of the operations unit on Jack Greig, its head, who has been granted early retirement but is facing disciplinary action.
It was apparent by late spring that there were insufficient experienced staff to cope with the problems. He suggests the major difficulties with data management and missing data were due to: l problems with electronic data transmission from centres;
* new, less user-friendly paper forms replacing well-established and familiar forms;
* too few staff to chase up centres which didn't submit data on time;
* duplicate unit entries caused when centres were asked to re-send data and duplicates created by errors in the new software;
* lapses in control of paper-based data;
* insufficient checks on the work of the data preparation bureaux.
THE SCOTTISH EXECUTIVE'S LETTER
The extent to which the Scottish Executive could see disaster looming was graphically illustrated in a remarkable letter to Ron Tuck on July 17 from Eleanor Emberson, head of the curriculum, international and information technology division. It revealed a hitherto unsuspected level of intervention by government officials in the SQA's operations.
In her letter of July 17 she noted that marking problems had been resolved "which must be a great relief to all concerned". Mrs Emberson also told Mr Tuck that "you are taking steps to ensure a daily spreadsheet" to chase up "missingdubious internal assessments", and she looked forward to seeing a copy "very soon".
There was also a welcome for the relatively successful experience up to that date of running the new awards processing system, the software for handling exam data leading to issuing of certificates.
But Mrs Emberson expressed concern at "possible overload of the system", and suggested the names of two expert companies that "might be good sources of advice on streamlining your ad hoc queries for management information to reduce the demands on the processor".
Mrs Emberson went on to outline deadlines for processing the assessment data and stressed that "in making decisions about these dates, the priority will be the accuracy and timeliness of the Higher results for the most candidates."
The letter discussed measures for checking data. It emerges that thought was being given in mid-July to delaying the publication of results. But Mrs Emberson said this should be done only if data problems affected "a significant number of candidates, with significance defined in terms of subject and level groupings, not just absolute numbers, and there would have to be some good possibility of solving them all or nearly all within a very few days".
It was agreed that Neil MacGowan, now acting head of the SQA's operations unit, would draft two different public statements, one to announce the achievement of the August 10 deadline for issuing exam certificates, the other to announce a delay. Mrs Emberson said she assumed this would cover missing data not just for the new Higher Still but for Standard and Higher qualifications too.
Her letter continued: "We do not want the next problem to be an appeals backlog. We agreed that it would be helpful to beef up the letter to centres to warn them that they should be prepared for telephone calls about missing results from August 10, while emphasising the support to be provided by SQA, and to give specific warnings and details to centres with affected candidates.
"Perhaps the press ad on delays should avoid the phrase 'sorry for any inconvenience', and should be in the tabloids as well as the broadsheets, but these matters are for you."
Mrs Emberson's next concern was that the telephone helplines, with separate numbers for candidates, schools and FE colleges, would be checked out to guard against the lines going down if there were a large number of calls. "I assume that, if there is a delay, SQA will have cover to deal with substantial numbers of enquiries from students who expected their results on 10 August, but missed the delay announcement," she commented.
Mrs Emberson concluded in a thinly-veiled instruction that "it would be helpful" to have the spreadsheet summarising the position on internal assessment data problems as soon as possible before the next meeting between the Executive and SQA on July 21.
Next week: more details emerge