TEACHERS themselves were often the cause of failed exam appeals in the first full year of Higher Still. They submitted inadequate evidence of candidate ability, misjudged standards and the value of prelims and national assessment bank test materials, and sometimes leniently marked internal work, raising false expectations.

But the independent review into last year’s appeals process set up by Jack McConnell, Education Minister, has fully vindicated the rigour of teachers’ work on behalf of the Scottish Qualifications Authority. Marking and appeals were fair. “The outcome was reassuring for the examination system as a whole,” the review’s report, published last week, concludes.

More than 99 per cent of original appeals were unchanged following the review. Independent panels dealt with more than 4,200 requests for a review of failed appeals, just under 9 per cent of the 47,136 unsuccessful appeals from Diet 2000. Many came from a small number of schools and colleges or authorities.

The total number of appeals from the original exams rose to 13 per cent of entries, nearly double the previous year under the old exam system, prompting the inquiry. Only 317 of the 4,000-plus appeals to the appeals were upgraded.

The report tellingly states: “Almost all review team members, across subjects, commented on the quality of evidence submitted to support requests for review from some centres. Typically they noted that in many cases evidence did not cover the whole course; that prelims and other work were over-generously marked, or sometimes not marked at all; that evidence appeared not to have been produced under controlled conditions (ie might not be unaided work); that cut-off scores for prelims and other class tests were set too low; and that estimates were too high on the basis of evidence submitted.”

The report also notes that prelims were frequently criticised for lacking sufficient challenge to demonstrate achievement at A or B grades or for failing to cover the course. “Many participants noted that centres’ understanding of how NABs could contribute to evidence for course assessment was very variable,” it states.

But in teachers’ defence, the review teams, comprised of experienced markers in each subject, say there was simply not the time or opportunity to read and digest the volume of material issued by the SQA and the Higher Still Development Unit over the past two years.

“SQA co-ordinators in particular felt inundated with a constant stream of material and found it hard to keep others in their centres up to date,” the report admits.

“Far from feeling that there was too little information, they felt that the material was too detailed and difficult to digest, insufficiently focused on the essentials and issued piecemeal, making it hard to identify the most recent advice.

“Certainly, many of the markers involved in the review felt themselves that their knowledge of the latest arrangements, NABs and procedures for their subject was only partial and they had to rely on others to pass information on.”

Schools, however, were also guilty of data error, leading the report’s authors to conclude pointedly: “Centres might benefit from advice about how to set up simple quality control procedures for checking data.”

There were often mistakes in candidate numbers, the appeal grade could be wrongly recorded and bands could be submitted instead of grades.

“In a few cases reviews were requested for the wrong subject, or a centre’s contact details were wrong. Many centres did not identify whether a Higher course was a National Qualification or SCE Higher,” the report adds.

Leader, page 10 “Independent Review of Appeals 2000” and a summary are available on the Scottish Executive website. www.scotland.gov.uk

CRUNCH ISSUES

THE evidence suggests lack of confidence in marking and results was at least partly due to “misunderstanding or ignorance of how the examination system operates”.

The review teams suggest centres, students and parents would benefit from more information from the SQA about fairness to candidates and who is involved in the exam system in which roles. Suggestions include:

* How pass and grade threshold marks are set.

* How concordancy is calculated and used.

* How markers’ work is conducted and moderated.

It is clear, the report states, that not all teachers and lecturers had “a good understanding of standards required to achieve particular grades in each subject”.

Teams therefore recommend the SQA gives more feedback to centres about their results. Information might include:

* Pass and threshold scores used by markers.

* Marking schemes agreed at markers’ meetings.

* Examples of a few marked and annotated scripts at each grade, showing the relationship to course outcomes.

The review teams also believe the new system of band scores “represented a level of discrimination that was difficult for centres to estimate” and should be abandoned.

Almost all teachers in the review teams stressed the need for far more professional development on assessment. More teachers had to be given the chance to act as setters, vetters, moderators, markers and members of examining teams to help them understand the standards required.

On the appeals themselves, centres should have clearer advice about decisions to appeal and what makes appeals valid. In each subject, teachers and lecturers need further advice about the quality of evidence needed to support estimates and appeals.

Equally, teams felt centres would benefit from more feedback about why appeals were unsuccessful.

The review teams also believe fellow markers need more support and suggest increasing team working at the time of marking. Other ideas including feedback from examining teams during the marking and a final verdict on the quality of their work after the process.

THE CAREFUL SCHOOL’S CHECKLIST

The review’s report advises:

* Keep copies of all material sent to the SQA.

* Senior managers should make every effort to release teachers for assessment duties.

* All evidence to be accompanied by an appropriate marking scheme.

* Departments need to have quality assurance systems to ensure evidence is valid and supportive of candidates’ abilities.

* More attention to prelim exams. Teachers need to work out detailed and rigorous marking schemes, agree how relevance and coherence marks will be awarded, include a range of questions for A and B grades, and ensure coverage of units.