The amazing process behind an exam paper

As you and your students await GCSE results, have you ever thought about how those exams came into being and why certain decisions in the process have been made? Kaye Wiggins finds that the life of an exam paper is full of surprising – and important – moments for teachers
19th August 2016, 1:00am
Magazine Article Image

Share

The amazing process behind an exam paper

https://www.tes.com/magazine/archived/amazing-process-behind-exam-paper

In a warehouse on an industrial estate in Rotherham, South Yorkshire, conveyor belts are rolling. But rather than the glass containers the town has been renowned for producing for the past 250 years, in this building it is freshly completed GCSE exam papers being transported through an assortment of machinery.

This is a processing plant for Pearson, the exam board that runs the Edexcel exams. If your students completed an exam produced by Edexcel a few months back, this is where it will have ended up.

Few markers in the modern GCSE exams process now sit before wobbling piles of paper and leave answers ringed with coffee-cup stains. The majority of exam papers arrive in places like this warehouse in Rotherham; barcodes are read and pages are scanned, then the photographic reproductions are distributed electronically to the markers. The answers are marked online, too. It’s a surprisingly mechanical and sterile end for a process that begins with such human emotions of hope, stress and fear.

This change is a relatively unknown facet of England’s vast and complicated GCSE exams regime (or at least it is unknown to those teachers who are not also GCSE exam markers).

And there are many more quirks and surprises in the journey of an exam paper about which the majority of teachers are unaware.

Although more than 5 million GCSE exam papers are taken each year, in at least 48 subjects, the process of how the papers come into being, what happens to them when they are sent back to the board and why pupils gain the results that they do remains a mystery to many in the teaching profession.

So here, as pupils wait to receive this year’s results, we present to you the life cycle of a GCSE exam paper.

Birth: the creation of an exam paper

An exam paper begins its journey of life approximately two years before it is due to be taken. The gestation period can begin, perhaps surprisingly, with just one parent mulling over the course content, rather than the orgy of debate between multiple experts one might expect.

This single parent is called the “principal examiner”, usually a teacher or recently retired teacher who is an expert in their subject and who has extensive marking and examining experience.

In some cases, the principal examiner may write the whole first draft of the question paper and corresponding mark scheme alone (though multiple experts each submitting questions into a pooled first draft is also common).

But the first draft is not wholly down to the parents; there is a basic genetic framework for the exam that has to be adhered to. The board can’t change the duration of the exam, for example, which is fixed when boards first receive accreditation from the exams watchdog, Ofqual, to run the qualification.

But they can make small changes - adding in more two-mark questions and fewer three-mark questions, for example. They can also decide the topics to be covered by the exam.

Unsurprisingly, the information that exam boards are prepared to give out about how topics are chosen is patchy.

All that Sharon Hague, senior vice-president at Pearson Qualification Services, will say is that core topics will come up on a regular basis and that principal examiners “will look at what has happened in previous years”.

With the topics decided, the lone writer or group of writers then has to pen some questions. This, like naming a baby, is something of an art form.

Questions have to be set out in a logical order and in simple language. A guidance document from the WJEC exam board recommends rephrasing a question from, “What kind of cleaning agent will remove the hard-water stains left by a dripping tap on a washbasin?” to, “A dripping tap leaves hard-water stains on a washbasin. What kind of cleaning agent will remove them?” The latter “follows a logical narrative flow”, according to the board.

Narrative is important. These exams are not meant to test knowledge in isolation, say the boards

And narrative is important. These exams are not meant to test knowledge in isolation, say the boards. Alex Scharaschkin, director of research and compliance at AQA, says that exams require students to show they can “apply knowledge to situations over and above being able to remember a formula and write it down”.

The way the boards test this ability is by writing the questions as scenarios. This can cause problems if done badly - for instance, by referring to things with which students are unfamiliar.

A good example was an exam paper that included a case study about the newsagent WHSmith, which was difficult for pupils in Northern Ireland where the chain did not have any branches.

Taking all this into account, a first draft is produced - and the seed has been sown. Over 18 months that seed develops into the final paper through multiple stages of approval: a single paper can go through as many as 10 rounds of redrafting and discussion.

The Braxton Hicks contractions (the practice labour contractions) in this scenario involve the “scrutineer”. This subject expert is trained to sit an exam and respond as a student would, checking the questions are understandable and the paper can be completed in the time allowed. This removes the need for the exam paper to be tested on real students.

If a paper has made it this far, we can view it as being head down and ready to go. It is signed off by the principal examiner and a “chief examiner”, whose job is to oversee a whole qualification rather than just one paper. This should happen by the autumn before the exams are taken. The Department for Education and Ofqual will not see the exam paper until the day that pupils take the test.

Reaching maturity: print, distribution and the sitting of an exam paper

In the early stages of any life, risk-taking and steep learning curves are common. Patience from those in charge is a must, as are strict boundaries.

As any parent of a toddler can tell you, a laissez-faire approach tends to land you in embarrassing situations and, sometimes, the queue for the fracture clinic at your local hospital.

Exam boards are very good at setting boundaries for their fledgling papers. The printing of an exam paper, which usually happens between January and April in the year that they are to be taken, is a tightly controlled affair.

Most boards use external secure printing companies, the locations of which have not been made public. WJEC’s Eduqas does its printing in-house. When the question paper is sent to the printers, it is in a digitally encrypted form to minimise the risk of hacking. No phones are allowed in the printing area and any employee who has a close family member taking a given paper is not allowed to see that paper.

Printing complete, it’s time to give the exam paper a little autonomy to see whether, when sent into the big wide world, it can stand on its own two feet.

But not too much autonomy: exam papers have a chaperone, of sorts - they come with a set of conditions that could teach prison wardens and bank managers a thing or two.

Schools become a bit like Fort Knox

“Schools become a bit like Fort Knox,” says Tricia Brennan, director of delivery at the AQA exams board.

When papers arrive - usually around three weeks before the exam - schools must not take them out of their plastic packets. They must keep these plastic packets in a secure, locked storage unit with a specific type of lock (a five-lever mortice, if you’re interested). Preferably, that locked unit should be screwed to the wall or the floor and it must be in a secure room (there are further rules on the definition of a ‘room’; it must have solid walls or reinforced stud walls).

There are reasons, though, for what might seem like excessive security. “A few years ago, someone broke into a secure room using a forklift truck,” says Brennan. “They destroyed the wall of a school, lifted the safe in the forklift and left.”

Schools’ security measures are inspected during spot checks, which are carried out every year by the Joint Council for Qualifications - a body that represents exam boards.

When exam day arrives, the exam paper is suddenly on its own. It is released into the wild and - like the parents of a newly minted adult - the makers of that exam will be judged by its actions.

The plastic packets are only opened 30 minutes before the exam begins. The papers are laid out on the tables. And then, those famous words are uttered: “You can begin.”

Actually, it takes a little longer to get around to uttering those words these days. At Wright Robinson College, an 11-16 school in Gorton, Greater Manchester, once pupils are seated in the exams hall and ready to start the test, all are told to put their hands in the air to show their wrists.

This new ritual has been introduced to check for Apple watches, which could be used to cheat, says Victoria Duffy, exams director at Wright Robinson. “They hand them in with their phones as they go into the exam hall,” she says. “They’re very good at handing them in, but we have to be sure [that they have].”

The harsh judgements of middle age - return of the papers and marking

Once completed, the papers are packaged up and couriered away to be marked. They have peaked. The passion, promise and peril of the equivalent of their twenties is now replaced with the plodding and pernickety process-driven nature of middle age.

Some papers are sent directly to examiners’ homes to be marked on paper and the rest are sent to exam boards’ scanning centres. At Edecxel, about 95 per cent of scripts are marked online; at AQA, it is about 70 per cent but at WJEC, the figure is just 40 per cent.

The numbers involved in the online process are staggering. At the Rotherham plant alone:

  • 4.7 million GCSE and A-level exam papers are processed.
  • Up to five lorry loads of papers arrive each day during peak season and each lorry contains 22 pallets of papers.
  • About 900 staff put papers through 30 scanners.
  • At peak times, the scanners run for 24 hours per day, six days a week, with staff on either eight or 12-hour shifts.

The number of paper cuts per season, meanwhile, is unconfirmed.

The exam board will do everything it can to make sure all the scripts can be marked, and will send them for paper marking if they can’t be scanned. But sometimes there’s nothing it can do. In one case, a pupil had screwed up a page of their exam paper, put it in a pencil case and filled the case with water. Their school made a valiant effort, posting the wet pencil case to the exam board, but to no avail.

Those papers sent for scanning get a new lease of electronic life - the paper carcass, hollowed out by experience (or by a student with a pencil case of water), is left in storage for nine months before being shredded and recycled. The scanned versions are sent to be marked.

The main difference between on-screen and paper marking is that, with paper marking, markers are not reading a pupil’s whole script. They may sign up to mark, say, 100 pupils’ responses to question 4 and another 100 responses to question 7.

Whether they’re marking on paper or on screen, boards have processes to stop unfair or inaccurate marking.

Those marking on paper must send batches of marked scripts to more senior assessors for checking. Those marking online are monitored through “seeded questions”, which are genuine pupils’ responses to that year’s paper that have been pulled out in advance by the most senior examiners as textbook answers. Markers will not know when they are marking a seeded question, which can make up about 5 per cent of the total number of questions they mark, but their responses to it will be checked.

Once markers have finished with the scripts, they are sent back to exam boards. At this point, the paper has the number of points scored on it, not a grade.

The grade is worked out through a complicated process called “awarding”, where senior examiners meet in person to discuss where the grade boundaries should lie. What happens in that room can be the key to whether thousands of pupils miss out on a pass grade or not.

Their decision is based on something called “comparable outcomes”. It starts from the premise that pupils at any ability level will - at a national level - make roughly the same amount of progress between the ages of 11 and 16.

Exam boards look at the amount of progress made in previous years between key stage 2 Sats tests and GCSEs. They then use the KS2 scores of the year group that have just taken their exams to predict where their GCSE grade boundaries should lie. The boards’ figures show how many pupils would score each grade if that same amount of progress had been made - and examiners set grade boundaries in a way that ensures these proportions are met.

The system has proven successful in bringing an end to years of so-called “grade inflation”, but critics argue that it prevents real improvements in standards across the country from being reflected in results. Only when the decision on grade boundaries is made is the paper’s job nearly done - and retirement beckons.

Last rites: appeals, results and repercussions

But, of course, the story is by no means complete. Just like retirees drawn back to the workplace as consultants when things go wrong, exam papers are hauled back into action when results day does not go quite as planned.

Those who believe that exams have been marked unfairly or inaccurately can apply for a re-mark, in which case a second examiner will look at their exam script. This person can give them a higher mark or a lower mark, or leave the mark unchanged - although, in practice, grade drops are uncommon.

“We put in for re-marks when students are significantly below their predicted grade and are close to a grade boundary,” says Duffy. “Last year, about one in five of the grades we challenged were given a higher grade on re-mark.”

This fits the national picture: last year, 325,550 GCSE marks were challenged, of which 62,350 - or 19.2 per cent - were changed. On average, it took nine days for boards to respond to challenges.

But this year, the number of challenges could be much lower. Reforms announced by Ofqual, to come into effect this year, will make it more difficult for boards to change pupils’ marks once they are challenged.

Ofqual argues that marks have been changed in the past simply because the re-marker was more generous than the original marker, giving an advantage to those who challenged their grades over those who did not.

But the reform has not gone down well with schools. Russell Hobby, general secretary of the NAHT headteachers’ union, said in May that it would “make things more vague than they already are” and was likely to “have a further adverse effect on confidence” in the exam system.

In better news, the mistakes of one exam are unlikely to be repeated the year after. Before the exam paper is put to rest, an analysis of its time in existence is conducted.

At AQA, the process is called a “question paper functioning report”, says Mr Scharaschkin. “For each question, we look at the distribution of marks,” he says. “If questions are particularly intended to distinguish between, say, the performance at A and A*or F and G, did that actually happen in practice, and if not, why not?”

Lessons are learned - an exam paper has its legacy.

And yet, it is not allowed to gently pass into a shredder to be reborn as pulp for recycled paper. No, exam papers are kept in an awful kind of purgatory. By teachers.

It’s all down to past papers. Teachers store these away in a cupboard to be rolled out as practice papers for ever more. One teacher admits that they still have some from 20 years ago, brought out only on special occasions.

It’s a cruel, cruel end to what is, more often than not, a very useful and productive life indeed.

The afterlife: what does the future have in store for exams?

The move to online marking has been the biggest shift in the way GCSEs are run over the past decade. But what does the next decade hold?

AQA’s Scharaschkin says that in 10 years’ time, all exams will be marked on-screen. But the exams won’t necessarily be taken on a screen.

“People say, will [exams be taken] on tablets by then, with no pen and paper?” he says. “We won’t quite be there in 10 years’ time.”

I’d hope online assessment would be something in 10 years’ time that we’d see in GCSEs

But Hague, from Pearson says that online assessment could play a much bigger role. “I’d hope online assessment would be something in 10 years’ time that we’d see in GCSEs,” she says. “But it’s about where it’s appropriate and where it really adds value for the candidates.”

The advantage of on-screen assessment, she says, is that it is “adaptive” so that, say, a candidate answering questions quickly and correctly would be given harder questions to stretch them.

“That would be a good experience for candidates because you could tailor the assessment to how the individual candidate is performing,” she says. “And it would give you more information about what the candidates have understood and the skills they have.”

So could this mark the end of the traditional GCSE process, with the whole year group sitting in an exam hall for a fixed period of time?

Scharaschkin doubts it. “Hall exams will still stay for quite a while,” he says. “There’s something about it being seen to be a fair process where it’s a level playing field and everyone is seen to be doing the same thing.”

@kayewiggins

You need a Tes subscription to read this article

Subscribe now to read this article and get other subscriber-only content:

  • Unlimited access to all Tes magazine content
  • Exclusive subscriber-only stories
  • Award-winning email newsletters

Already a subscriber? Log in

You need a subscription to read this article

Subscribe now to read this article and get other subscriber-only content, including:

  • Unlimited access to all Tes magazine content
  • Exclusive subscriber-only stories
  • Award-winning email newsletters
Recent
Most read
Most shared