How are exam questions created?
You have a crate of cola that costs £10. The crate contains 12 cans. What is the cost of seven crates? The question may prompt several responses. First: that’s quite expensive for unbranded cola. Second: since when is cola packaged in crates? And third: why do we care how many cans there are in a crate? Time for a Diet Coke (or Pepsi Max/own brand equivalent).
Exam questions are often set in real-world contexts but the way they’re is framed is of vital importance. Exams should measure a particular understanding or skill, and there is a risk that the thought processes provoked by the wording of a question can interfere with a candidate’s ability to answer it.
Experts call this “question fidelity”: questions should be written in a way that prompts certain processes in students’ minds. When examiners write questions based “in reality”, they do so to make abstract ideas more concrete.
There are several reasons for writing papers in this way: as well as making questions more relevant, context adds variety, which is important because many students will have used the same books and case studies.
Contrary to popular belief, context is never used to make questions more interesting – although sometimes this happens inadvertently. When a GCSE maths question recently stumped some candidates, they parodied the featured characters and settings on social media (“I was going to get another pencil during the exam but too bad that John gave them all away”).
Writing questions set in real-life contexts is harder than you might think. The word count, tone and setting must all be carefully considered. Questions that are too long may affect candidates’ understanding of the task, while a too-familiar context or case study will test their ability to recall facts rather than apply knowledge. Extraneous information should be kept to a minimum, unless relevant facts have to be identified.
Questions should also be logical: the aforementioned cola conundrum, which is adapted from a sample question that was used to measure candidates’ responses to context, does not meet this criterion: most readers would expect in real life to work out how much one can costs, not seven crates.
Contexts should also be sensible. Consider the following question from a 1984 maths textbook: Alan drank five-eighths of his pint of beer – what fraction was left?
Beer is not measured in eighths of pints. Even if it were, it’s unlikely that Alan would be thinking this precisely while he was drinking it – unless it was his round.
This brings us to another important point: contexts should keep the target audience in mind. Questions that relate to alcohol consumption are therefore not appropriate.
Critics joke that if you want to set questions in contexts that teenagers understand then they should refer to cigarettes, smartphones and YouTube hits. These cultural stereotypes are not helpful and do young people a disservice.
In fact, a question involving, say, alcohol, might alienate some audiences or might simply not be accessible to them and therefore raise concerns about fidelity. Clearly, a balanced approach is appropriate.
Research also suggests that adult concepts such as interpreting wage slips or buying carpets actually hinder teenagers’ comprehension of the task. (Who can blame them? The idea of having to do something as boring as buying floor coverings would probably also distract many adults.)
Culturally specific activities such as purchasing “pick your own” fruit, sailing or cooking a particular dish are also no-go areas for question writers. Cultural references, too, should be avoided. Several years ago a paper cited a case study that referred to the newsagent WHSmith. This was difficult for students in Northern Ireland because the chain doesn’t have branches there.
Similarly, certain topics should be avoided to ensure equal opportunity for all candidates. For example, questions that refer to music might alienate a deaf candidate.
Writers are also encouraged to avoid particularly novel ideas – for example, how temperature influences the sex of alligator hatchlings (yes, really; this setting was used in research trials that assessed the importance of context).
Perhaps unsurprisingly, writing contextual questions that meet all these criteria is a time-hungry process. At AQA, work begins on a paper more than a year and a half before students are due to sit it. A committee puts together the exam, based on the findings of our Centre for Education Research and Practice which considers, among many other things, the importance of context in questions.
So the next time your students are asked to contemplate the price of bread or the probability of picking a pink snooker ball from a bag, you can be sure it’s for a good reason.
Alex Scharaschkin is director of AQA’s Centre for Education Research and Practice