As marked papers arrive in schools, Tim Cornford explains how the SATs are formed
y now, all primary schools in England will have completed their key stage 1 marking and by early July will have received back the marked scripts of their pupils' key stage 2 tests. It is only a matter of weeks since these children sat down to take the tests, but it is two years since the seeds of these tests were first sown.
All the national curriculum tests follow a similar path of development. It begins with an extended brainstorming session and ends with the distribution of carefully formatted test papers and mark schemes (hopefully without a single typographical error!) to schools. During the journey the questions are tried out by thousands of children and commented on by hundreds of teachers before the Qualifications and Curriculum Authority and its test development agencies are satisfied that we have a high quality set of papers.
The first phase for maths and science tests is to generate a list of ideas for questions. This involves considering different ways of posing familiar problems; interesting contexts for number operations or data-handling; engaging ways of illustrating questions about the food-chain; ensuring coverage of the programme of study. In English, the search is for reading material - whether it be fiction, non-fiction or poetry - which will engage the interest of 600,000 boys and girls at each key stage and be rich enough to stimulate questions about language, meaning, characters, events and purpose.
And as if this wasn't enough, there needs to be a variety of types of question, from multiple-choice to short response, multiple-stage and extended writing. These will test the breadth of children's skills and understanding as well as reflecting different pedagogical styles.
Trialling questions is vital. Test design may be in part a technical process, but it is also about ensuring that children can demonstrate what they know. Reality checks in the classroom are crucial. For the 1999 key stage 2 maths test we wanted to focus some questions on using a calendar. As teachers know only too well, it's important to get the language of mathematical problems right. In the first pre-test, children found the scene-setting confusing:
"Raymond's birthday was on August 20th. He had a party on the Saturday closest to his birthday."
The past tense and the word "closest" threw them. The wording had to be changed. The most difficult question, givn an August calendar, asked children to identify the date of one of the Saturdays in July. Through changes of wording and two separate pre-tests, this proved unfairly tricky and had to be dropped.
Walls featured prominently in this year's KS2 reading test. In the section "Great Walls of the World" we wanted to have a question about imagery. Typically these are quite difficult for 11-year-olds, but the presence of descriptive language such as "molehill" and "armour-plated centipede" was a good basis for a question. Originally one question referred to both these phrases: "Explain what these phrases tell you about the walls they describe".
This confused too many children, who thought that only one answer was required. The solution here was to split the question clearly into two, one being about the intention of the writer, the other about the effect on the reader.
Without seeing first hand how children react to these, and hundreds more trial questions, our tests would be much less reliable than they are and the mark schemes would be far less comprehensive.
The input of teachers is equally important. For each subject at each key stage there is a panel of teachers and a test review group. These meet at critical points in the development process to examine the test questions in minute detail. They comment on the use of language, the difficulty of questions and how children will react to them, the style of illustration, the appropriateness of contexts, and much more. Our test development agencies also have panels of teachers who meet to rate the level of difficulty of each year's test as it approaches its final form. This is an important piece of evidence when it comes to setting the level thresholds.
Professional input doesn't end there. Groups of practitioners advise us on special needs issues; help us trial materials with pupils for whom English is a second language; and help us modify the tests for the visually and hearing impaired. When the test weeks are all over, QCA picks up more feedback. We send questionnaires to a sample of schools about the quality of the tests; we ask the markers for their views both about the tests and the quality of preparation they were given; and we invite some teachers to discuss the tests with us.
So while you are reading this, we at QCA are checking teachers' reactions to this year's tests to see what improvements we can make in 2002.
Tim Cornford is head of the QCA'sassessment division