INTERNATIONAL pupil achievement surveys that rank countries as if they are football teams make fascinating reading. Which nation is the Manchester United of maths? And which has slumped into the relegation zone?
But a new study by an Irish researcher reminds us that such league tables should be taken with several pinches of salt.
Michael O’Leary, of St Patrick’s College, Dublin, began his investigation after noticing discrepancies in two international surveys of 13-year-olds’ science achievement. The 20-country Second International Assessment of Educational Progress (IAEP2), conducted in 1991, indicated that Ireland was near the bottom of the league. But the 40-country Third International Maths and Science Study (TIMSS), just four years later, gave the Irish a mid-table rating.
In 1991, only the bottom 10 per cent of Swiss children had performed as poorly as the average Irish 13-year-old. By 1995, Ireland’s science scores were at least as good as Switzerland’s. How could this be?
O’Leary found that most of the Irish pupils involved in the IAEP2 test were in grade 7 whereas almost three-quarters of the Swiss children were in grade 8 and had therefore had an extra year of secondary schooling.
He also points to another complicatig factor highlighted by Professor Margaret Brown, of King’s College, London; the number of children who repeat grades in some countries (Research Focus, February 4).
“In Spain, up to 25 per cent of secondary students are repeaters,” he points out. “An even higher percentage of repeaters are found in Portugal.”
O’Leary acknowledges, however, that age and grade factors do not explain all the inconsistencies. The format of questions can also have a bearing. Some test surveys use only multiple-choice questions while others include short-answer and extended-response items. The degree of overlap between test topics and the national curriculum, and the quality of data-collection, can also affect countries’ rankings.
“The motivation of students participating in international tests to do well may also be relevant,” he adds. As there have been reports of Taiwanese pupils chanting nationalistic slogans as they marched into their TIMSS tests, that may be one of the most important factors of all.
“The effects of age-based and grade-based sampling on the relative standing of countries in international
comparative studies of student achievement”, by Michael O’Leary,
St Patrick’s College, Dublin. Contact: MICHAEL.OLEARY@SPD.IE
