Great care has to be taken with international comparisons of pupils’ performance because so much of the data is unreliable, according to a leading maths specialist. Political capital was being made out of leaking and recycling invalid data, Professor Margaret Brown, of King’s College, London, told the BERA conference.
Some comparisons, such as the relationship between spending and performance, were being ignored. Even so the international comparisons produced by the First and Second International Mathematics Studies were useful if they were not used in a simplistic way.
Only a very limited number of international comparisons of maths performance are valid, she said. There are serious difficulties in comparing like with like in terms of the sample of the children studied and the work they were doing in schools, Margaret Brown argued.
Even in the most reliable studies a “year group” of 13-year-olds was liable to be interpreted differently in different countries. In the UK it was likely to include most children aged 13.1 to 14.1, but in the Netherlands 17 per cent of children, and in France 12 per cent, were routinely excluded from comparisons because they were in special or vocational schools.
In countries such as Germany and Switzerland results were distorted because the least successful children in a year group were held back a year to repeat the work. This meant that a sample of 13-year-olds would include few low-ability 13-year-olds, but a variable proportion of 14 and even 15-year-olds on their second attempt at the syllabus.
There were also real difficulties, she said, in devising tests which accurately reflected the work which had received most attention by a year group during any given year. Maths had been regarded as a relatively easy subject to compare because of its reliance on internationally-recognised symbols and procedures, but in fact teachers had very different objectives in different countries as the 1991 International Assessment of Educational Progress survey had shown (see table). In Canada, for instance, a high priority was given to problem-solving and maths for everyday life, while in France there was far more emphasis on classical mathematical proof and inquiry.
Within the basic curriculum the emphasis given to different topics varied widely and so did the amount of time devoted to the subject. In Japan, which always performs well in international comparisons, children were being taught for far longer than in most other countries. These differences tend to be reflected in comparative results, Margaret Brown said.
More interesting, she added, was the variation within each country’s results. These show that there is actually very little difference between the mean performance of the major countries in the Organisation for Economic Co-operation and Development. Most countries, even on the more successful “Pacific Rim”, have a “tail” of around 10 per cent performing at the same low level.
