The standard practice of setting pupils according to ability is counter-productive and affects the performance of the most able girls in particular, according to a team of maths educators at King's College London.
Despite claims that girls are outperforming boys in all subjects at GCSE,a gender breakdown of maths results reveals that there are five boys for every four girls in the A and A* bands.
The researchers believe that the unequal distribution of the highest grades is linked to the unhappy and pressured environment of many top-set classrooms.
They interviewed eight of the most able girl mathematicians in four secondaries and found that they all wanted to move down from the top set. "All the time you are rushing through and not understanding," one girl said.
Class observations confirmed that the 14-year-olds' complaints were valid. "In a range of top-set classes the teachers raced through examples on the board, speaking quickly, often interjecting their speech with phrases such as 'come on, we haven't got much time'," the researchers say.
The study, which includes two secondaries where GCSE classes are taught in mixed-ability groups, builds on earlier work by one of the team, Dr Jo Boaler.
She believes that able girls are badly affected by the top-set experience because they are especially keen to understand what they are doing.
The new study, however, confirms that the most able boys are almost as unhappy as the girls and that, contrary to common belief, top-set pupils enjoy maths less than children in lower sets or mixed-ability classes.
But the researchers believe that setting - a practice adopted by 96 per cent of maths departments - can also have pernicious effects on pupils in lower-ability groups.
Frequent change of teachers is one of them. "If they have a teacher who knows nothing about maths, say a PE teacher, they will give them to us," one pupil said.
Pupils in low sets said that teachers ignored their pleas for more difficult work.
"Students were given answers to exercises a few minutes after starting them or required to copy work off the board for the majority or all of the lesson, " the researchers report.
Pupils in intermediate sets were less frustrated but, overall, only one in six of the pupils interviewed was comfortable with setting.
The report says that setting encourages teachers to see pupils as identical - in terms of ability, preferred learning style and pace of working.
In mixed-ability groups, on the other hand, teachers often let pupils work at their own pace, say the researchers who will present their findings to the British Educational Research Association conference in Belfast on Sunday.
They do not advocate an immediate end to setting, but say that the Government should promote research into mixed-ability teaching rather than discourage its use.
"The strength of the curriculum polarisation (from setting), and the diminution of the opportunity to learn that we have found, if replicated across the country, could be the single most important cause of the unacceptably low levels of achievement in mathematics in Britain," they warn.
"Students' experiences of ability grouping - disaffection, polarisation and the construction of failure", by Jo Boaler, Stanford University, California, Dylan Wiliam and Margaret Brown, King's College London. Contact: Dr Wiliam: 0171-872-3167.