Pupils who are placed in the wrong set could underachieve in GCSEs, according to a report.
The study, from London university's institute of education, found that children of similar ability can attain a grade higher in some GCSEsubjects if they are placed in a high rather than a low set.
But those in lower sets may fail to improve when their results are compared over time because teachers have lower expectations of them.
However, ability grouping had little effect on student attainment overall, particularly in English and science.
Setting has been advocated by Prime Minister Tony Blair, who in 2000 called for more ability grouping and personalised learning for pupils.
The study, which followed 6,000 pupils from 1998 when they were in Year 9, to 2000 when they took their GCSEs, was presented at the annual conference of the American Educational Research Association, in San Diego, last week, Judith Ireson, of the institute's school of psychology and human development.
All of the 45 secondaries taking part in the study were considered by the Office for Standards in Education to offer a satisfactory or good education, and they were balanced in terms of size and the social mix of their intake.
Although researchers found that setting had little effect on attainment overall, it had a big impact on children of average ability.
In maths, middle-ability pupils gained as much as one and a half grades by the time they were 16, if they were in a top rather than a low set. This group was most affected because it tended to be widely dispersed with pupils in different sets for different subjects, the study found.
Researchers found that intermediate students tended to be judged by factors other than achievement when they were allocated to sets.
The study said that behaviour, as well as curriculum and organisational constraints, influenced how pupils were grouped and their mobility between sets.
Teachers also tended to have higher expectations the higher the set their pupils were in, which in turn affected children's motivation.
Differences in achievement were most pronounced in maths. The paper echoes criticisms of the three-level structure of GCSE exam papers made in the Smith inquiry into maths published earlier this year: "In many schools, students in a set are prepared for a particular paper and only students in top sets are able to achieve the highest grades. This places an artificial ceiling on the more able students in intermediate and low sets. Students placed in low sets who are entered for the foundation paper may lose motivation when they realise that they are unable to achieve a grade C."
In Britain, setting varies across curriculum subjects. Ofsted sas that about half of secondary schools set pupils in maths, science and languages.
It is less common in the humanities.
The study recommended that schools should monitor the composition of sets from one year to the next to ensure flexibility, and warned that pupils should not be moved down just because of their behaviour. "Teachers should be encouraged to think of the heterogeneity of students in their classes, even when they are ostensibly 'ability' groups," the study said.
For a full report of the AERA conference see next week's TES