This is the main finding from key research which, unusually, set out to discover pupils' views of setting and mixed-ability teaching.
Pupils from top to bottom sets reported stigmatisation of individuals and even bullying. One said: "Some people go round slagging others saying 'you're in a lower set, you're not as good as me,' and they call other people thick."
And, despite legislation that requires children to be consulted on factors affecting their education, they are often excluded from decisions on class organisation.
Pupils believe that being set in classes according to their ability can place them under undue pressure, which in turn can affect how well they learn, according to a report based on the study by Chris Smith and Margaret Sutherland, of Glasgow University.
Research into pupils' views of setting and mixed-ability groupings shows pupils are acutely aware that formal testing is the most important criterion for selection into sets.
This causes anger and resentment - particularly among those who feel they do not perform well in exams, or who believe that they are subjected to too much formal testing, say the researchers.
They also report that some of the pupils they interviewed felt that bad behaviour was used at times as an unjustified reason for moving a pupil down to a lower set - and, once they were there, there was little chance of them being moved.
This leads the authors to conclude: "The realisation that such factors (tests and behaviour) were involved seemed to indicate to some pupils that allocation to sets was not about merit, but was based on inaccurate and subjective opinions by staff."
Primary pupils acknowledged that setting prepared them for secondary in that they were moved from class to class, "more like high school". Others said: "If you're in the bottom set, it could make you less confident."
The findings, published in the Journal of Research in Special Educational Needs, appear to confirm that little has changed over the years in removing the rigid nature of setting. And the very narrow range of criteria for creating sets, namely test results and behaviour, ignores class work and effort which are rated highly by pupils, the report states.
"The main advantage of sets for most pupils was the possibility of increased teacher attention appropriate to their needs," the report says.
The researchers interviewed 78 pupils in P6-S2 in 11 schools to assess how well setting and mixed-ability arrangements were perceived to be working.
Pupils in the top sets reported feeling undue pressure, which for some was "a real problem when it came to demonstrating their abilities in formal test situations, and these same pupils felt that they were being judged unfairly as a result."
The report adds: "There is an issue to be addressed if it is the case that pupils as early as in primary 6 are reporting pressure from work in school and are noticing adverse effects on school progress and learning.
"It was clear from the comments made by the pupils that pressure was affecting their ability to perform in class and in formal testing situations."
The researchers also found that only one group had been asked by the school for their opinions about class organisation, despite the requirements in the Children in Scotland Act 1995 and the Standards in Scotland's Schools etc Act 2000 that schools and local authorities should consult pupils on such issues.
Mrs Smith and Mrs Sutherland carried out a parallel exercise, reported three years ago in the same journal, into teachers' views of setting and mixed-ability organisation of pupils. Teachers reported that it was harder to motivate pupils in the lower sets, in contrast to the top group.
The earlier research also showed that teachers tended to assume that pupils in a set class were much more homogeneous than in a mixed-ability group, which led to the increased use of certain pedagogical methods such as whole-class teaching.
This assumption, however, is contradicted by one of the pupils, who said:
"Although it's ability, it's still mixed."
Teachers also reported that sets were easier to manage and required much less work and preparation than mixed-ability groups.
Both pieces of work were prompted by the 1996 inspectorate report Achievement for All which, while acknowledging that there is no "one, universally best method of organising pupils into classes", nevertheless recommended the introduction of setting in primary schools and greater use of it across more subjects in secondary.