Teachers are back in the staffroom; children are back in the classroom. A new year has begun.

Everyone will be full of good intentions to do their best by all their pupils; to ensure that no child loses out because of their background; to make sure that everyone has equal access to the best teaching. But what if huge numbers of children are already destined to be educationally disadvantaged before they have even finished their first week?

That is the worrying conclusion of the academics behind a landmark study of setting and streaming in secondary schools in England. Among the startling findings of the team from the UCL Institute of Education and Queen’s University Belfast is that almost a third of pupils are put in the wrong maths set.

And when it comes to such misallocation, clear patterns emerge about who loses out. Girls are more likely than boys to be misallocated to low sets in maths, while the reverse is true for English. And black pupils are 2.5 times more likely than white pupils to be misallocated to a lower set for maths.

The study also found that “working-class” children and those on free school meals are more likely to be in the bottom sets, as are black pupils.

Teachers who are highly qualified in their subject are less likely to teach lower sets, while some pupils in lower sets feel limited or “babied” by their teachers. Pupils in lower sets also have less confidence in the subject, the research shows.

The implications reach beyond the immediate academic consequences: researchers have pointed to negative effects on pupils’ self-confidence, opportunities, identities and wider life outcomes.

Professor Becky Francis, director of the UCL Institute of Education, who led the UCL and Queen’s University team, is careful not to dictate to schools but, given the “risks” involved, she suggests that teachers consider “minimising setting and certainly streaming”.

“In the UK, within-school grouping is seen by many as the natural way to do things, yet the evidence indicates that it widens the attainment gap whilst making little overall impact on attainment,” she says.

Yet while the academics’ findings are stark, and follow a familiar trail of research over many years, this is contested territory.

The assumption that children learn better when they are grouped with others at the same level is deeply ingrained. According to Organisation for Economic Cooperation and Development (OECD) research, more than 95 per cent of students in the UK “attend schools where students are grouped by ability across classes”.

And a YouGov survey published this week found that setting was more popular among pupils than mixed-ability classes.

Privately, some headteachers admit they dare not move away from setting because of the fear of a parental backlash. Indeed, in 2010 the Conservatives judged ability grouping to be such a vote winner that the party’s general election manifesto promised to “encourage setting so those who are struggling get extra help and the most able are stretched”.

The contrast between the ubiquity of the practice and the seriousness of the academics’ concerns is jarring. So why do they think there is a problem with setting?

Francis notes that children’s Sats results already reflect disadvantages faced by children from poorer backgrounds or some ethnic minorities. If key stage 2 data alone is used to put children into sets at secondary school, these disadvantages are entrenched. But if schools use other factors when deciding ability groupings, even more biases come into play.

‘Entrenched’ disadvantage

Take misallocation. An unpublished research paper from the UCL/Queen’s team, seen by Tes, says that, once socioeconomic background has been controlled for, based on their key stage 2 results, black pupils are 2.54 times more likely than white students to be wrongly put in the lower maths set, while the figure for Asian students stands at 1.77. Girls are 1.55 times more likely than boys to be misallocated to the bottom set.

“I simply think it shows the power of stereotypes,” says Francis. “It’s not to suggest that individual teachers are making racist or sexist decisions on the spot, but in terms of overall trends you can see how these gender and ethnic stereotypes may be bleeding into set placements.”

Another paper, published by Francis’ research team in February, which attracted attention for its provocative title about “the symbolic violence of setting”, found that top-set students were more likely to be white and middle class, while bottom-set students were more likely to be working class and black.

“The concentration of working-class and black students in low sets within schools in England is a powerful and pernicious tool within the social reproduction of unequal power relations,” the authors wrote.

But with setting so entrenched in the English secondary system, it is not surprising that it has staunch defenders. When he was Ofsted chief inspector, Sir Michael Wilshaw railed against “the curse of mixed-ability classes without mixed-ability teaching”. Six years on, his concerns remain - particularly when it comes to less experienced teachers.

“Mixed-ability can work,” he says, “but unless you have got good teachers who know how to differentiate effectively, then it can go badly wrong, having a class where you have got potential Oxbridge candidates in the same classroom with youngsters who have difficulty with basic skills.”

Wilshaw draws on his own experiences as a headteacher. “I believed in setting because I found, particularly with young teachers, and most of my teachers were under 30, unless they are really, really skilled in teaching in a mixed-ability way, then mixed ability did not work,” he explains. “Differentiating resources, differentiating the way they plan their lessons, was much tougher, and the big issue is workload, and I didn’t want to massively increase their workload.”

However, he is not blind to some of the negatives highlighted by the academics, and says that good school leadership is vital to avoid the “real danger” that pupils in lower sets lose self-esteem, and to ensure that they receive high-quality teaching.

The academics believe setting is harmful, but, given its prevalence, the school system is unlikely to abandon it entirely in the near future. Can it, therefore, be done better?

Francis says that “if, on balance, [schools] feel there are areas where attainment grouping is important, then the challenge is to do that mindful of fairness and equality of opportunity and also efficacy”.

Her group is publishing a list of “dos and don’ts” for schools to follow. The recommendations include grouping students by current attainment only, and making “setting as subject-specific as possible” (see box, below).

The unpublished paper from the group on misallocation raises a particular issue with pupils whose set allocation is not clear-cut, and Francis has a solution that is certain to provoke controversy.

She suggests that: “Where inevitably there will be some kids on the borderline - there is quite a sizeable group that will always be on the borderline for set decisions - lottery or random methods are used to assign those pupils because that’s clearly a point at which bias can seep in.”

The, perhaps slightly less controversial, idea of randomly allocating teachers to sets was looked at when the government-backed Education Endowment Foundation (EEF) funded a two-year trial of setting. Developed by the UCL Institute of Education and involving 130 schools, it was designed to see what would happen if some of the poor practices associated with setting were stripped away. Focusing on maths and English in Years 7 and 8, the trial was based on three ideas: randomly allocating teachers to sets to prevent the lower groups having less experienced teachers; assigning pupils to sets based on independent measures of attainment rather than subjective judgements; and providing opportunities to reassign pupils through the year based on up-to-date measures of their attainment.

Its most notable finding, however, was just how difficult it is for schools to make such changes. Independent evaluators from the National Foundation for Educational Research (NFER) found that some teachers believed that what they were being asked to do was no different to their existing practice; others said that changing the way they put pupils in groups was onerous and hard to do.

Indeed, most schools simply did not implement some aspects of the programme, such as randomly allocating teachers to sets. In an update to its Teaching and Learning Toolkit, the EEF says that this attempt to introduce best practice in setting “had no overall impact on attainment”, and this was “largely because it was a challenge for schools to change their setting practices”.

What, then, of the alternatives? A second EEF-funded project sought to examine mixed-attainment teaching.

Again, the pilot demonstrated how ingrained the concept of setting is in England, with the researchers finding it hard to recruit schools willing to try mixed-attainment teaching for the study.

The NFER found that staff who did take part had mixed views, with some enjoying it, and others struggling with differentiation of pupils. But the evaluators concluded that “most interviewees felt that the intervention had a positive effect on student outcomes; with most believing that the least able students had particularly benefited”.

The EEF has updated its toolkit to contain a new section specifically on grouping within mixed-ability classes. It tells schools that “the evidence on within-class attainment grouping indicates that it is likely to be beneficial for all learners, providing an average benefit of three months’ additional progress. However, there appears to be less benefit for lower attaining pupils than others.”

For Steve Higgins, a professor at Durham University’s School of Education and lead author of the Teaching and Learning Toolkit, the updated analysis “shows us that the impact of attainment grouping is dependent on the type of grouping used”.

“Based on the evidence we have,” he says, “setting or streaming pupils in different classes by prior attainment appears, on average, to have had a small negative effect. In contrast, grouping pupils by attainment within classes had a positive overall impact in the studies we reviewed.”

However, he echoes Francis’ warnings about the impact on children from disadvantaged backgrounds of putting pupils into any groups based on their attainment.

“Both types of grouping appear to be more beneficial for higher-attaining pupils than for lower-attaining pupils, and lower-attaining pupils tend to be disproportionately from disadvantaged backgrounds,” he says.

“The effects are small and we do not yet fully understand the reasons for these effects. Overall, the quality of teaching is going to be more important than the type of grouping.”

And there is a wider warning in the EEF toolkit that any groupings of pupils based on their attainment “may also have an impact on wider outcomes such as confidence”, noting evidence that it may have “longer-term negative effects on the attitudes and engagement of low-attaining pupils”.

If schools do decide to move away from setting altogether and towards mixed-attainment classes, one approach favoured by ministers and already used by many schools seems a natural fit: mastery.

It involves giving almost all of the children in a class the same content, and focusing on developing deep understanding rather than giving different tasks to different groups. The class should not move on until everyone understands the concepts.

For Francis, “that would fit very well” with teaching mixed-attainment classes, and it is an approach that The Elmgreen School in Lambeth, South London, is adopting this year as its maths department moves from sets to mixed-attainment teaching.

“We are not dumbing down,” says headteacher Dominic Bergin. “We are saying that these are the expectations that we have, and this is where we expect everybody to go, and the task of the maths department is to ensure that that happens” (see box, below).

The research raises problems with setting, but it also vividly demonstrates the reluctance of schools to move away from this approach. Will Elmgreen prove to bean isolated example of a school that phases out setting?

Francis is in no doubt about the scale of the barriers. The academic sees the desires of “demanding, perhaps middle-class, parents” as a “genuine pressure for schools … especially in a marketised context where schools want to encourage families and pupils on to the school roll”.

“Whether it’s right or wrong that you have lay parents dictating school policy, I don’t doubt that the pressure feels real,” she says.

Martin George is a Tes reporter. He tweets @geomr