Progress 8 isn’t fair if pupils have low prior attainment
Schools with large numbers of lower ability pupils will be disproportionately penalised by Progress 8 – the government’s new “fairer” secondary accountability measure – according to new data analysis shared with TES.
The Department for Education had specifically designed the measure – which will be used to trigger forced academy conversions – to avoid giving schools with relatively low-ability intakes an in-built disadvantage.
But the results of a new study by Education Datalab director, Rebecca Allen, suggest that this goal has not been met. Dr Allen used summer 2015 GCSE results to work out Progress 8 scores for all state secondaries in England.
Her study found that secondaries with lower average prior attainment, according to pupils’ key stage 2 test results, were more likely to have low scores on Progress 8, which will be introduced for all secondaries in 2016-17.
“It has real consequences for the accountability threat to schools since those with lower ability intakes are far more likely to fall below the announced floor standard,” Dr Allen said.
Failing to reach the floor standard leaves schools subject to intervention and the possibility of forced academy conversion.
Dr Allen confirmed the picture by plotting the number of secondaries that would have fallen below the Progress 8 floor standard and finding a direct relationship with pupils’ prior primary attainment. The lower pupils’ average primary test scores were, the more likely a secondary was to miss the official DfE target.
In theory this relationship should not exist, as Progress 8 is designed to be fair to secondaries by measuring the progress pupils make from any given point, rather than overall achievement. So pupils’ lower prior attainment should not disadvantage a school.
Dr Allen said that she expected much of the relationship between prior attainment and Progress 8 to persist even after schools had adapted their curriculum to the new measure.
‘Weak accountability measure’
Duncan Baldwin, deputy director of policy at the Association of School and College Leaders (ASCL), told TES he had also found that secondaries where pupils’ average primary prior attainment was higher scored better on Progress 8. “I think what people will find difficult to understand is that Progress 8 is statistically unbiased on a pupil-by-pupil basis. But it does favour schools where attainment is high and we don’t know why,” he said.
Dr Allen posed two scenarios. In the first the relationship could be explained by schools serving “relatively affluent” areas having pupils with “a supportive home environment”. In this case, “Progress 8 would seem unfair to schools serving disadvantaged communities, unfit as an indicator for school performance for parents and a weak accountability measure,” Dr Allen said. “It is almost impossible for them to compensate for home environment. It is unfit as an indicator because parents might wrongly infer their child is likely to underperform. And it is a weak accountability measure because we hold some schools responsible for things outside their control, whilst others can coast without fear of intervention.”
An alternative explanation was that “there are more effective schools serving affluent communities than there are those serving deprived communities”. She said this could be because schools in disadvantaged areas had higher teacher turnover and recruitment problems.
Fairer, but still imperfect
The founder of the National Association of Secondary Moderns (NASM) Ian Widdows represents schools that have, by definition, lower-ability intakes. “Ofsted inspectors need to understand the context of these lower Progress 8 scores at secondary moderns. The danger is that people will assume schools with low-ability intakes are second best,” he said.
Others fear Progress 8 will not work because it is underpinned by primary assessment scores, which they see as unreliable (see box).
Mr Baldwin said the link between Progress 8 scores and pupil ability was a “system issue” but that should not detract from its benefits.
“No measure is perfect. It is a much fairer measure than what we had before. It’s a much more helpful measure than the 5 A* to C was for disadvantaged intake schools,” he said.
The DfE said: “Progress 8 will reward better teaching of all pupils, no matter their background, and make the system of measuring performance fairer by focusing on progress. Every increase in grade will attract additional points in performance tables.”
‘Progress 8 is a Jenga tower’
Imagine losing your chance of career progression or even your job based on something that happened five years ago and was out of your control. That is the reality thousands of secondary school teachers will face in autumn 2019.
When GCSE results come out in August that year, students will, as usual, celebrate or commiserate. But the teachers left behind will be more nervous than ever: they will be the first to experience the full effects of Progress 8 linked to performance-related pay.
Pronouncements such as “One of my groups got 70 per cent A*-A” will be consigned to the dustbin of history. It will be all about the “progress” of all students measured against their KS2 scores to create a plus or minus residual value for each teacher.
There are some significant problems with this. First, this entire system relies on KS2 assessment data being so consistently accurate and so intrinsically reliable that it can underpin the whole system of measuring progress.
Students at KS2 are assessed through two potentially flawed assessment methods. The one-off Sats test examining students on a narrow set of parameters with all the associated “teaching to the test”, together with the internal teacher assessment where grade inflation and deflation have long been a cause of concern among primary teachers.
Add to this recent changes to the format and content of the assessments, and you have a changeable and unpredictable system. According to one recently retired primary teacher, this “intrinsic need for students to make progress” has created panic and confusion in equal measure and, ultimately, “chaos”.
Another primary teacher told me, “We cannot use teacher assessment in these high-stakes environments. Some will deflate. There is no way of making an absolute judgement. Teacher assessment is inherently unreliable.”
The Progress 8 measure is creating an assessment matrix not dissimilar to a Jenga tower, where one block out of place can lead to the collapse of the entire system.
Read a longer version of this article here: bit.ly/RogersSecondary Teachers www.rogershistory.com, @RogersHistory