A few years ago, I finished a talk at an assessment conference with an anecdote.

I recounted phoning someone who until recently had been a senior figure at one of the government’s education regulators, which relies heavily on statistics, and had recently spoken out about the number of pupils across England failing to do well at GCSE.

I had asked my interviewee whether part of the cause of that might be the normal distribution. This, as the statisticians among you will know, is a bell-shaped curve on a graph.

‘It’s outrageous’

Traditionally, examiners have tended to design tests so that pupil results follow the normal distribution shape. This means that most students will score somewhere in the middle, but there will be a guaranteed, if smaller, number of test-takers given results at either extreme of the graph.

I didn’t say this explicitly to my interviewee. But by implication the question was whether the fact that there were a certain number of lower performers was just a reflection of how statisticians had ensured that the results would come out, due to the shape of this statistical curve.

In other words, the fact that a certain number of students were below average might not be news.

The response I received, on this question of whether the normal distribution might be an influence on the seemingly scandalous numbers of “low performers”, was: “But it’s not normal! It’s outrageous!” Horrified laughs followed at the conference.

My mind flew back to this on Monday.

Some of the statistically-driven stories we read about in education seem to come dangerously close to tautology, I reflected again, following widespread coverage of a new Teach First study seemingly showing that children from poorer backgrounds were less likely to be being educated at Ofsted-outstanding schools.

This was reported as scandalous. It even prompted strange new claims, in the Daily Telegraph, that more grammar schools were the answer to this problem.

Family fortunes affect academic achievement

Of course, at face value it is very concerning if poorer children are less likely to attend a good state school than their better-off peers. But I wondered about the data on which this had been built. Was this story almost guaranteed by definition in terms of the way the statistics work? 

If a pupil’s family background tends to be very important when it comes to predicting their academic results - and, sadly, this is still the case - and if pupils’ academic results are important in Ofsted judgements, as they are, then really how surprising is it that schools with poor pupils get worse Ofsted results, on average, than those with fewer disadvantaged pupils?

In fact, a separate report last month from Teach First actually highlights this, documenting that a young person’s family background statistically explains four-fifths of all variations in each pupil’s academic results, with the school accounting for only a fifth.

If that is the case, and if Ofsted reports are highly influenced by the results of their pupils, then those reports would seem to be saying much more about the backgrounds of the children each school educates than anything the school does.

I guess if we could be confident that Ofsted reports were able to separate out the contribution of the school from the background help of families, in generating the good results on which schools are judged, we could be more sure that the statistics themselves showed up a genuine scandal. But we can’t.

We know that schools with disadvantaged intakes are less likely to be rated outstanding than other schools. Is this because schools in poorer areas are uniformly “worse” than others, or is it because their pupils are more challenging in terms of the business of generating the results on which the school is judged?

The answer is: we don’t know.

Pinpointing the influence of teachers

Ofsted inspections are largely a product of exam result data which cannot help but be heavily influenced by the educational experiences children are having - or not - from birth onwards at home.

None of this is to say that pupils in poorer areas don’t have a right to schools every bit as good as those elsewhere. Some schools clearly do buck the trend.

Teach First is obviously right to stress that the contribution a school can and does make to a pupil’s trajectory should never be dismissed. And Ofsted’s statistics may genuinely be picking up on aspects of real concern, such as the fact that schools in poorer areas may find it harder to recruit teachers.

But until we can be sure that we have truly independent measures of school quality, isolating the contribution of teachers from those generated by pupil backgrounds, we should be cautious in drawing conclusions.

Otherwise, all we’re really saying may be, sadly, that poor pupils are more likely to be going to schools attended by other poor pupils. Which is a scandal in its own way, reflecting segregation within and between communities. But it’s not the same thing as the headlines suggest.

The bigger danger, of course, is also in suggesting that this is some kind of scandal only to be understood at the level of the school.

The “failure”, then, may be pinned on the institution even though this seems both statistically highly unfair and dangerously misleading. It masks the reality of the impact poverty has on families and communities.

As it is, unfortunately this seems another example of something close to statistical tautology in education.

Warwick Mansell is a freelance education journalist and author of Education by Numbers. You can read his back catalogue here. He tweets @warwickmansell.

Want to keep up with the latest education news and opinion? Follow Tes on Twitter and like Tes on Facebook