Two years ago, I set out to answer a question: to what extent could schools be held responsible for the failings of their pupils? Over the past 20 years, the overwhelming thrust of educational policy and reform had been pinioned by a single assumption: that the way to raise standards, and thereby to improve young people’s life chances, is to make schools better places.

For years, the debate about educational standards had revolved largely around this. Are schools teaching the right things? Are standards too low?

Should there be more specialisation, or more selection? It occurred to me none of this had made a jot of difference to a significant minority of pupils who seemed to have been born to fail. Why?

I began my research at the end of a prosperous decade during which the education system had benefited from significant investment - school buildings had been improved, class sizes had become smaller, teachers had seen rises in their pay. Yet one-fifth of all pupils still left school without a single good GCSE.

Ofsted had reported that at least 300,000 16 to 19-year-olds lacked the skills to access training or worthwhile employment. And across the UK around a million young people aged 16 to 24 were deemed “Neet” - not in education, employment or training.

I hoped to answer my own question by identifying a group of drop-outs and potential drop-outs in order to investigate their educational experiences, family backgrounds, the communities from which they sprang and the changed labour markets into which they had been born. In truth, the answer was self-evident: that the amount of difference schools could make was severely limited.

Of course, schools do matter. Way back in the 1970s, a landmark book called Fifteen Thousand Hours: Secondary Schools and their Effects on Children established beyond doubt that schools with positive attitudes could change some pupils’ lives.

Previously, the prevailing view had been that children’s social baggage ensured they left school with, essentially, the knowledge and the qualifications they were destined to come out with when they went in.

It would be hard to overestimate the profound effect of that research on education policy in the past two decades. Yet the book’s authors never set out to suggest that social factors were less important than schools in determining children’s life chances.

More recently, researchers at the London School of Economics used annual schools census data to try to quantify the major factors affecting pupils’ achievement and their relative effects - school quality, eligibility for free school meals, gender and ethnicity. Strikingly, it put a figure on the proportion of low achievement that was down to schools: 14 per cent, or about one-seventh.

I set out to investigate the “other” 86 per cent of factors causing underachievement. The association between poverty and underachievement was well known - 40 per cent of pupils on free school meals achieved five good GCSEs in 2008, compared with 67 per cent of their peers. But clearly there were deeper, underlying social issues beneath these statistics.

Following the fortunes of small groups of teenagers in Barnsley, Manchester and east London, it seemed to me that somehow their path to failure started not just in their homes but also in their communities. And it seemed to me that some of the communities I visited had gradually become dysfunctional over the past 50 years - less because they had changed than because they hadn’t. It seemed to me that, faced with often cataclysmic labour market change, white working-class communities in particular had become stuck in the past.

In Barnsley, for instance, I met Ashley, aged 16 and about to leave school without significant qualifications. Ashley wanted to work in child care, which meant going to college. Her father, a quarry manager, couldn’t see the point. He had left school at 15 with no qualifications and it had worked fine for him, he said.

But, of course, the world has changed. The sewing factory in which Ashley’s mother had stitched Arnold Palmer’s golfing trousers was long gone. Yet young people were still growing up believing that they were destined for unskilled or semi-skilled manual work. I saw this attitude reflected in the white working-class families I met in all the areas I visited.

“If it worked for us, it can work for you,” parents would say. But while the world had changed, there seemed to be no-one qualified to explain to the next generation what this meant for them.

These communities had simply been unable to move on. Yes, the families I met often lacked ambition and drive when it came to gaining educational qualifications, but the truth was that in the past they had not needed it.

That these attitudes persisted despite the decline in manual labour was bad for pupils’ employment chances, of course, but also for their ability to settle at school. It left a kind of gulf between the cultures of some pupils and that of their schools.

In his 1977 book, Learning to Labour, Paul Willis had described how the subculture and language of white working-class boys at school was a sort of prototype for what they would experience in the workplace. In his view, their adolescent irreverence had a social function - that of preparing them for the larky, male-dominated world of manual work.

Although that manual work had largely disappeared from Britain, it seemed that this counter-culture still lingered. The boys I met, although very different characters, had all experienced it to some extent. All said they had enjoyed primary school, but that at around the time they hit adolescence they started to kick against education.

It was almost as if the language in which they learned was different from the one in which their schools spoke. While in the 1970s their youth culture had a function, today it was harder to see what it was for.

It was the white working-class boys who were hit hardest - although girls, who gained more qualifications at school, were, in fact, just as likely to drop out later. While boys went on to claim Jobseekers’ Allowance, girls were more likely to claim lone parents’ income support.

Yet there were significant ethnic differences in attitudes to education. Elvige, who was born in Togo, told me: “I am not a person who gets Cs or Ds.” She would tell me how her grandmother, on the phone from Africa, used to exhort: “Make us proud!”

Researchers at Warwick University had tried to unpick the reasons for this ethnic achievement gap. They found that white British pupils had the lowest aspirations, followed by Caribbeans, but they found the reasons for those low aspirations varied.

The white pupils’ lack of ambition was driven by a lack of belief in their own academic abilities and by low aspiration in their homes, but for the black Caribbean pupils it was more likely to be caused by disaffection and negative peer-group pressure.

In a sense, the white working-class pupils I met had been hampered mainly by huge geopolitical shifts that were well beyond their control. Meanwhile, politicians were busy tinkering at the margins with literacy strategies and specialisation.

The educational reforms of the past 20 years had brought improvements - better exam results, higher staying-on rates, smaller classes - but they had done little or nothing to improve social mobility. Research by London University’s Institute of Education, based on cohort studies of children born in 1946, 1958 and 1970, confirmed this. While every section of society had become better qualified since the war, the gap between the richest and the poorest in terms of their chances of gaining a degree had widened steadily - from 42 per cent for those born in 1946, to 59 per cent for those born in 1970.

It was easy to see why policy reform had focused on the small proportion of underachievement that could be attributed to schools - it was the only way an education minister could ever seek to make a difference, because everything else would be outside their remit.

Yet in doing so, successive governments had pushed schools in a direction that actually mitigated against the raising of standards for pupils at the bottom of the heap. A greater emphasis on academic standards could work for the majority, perhaps, but for the pupils whose lives I studied it was a major demotivating factor.

Politicians would do well to step back sometimes and try to see the bigger picture. Until they have found a way of doing so, a significant minority of pupils will continue to spend their school days learning to fail.

- Fran Abrams’ new book, `Learning to Fail: How Society Lets Young People Down’, is published this week by Routledge, priced pound;18.99. Links to the research mentioned can be found on www.tes.co.uk

`OFF THE RECORD, NOBODY CARES ABOUT THE NEETS’

Not long ago, I went to visit a comprehensive school in the sprawling white working-class suburbs of north Manchester.

- I was looking for information. In the course of my investigation into the root causes of educational failure, I had met - quite separately - two of the school’s former pupils. One had left early with no qualifications and was drifting into petty crime. The other - now 18 - had already done time for dealing “smack and crack”. This seemed more than a coincidence.

- I expected the headteacher to be defensive, but almost as soon as I sat down he launched into a sort of mea culpa about the almost impossible challenges he faced.

- “I’m always conscious of whether we’re doing our very best for every child. And for lots of reasons, I’ve got to answer `No’. That hurts, because you don’t come into this job to give up on kids,” he told me, adding that each year a dozen children left his school with no qualifications at all.

- The brutal truth was that, for him, pupils like the ones I met - likely to drop out, unlikely to get five good GCSEs - were not a priority. They were not among that borderline target group who could be pushed up from D grades to C grades with a minimum of effort. There were others in their classes for whom a less vigorous shove was needed in order to get over the magic line.

- The problem for this head was that if he failed to concentrate on the pupils who were easiest to reach - those who were failing to make the grade, but only narrowly - he would face sanctions.

- “The Government’s looking at me, saying I must get 30 per cent through 5 A-Cs including English and maths, and they’ll let me know when I haven’t done it,” he told me.

- “There’s pressure on us in so many ways - and I suppose some groups of students do suffer.”

- Then he paused, almost as if he wondered whether anyone was listening: “Off the record,” he said,” nobody cares about the Neets.”