Diane Abbott says it is a “silent catastrophe”: it is also a long-running one. More than 30 years have passed since the first attempts to address the academic failure of children from Caribbean and other minority backgrounds - long before the Rampton and Swann reports of 1981 and 1985 recognised their plight officially.

Decades of hand-wringing have made little difference. Children from minority communities still struggle in the classroom. For the moment the issue is highly visible, thanks in part to the publicity machine cranked up by Ms Abbott, the black Labour MP for Hackney North and Stoke Newington in London. The Commission for Racial Equality has also been applying pressure, particularly since last summer’s riots in Asian areas of Bradford, Burnley and Oldham, where educational breakdown was cited as a contributory factor.

But one development which has focused attention on minority ethnic pupils is a startling piece of analysis commissioned by the Office for Standards in Education. Published in 2000, it concluded that, despite years of progress for most pupils, things are worsening for black and Muslim children. Standards in education, educational inequality, mapping race, class and gender found that since the early 1980s, the achievement gap between white children and their black and Muslim peers had almost doubled.

While all pupils have improved to some extent over the past two decades, the majority have been able to progress very much faster. Conducted by Professors David Gillborn from the Institute of Education in London, and Heidi Mirza from Middlesex University, the analysis concluded that league tables and targets had only made things worse. Black pupils, said the authors, were likely to be overlooked when schools decided who could be coached up to a C-grade at GCSE. Gillborn, believes that the Government’s interest in “gifted or talented” pupils will have the same discriminatory effect.

The report is scathing about the lack of a coherent national strategy, pointing out that a third of local authorities still fails to keep adequate statistics.

There is, as a result, no accurate national breakdown of how different ethnic groupings perform (although this may be remedied by recent amendments to the 1976 Race Relations Act which oblige public authorities to keep such data).

Ministers point to encouraging results from their recent Youth Cohort study. The Department for Education and Skills concludes that black pupils have seen rising attainment at GCSE. Between 1998 and 2000, the proportion getting five good grades (A*-C) rose from 29 to 39 per cent. At the same time, the proportion of white pupils doing so rose from 47 to 50 per cent. The only groups to make no progress were Bangladeshi and Pakistani, who each registered 29 per cent in 2000.

Ms Abbott has, however, cast doubt on this analysis, arguing that when the results of African pupils are removed from the “black” figures, Caribbean pupils show little progress. She says this has been confirmed to her, privately, by the DFES.

For more detail, researchers must look to figures kept by some of the more progressive local authorities. The statistics from Birmingham are typical (see boxes) of the national trend: black pupils arrive scoring well at the age of five, by 11 they are beginning to fall behind and by 16 they are way behind. Muslim pupils, in contrast, are struggling when they first reach the classroom but then make steady, if unspectacular, progress.

Only 17 per cent of black boys in the city get five GCSE passes at grades A*-C. This compares with 27 per cent of Bangladeshis, 31 per cent of Pakistanis, 39 per cent of whites and 49 per cent of Indian boys. Black girls do twice as well as the boys, with 34 per cent getting five good grades, but they still trail behind. Bangladeshi and white girls are equal at 50 per cent, while Indian girls come top, with 65 per cent gaining five A*-Cs.

Black pupils, boys in particular, have tended to dominate the debate. One reason is that little is known about Asian pupils, many of whom also struggle. There is scant research and it is only in recent times, for example, that Bangladeshi, Pakistani and Indian-origin pupils have been separated out statistically.

But in the end black pupils get the most attention because, however variable or non-existent the statistics, no one is in any doubt that they leave school with the worst academic results of all - despite having no language problems.

Even more demoralising is the steep decline in performance over the period of compulsory education.

Nationally, as in Birmingham, black pupils score as well as white at the start, Professor Gillborn says. But by 16, they are at the bottom of the exam tables. Many never even reach the exam hall, as the exclusion rate for black boys is extraordinary. Depending whose figures you choose, they are between three and five times more likely to be excluded from secondary school than their white peers. Official figures show around 38 black pupils per 10,000 are expelled every year compared with only 13 whites.

Since the early 1970s they have also been over-represented at the “medical” end of the education system: categorised as “educationally subnormal” or “disturbed”.

The explanations for all this are varied and uncertain. Racism, direct or indirect, is often held responsible. Many black parents are convinced their children are picked on by white teachers. Some suggest that there are cultural misunderstandings, with the body language of African-Caribbean youngsters blamed for appearing threatening or insolent. Others, including Ms Abbott, believe the sheer size of many black teenagers creates fear among teachers.

Black parents also complain bitterly about the low expectations they feel schools have for their children. Mike Phillips, the novelist and academic whose brother is London politician Trevor Phillips, has famously refused to send his youngest son to school and is educating him at home.

The shortage of black and Asian teachers is another problem. They account for less than 4 per cent of the teaching force in England and Wales, even though around 12 per cent of pupils are from minority families. The Teacher Training Agency says, however, that 7 per cent of recruits are now from these backgrounds.

Ministers can also take credit for recent legislation which means that from today schools must publish a “race equality policy”. This requires them to monitor the ethnic breakdown of admissions, achievement and exclusion and draw up a strategy to improve them.

As for solutions, take your pick. Schools and education authorities have been urged to engage with black pupils and families, to raise expectations and to cut exclusions - views aired recently at a major London conference on black pupils staged by Ms Abbott and the Greater London Authority.

But another voice is now emerging: some black teachers, commentators and community leaders are urging parents to champion education and steer children away from a street culture that is at best anti-learning and at worst criminal.

One of Britain’s most prominent black heads, William Atkinson, is in little doubt that a street culture characterised by rap, violence and misogyny is causing serious disruption in the classroom. The proportion of pupils involved is small, says Mr Atkinson, head of Phoenix high school in Hammersmith, west London. But it is enough to cause serious disruption - often to the education of other black pupils, a phenomenon he describes as “a new form of black-on-black crime”.

Dr Tony Sewell, a lecturer in education at Leeds University, has said that “teacher racism and low expectation do exist, but there is also an issue of black peer pressure and how that relates to motivation. African-Caribbeans in particular are caught up in a culture that sees learning and intellectual activity as anti-black”.

In return for accepting this, says Mr Atkinson, black families have a right to expect a fair shot at getting places in the most successful state schools. How, he asks, can we truly judge the performance of any group if they are imprisoned within the least successful, and often most disrupted classrooms in Britain? Black and Muslim pupils are rarely seen in high-achieving schools. And until they are, he says, there is little chance of anyone finding the answer.