Two prestigious projects in Scotland are facing strong criticism over their "non-intervention" stance in alerting the authorities where surveys reveal children have suffered abuse or are at risk.
The Edinburgh Study of Youth Transitions and Crime, and Communities that Care, both part-funded by the Scottish Executive, raise difficult ethical dilemmas about whether promises of complete confidentiality to children can be squared with child protection; and whether collecting important data about young people en masse means overlooking the acute needs of individuals.
One child protection specialist has branded the Edinburgh study's non-intervention with suicidal pupils "outrageous".
The Edinburgh Transitions Study of young people's criminal offending and anti-social behaviour has given six "sweeps" of annual questionnaires in 40 state and independent schools to about 4,300 pupils who started secondary in 1998.
Communities that Care, which originated in the USA and is UK-wide, has carried out research in Edinburgh, Glasgow, Lanarkshire and Midlothian. It estimates that between 12,000 and 15,000 questionnaires had been distributed to P7 and secondary pupils in 77 Scottish schools by late 2004.
The responses and other data are used to decide priorities for action in their areas. The aim is to reduce youth crime, school failure, drug abuse and teenage pregnancy in disadvantaged high-crime neighbourhoods.
Confidentiality is the key element for both projects. Young people, parents and schools are promised that, in order to encourage truthfulness, data on individual youngsters or schools will not be disclosed, published nor disseminated. Teachers must explain that questionnaires are confidential and that neither they, parents nor police will look at the responses, which may implicate the youngsters in crime But such surveys also incidentally build profiles of a minority of children in distress, or apparently at significant risk. Indeed the Edinburgh Transitions Study is likely to have assembled a unique year-on-year information profile of this minority since the age of 11. Yet no questionnaire responses, however worrying, have triggered intervention with the children.
Susan McVie, the senior researcher for the Edinburgh Transitions Study, said some young people have reported intercourse at a very early age. "We hoped the Scottish Executive would be interested in this data, which has not yet been written up for publication", she said.
However, Liz Davies, a child protection specialist at London Metropolitan University, said: "Some of these things involve crimes and illegal activities committed against children. Are we saying we ignore these matters?
"If any child has said they tried to end his or her life and no contact has been made to support that child, it would be absolutely outrageous."
Dr McVie insisted that the Transitions Study follows the strictest ethical guidelines. She said: "The right to confidentiality and the welfare of the cohort members will continue to be of paramount importance."
While the Communities that Care surveys cannot trace children through time, they too reveal worrying information, which has not triggered intervention.
For instance, children are asked if their parents try to hurt each other at home, or if adults in their home try to hurt them. In one Scottish CTC neighbourhood alone, 100 pupils answered yes to one or other of these questions.
Norma Baldwin of Dundee University, an experienced researcher and leading specialist in child protection, called for a range of safeguards to be introduced, whenever researchers are getting information from vulnerable groups such as children. She said: "For me, that's a point of principle.
It's about researchers taking full responsibility for their research and its consequences". Professor Baldwin's own research teams provide support leaflets in various community languages, and put children in touch with specific local helplines.
Dr McVie said the Edinburgh Transitions Study will now provide information on suitable sources of help and support for youngsters who have suffered from some form of neglect or abuse. Dennis Daly, CTC's director for Scotland, said he agreed that information and contact sheets could be useful.
The Transitions Study's experience, however, suggests negotiations between research projects and child protection officials currently produce unsatisfactory outcomes for both sides.
Edinburgh City Council's education department insists that information gained about child abuse must always be reported to officials and the children identified. So, to preserve young people's confidentiality, the research team agreed at the start that its surveys would not ask direct questions about physical or sexual abuse, or about sexual activity, at least until after respondents were 16. This is intended to preserve the accuracy of self-reported offending.
But since abuse, neglect and maltreatment are significant contributors to youth offending, this means the research is likely to have missed some vital data. So the Transitions Study will ask retrospective questions on abuse to young people in the next questionnaire sweep.
For education officials, their strict policy has not protected the vulnerable children. Rather, it has left disturbing information in limbo and beyond their access. However, if all guarantees of confidentiality are simply removed in future, many children could fail to answer surveys honestly about their anti-social behaviour. They might also remain too fearful to say openly if abuse was happening to them. Dr Davies, formerly a child protection manager in London with wide experience of surveying young people's experiences and safety concerns, does not believe children are put off if sensitive ways are built in of enabling researchers to put the distressed minority in touch with people who can help them.
She said: "Young people do actually respect the fact that adults care about their safety, and will take questionnaires more seriously as a result of inbuilt safeguards. Research steering groups need someone skilled in child protection who can look through the response forms to identify young people at high risk."