Research by a team from the Centre for Educational Sociology at Edinburgh University reveals that most local authority schools and a half of independent and special schools had used NQs, mainly to replace Standard grades.
The report says opening up opportunities for 14-16s through a "climbing frame" approach looks impressive on paper but has not yet been fully evaluated. So far the new courses "appear to have been more successful in respect of access than of progression".
The researchers observed that NQ courses "have provided more flexible and accessible entry points, especially for the middle and lower attainers who were poorly provided for in the previous system. However, they may have been less successful in facilitating progression."
The study says more work needs to be done. But early conclusions point to disappointing success rates as students progress through NQ courses in S3 and S4 and lack of continuity in the curriculum and in pupils' learning.
The research also highlighted potential disadvantages of the "mixed economy" in which some pupils take Standard grades and others NQs. "Higher Still aimed to introduce a unified curriculum and assessment system, and many of the problems which it sought to redress arose from the mixed economy of Scotvec and SCE provision in S5 and beyond.
"It would be ironical if it were to introduce a similarly incoherent mixture into S3 and S4."
The report says local discretion may be commendable "but a policy for local flexibility needs to be pursued within a framework which ensures coherence and consistency. It is not certain that the emerging mixed economy offers such a framework."
The researchers, who earlier looked at the way Higher Still was introduced post-16, said this work concluded that the programme "could not be left to steer itself or be steered by the disaggregated decisions of participants".
It needed consensus around a direction of change and this applied equally to the S3 and S4 curriculum.
The Use of New National Qualifications in S3 and S4 in 2002-03 by Cathy Howieson, David Raffe and Teresa Tinklin.