It found that the very basis of the Scottish Executive-backed project - online learning for Higher and Advanced Higher school and college students - has yet to be properly exploited. And the study suggests that the jury is out over whether there is a direct impact on student attainment.
Yet the researchers, Kay Livingston and Rae Condie of Strathclyde University's quality in education centre, report that 78 per cent of the 234 teachers who took part in the evaluation would recommend Scholar to their colleagues, while 85 per cent of 875 students questioned would recommend it to other students.
The programme, the report concludes, "has provided teachers with high-quality resources to complement traditional classroom teaching approaches. However, the potential of the Scholar programme to develop e-learning approaches has not been fully explored by teachers."
The result is that, while there is some evidence that Scholar has improved attainment, this may be because students who have registered are more motivated and able to begin with.
All 32 education authorities and many colleges are signed up to offer Scholar. But Dr Livingston says that schools are missing out on the scope it offers to improve attainment through "blended learning", in which class work complements independent study and online learning is linked to traditional teaching methods.
"Many teachers haven't really sat down and thought about how Scholar should fit in with what the school is doing," she said. "In some cases, teachers saw it as an extra resource, gave the students their passwords and left them to get on with it."
The report states: "A more effective online discussion facility, along with ongoing support to develop the necessary skills for teachers and students to use it, would provide a mechanism for improved networking."
It emerged that students were using Scholar more than their teachers realised, but this "reinforces the point that teachers were not discussing the use of Scholar with their students and were not using the online reporting mechanism to monitor student learning".
Both students and teachers did not appear to know enough about online learning. The majority of Scholar co-ordinators in schools and colleges and of the teachers themselves felt they had not been well prepared for the programme. Half of the co-ordinators would have preferred more staff development and almost half of the teachers would have liked more help.
The report comments: "The focus of staff development appeared to be on introducing the Scholar programme at a very general level. There was little evidence of staff development in e-learning or in pedagogical approaches.
This is an area that should be addressed.
"The majority of teachers said that they had made no changes to learning and teaching approaches in order to implement Scholar."
The study also questions whether Scholar has been able to satisfy its aim of encouraging more students to take up places in higher education - not even, ironically, at Heriot-Watt University which launched the programme four years ago.
Where schools already had good links with colleges and universities, "these have been maintained or slightly strengthened, but no new ones appear to have been established".
The researchers state: "The independent use of the materials by students, particularly when it was self-directed rather than prompted by their teachers and when they selected the mode of study, provided an opportunity to experience and develop learning strategies that will be useful if they continue into further and higher education."