The report by the University of London's Institute of Education, backed by the Gatsby Foundation, says funding National Vocational Qualifications according to output has worked well in some ways but "the overall system does not include the necessary checks and balances to ensure probity and reliability".
The claim follows several cases of alleged fraud involving NVQs. In the latest episode, two people were arrested at the Centrex training organisation during enquiries into an alleged Pounds 1 million scandal concerning the awarding of NVQs without sufficient documentation.
The report, Output-related Funding and the Quality of Education and Training, by Geoff Stanton, senior research fellow at the school of post-compulsory education and training at the University of Greenwich, was being studied at a conference of training agencies and academics this week.
It says that NVQs are least suited to payment by results because trainees are assessed by the same people who teach them, whileA-levels, assessed by an externally marked written examination, are best suited.
Most output-related funding, the report explains, is linked with NVQs, provided mainly by training and enterprise councils. Schools are funded for A-levels according to numbers recruited, while in further education colleges finances are given according to the number of students doing courses.
Mr Stanton states that output-related funding has been developed by TECs to encourage innovative and efficient means for people to acquire skills and to discourage trainees from being put on unsuitable programmes. It is also simple to operate. But it means the burden for maintaining rigorous assessment procedures is shifted to awarding bodies.
He concludes: "Output-related funding has advantages but, if used alone or to excess, it carries serious risks. These risks are increased if it is used for a qualification which is still in the early stages of development, or where the nature of what is being measured necessitates the extensive use of teachers or trainers as assessors. Both circumstances apply to NVQs."
The report follows the recent high-profile study by Peter Robinson of the London School of Economics, published two weeks, which challenged government claims about the success of NVQs.
Mr Stanton told The TES: "A certain amount of output-related funding is good but it doesn't need more than 10 per cent to encourage flexibility and reward efficency.
"If you go over 10 per cent, it increases pressure on the assessment system to produce results and can skew recruitment of students towards those who are likely to get good results."
The Association for Colleges has also said it supports an element of payment by results as long as it is limited to about 10 per cent of total funding. John Brennan, the AFC's director of further education development, said: "A small element of output-related funding is good for providing a spur to achievement and reducing drop-out rates.
"But it should be limited to a small proportion of the total, otherwise it gives people an incentive to cheat by exaggerating successful results."
But Chris Humphries, director for policy and strategy at the TEC National Council, defended the current system.
"The evidence suggests that having a reasonable amount of output-related funding encourages greater attention to supporting the trainee to a successful completion of their qualification. There are good arguments for extending it to schools."
Ministers have said they want to give academic and vocational qualifications equal prestige and create a "level playing field" for funding, and a government White Paper due this autumn is expected to include proposals for a shake-up.
Earlier this year, Sir Ron Dearing heralded a fundamental reorganisation of the post-16 education and training system, involving the three "pathways" towards qualifications - the academic A-level route, the applied academic involving GNVQs and the vocational, centred on work-related NVQs.
But the National Association of Head Teachers is strongly against any output element in school funding. Director of professional services Rowie Shaw said: "It's a heavy-handed approach which would mean some schools becoming much more selective and others becoming unviable because senior staff would not be able to plan their provision."
Mr Stanton suggests that funding for all courses should be related to the numbers of students enrolled, the nature of the learning programme and the results obtained. No single agency should have primary responsibility for setting standards, awarding national qualifications or providing related education and training, he said.