A study into the comparative value of the A-level and its competitors, such as Cambridge University's high-profile Pre-U, by the qualifications regulator is "full of flaws" and "doesn't work", according to the head of England's biggest exam board.
The research by Ofqual began this autumn and goes to the heart of one of the most contentious issues in education - the true worth of the ever-widening array of exams from which schools can choose.
But new Assessment and Qualifications Alliance (AQA) chief executive Andrew Hall has seen the methodology the watchdog intends to use for its comparison between A-levels and Pre-Us and is not impressed.
Mr Hall warned that the regulator needs to go beyond looking at exam papers. The former Qualifications and Curriculum Development Agency chief executive said he wanted it to conduct what are known as "predictive validity investigations" that assess the worth of the different qualifications in the real world.
Ofqual accepts such investigations "might be valuable" but says the data needed for them will not be available until 2013.
When Ofqual announced the research, its chief executive Isabel Nisbet said it was to "help learners, employers and teachers understand the similarities and differences across the range of qualifications available in an open market so users of the qualifications can make more informed choices".
But for that to happen, Mr Hall says the comparison of English literature and physics A-levels and Pre-Us needs to be a "bigger, longer study".
"We have seen Ofqual's methodology ... and we think it is flawed," he said. "It is a comparison of looking at scripts. That is full of flaws technically. It doesn't work.
"It doesn't do predictive validity investigations of actually saying: 'If you get this grade, what should it reliably tell you you can go on to do?'"
Ofqual says it has consulted AQA - the biggest A-level and GCSE board - and other exam boards and will finalise its methodology this month. The watchdog will conduct a second study comparing GCSE, IGCSE, BTEC and OCR National science qualifications.
It is also responding to ministers' wishes with major international research, comparing the difficulty of A-levels with equivalent exams in China, India, Canada, Singapore, South Korea, Australia and New Zealand and at least one other European country.
"I have some serious nervousness about the methodology that is going to be used for this," Mr Hall said. "Often the research that Ofqual does is not substantive and deep enough. My wish would be that they commission a university research department to do it because I think they would have that expertise."
An Ofqual spokesman said its comparability work was "rigorous" and built on work carried out by the Qualifications and Curriculum Authority, which was the "best in its field".
"We believe it important to make a start on considering the comparability between the Pre-U and A-levels at this early time, even if there will be limits on the conclusions that can be drawn," he said.
"While a full predictive validity study might be valuable, the first data on which it could be based would not be available until at least 2013. The methodology we use will make best use of the evidence that is available to us over the next year."
OLD GUARD CHALLENGED
Recent years have seen the beginning of the end of the A-level and GCSE monopoly on public examinations. At post-16, schools dissatisfied with modularised A-levels can choose from alternatives that include the more "linear" Pre-U and the increasingly popular International Baccalaureate.
National curriculum requirements have meant schools searching for alternatives to GCSEs at key stage 4 have tended to have less flexibility.
But "vocational" qualifications like OCR Nationals, often worth multiple GCSEs in the league tables, have seen a huge rise in popularity, despite critics like think-tank Civitas claiming they are being mis-sold to pupils.
The academic side is also being freed up, with state schools able to offer IGCSEs from this year.