“There is no international standard that we can benchmark to.”

Yesterday’s frank admission from Cath Jadhav, Ofqual’s director of research, neatly sums up the bind the exams regulator finds itself in as it seeks to ensure England’s reformed GCSEs match the world’s most “rigorous” standards.

The watchdog has already been told by education secretary, Michael Gove, that “there must be an increase in demand, to reflect that of high-performing jurisdictions”, “at the level of what is widely considered to be a pass (currently indicated by a grade C)”.

That instruction, made in February last year, sounds clear enough. 

It appears to suggest the exams used in these “high-performing” education systems are more demanding than the GCSE. So the answer should be simple - identify the “jurisdictions”, look at their exams and adjust GCSEs accordingly.

Alas, as Ofqual’s chief regulator Glenys Stacey revealed yesterday, it has not proved to be that straightforward.   

“We have done some background work about how other systems test their students at GCSE level and how their students actually compare,” she said. “We looked particularly closely at other systems where their education systems might be best comparable to ours. We have not found, in that work so far, a sufficiently direct comparison to make.”

The whole process had been “quite challenging,” Ms Jadhav added.

But what Mr Gove wants, Mr Gove gets. So, despite confessing its inability to make a direct comparison with top performers, Ofqual has said that the standard for its proposed grade 5 will be “internationally benchmarked”. Indeed, it has gone as far as suggesting this standard will be “about a half to two-thirds higher than that required for a current grade C”.

To work this out, the regulator has had turn to a Department for Education (DfE) research paper published in October 2011, which sought to quantify the gap between pupil attainment in England and “top-performing countries” in the 2009 Pisa (Programme for International Student Assessment) study.

The DfE researchers have attempted to translate that gap in Pisa test points into GCSE grades. As teaching unions have been quick to point out, the exercise seems fraught with potential difficulty as they are two very different types of test.

Pisa is designed to measure how well pupils can apply knowledge in the real world. The OECD, which runs the tests, stresses they are not supposed to be something you can prepare for, or teach to. Indeed, they are not even designed as tests of school effectiveness, but to give a more holistic picture of 15-year-olds’ abilities.

GCSEs, by contrast, are school exams you revise for and under Mr Gove the qualifications are moving back towards knowledge acquisition.

The idea that an improvement in GCSE performances will mean better Pisa scores, and vice versa, seems far from certain. As heads’ leaders warn, it is not comparing like with like.

But that is what the DfE analysis is based on. It also throws further complications into the mix by using GCSE point scores, which include the “equivalent” qualifications that can be very different from GCSEs, and subjects like drama that are a world away from the reading, maths and science tested by Pisa.

The researchers even use a third different type of test - England’s national curriculum assessments - to look at progress over time.

But not everyone shares the unions’ concerns. Professor Robert Coe, a respected assessment expert who advised Ofqual on their current GCSE consultation, told TES that evidence suggested Pisa scores and GCSE grades were comparable in this way.

He admits the DfE analysis “is a bit quick and dirty” and “rests on a few assumptions you might quibble with”. But he adds: “Broadly speaking I think it’s OK.”

OK or not, it appears to be the best Ofqual has got at the moment. The paper has expressed the difference between England’s Pisa performance and 21 other systems including Shanghai, Korea, Finland, Canada and New Zealand. The regulator has taken an average of all of them as its target and proposed setting grade 5 accordingly.

In the long term the rights and wrongs of that initial analysis could be the least of the regulator’s worries. Ofqual is committed to periodically reviewing the GCSE’s benchmark with international standards.

At the same time it must ensure national grading standards remain constant from one year to the next. Those two aims could come into conflict. If Pisa tests suddenly suggest GCSEs should be toughened again, will Ofqual be prepared to abandon its aim of “maintaining standards over time”?

There is also the wider issue of whether there is any real purpose to the international benchmark.

Asked yesterday by journalists whether the thinking was that a tougher grade 5 would move England up the Pisa rankings, Ms Stacey did not disagree, though she did say it could take “a decade”.

But Professor Coe has his doubts: “Just saying ‘we will make what was a grade C a grade higher’ that is not going to change anyone’s performance is it?”