You are a teacher who wants to be as effective as possible. You are a school leader who wants to make good use of finite resources. You are a policymaker who wants the best possible school system.

These are aims that sit at the heart of any education system. And, usually, an essential part of achieving such aims would be looking at what the evidence says.

On issues ranging from how you give pupils feedback to whether you set homework, from class size to the length of the school day, there is a huge body of academic research that aims to give schools a solid, factual basis for making vital decisions.

But what if education research is, in fact, facing an “existential crisis” with “terrifying” implications? This is not a hypothetical question - these ominous words are those of a leading academic who last year won the $4 million (£3 million) Yidan Prize for Education Research.

At the very time when England’s teaching profession is becoming more evidence-based than ever, it is facing the disquieting possibility that some of the research foundations on which it seeks to build may be made of little more than sand.

At the heart of the problem lies an issue that has already shaken the world of psychology to its core: the replication crisis.

It concerns a key tenet of the scientific method, which holds that if someone repeats an experiment based on reliable research, they will get the same result. Replication is something that Larry Hedges, the Yidan Prize laureate, says “holds a cherished place in discussions of science”.

But in recent years, researchers in medicine and psychology have reported that efforts to replicate many key findings have failed, and Hedges says “there were quite a few cases where there was an outright contradiction”.

“Having observed that, one of the things that I became convinced of is that - especially because I think I know what some of the causes are - it’s pretty clear that at some point this replication crisis is going to be observed in education if we don’t head it off,” he argues.

He is not alone. In February this year, Barbara Oakley, a professor of engineering at Oakland University in the US, told an audience in London that “there is truly a problem in education” when it comes to replication (see bit.ly/ResearchProb). “I have to give a lot of credit to psychology as a discipline because they’re putting their house in order. They’re publicly admitting that they have a problem, and they’re doing something about it. Education is not,” she warned.

And Hedges is clear about the gravity of the situation. “It is terrifying, and it should terrify us,” he says. “When I talk about this very issue to education researchers, what I have said is that this is an existential crisis for us.”

He is using some of his Yidan Prize money to help to address the situation before education research is discredited.

Replication may at sound like a theoretical problem, but with millions of pounds a year being poured into research, and its findings influencing everything from schools to policy, it is far from just an abstract concern.

Jeremy Hodgen, professor of mathematics education at the UCL Institute of Education, says he “agrees there is a replication crisis” and points to a concrete example where an attempt at replication failed.

It concerns PowerTeaching Maths, a technique that involves cooperative learning and embedded multimedia. Studies in the US showed it to be effective. But when the Nuffield Foundation funded an English study, “the effects found in US experimental studies were not replicated in the UK”, an Education Endowment Foundation (EEF) evidence review summarises.

Hodgen says: “You have got a study there that showed effects in the States and showed effects in a number of studies in the States, but then when it came over here [and was repeated] by the people who developed it, it didn’t show that effect.”

Often no efforts are made to replicate research at all. But, without such attempts, teachers and whole education systems may be unwittingly using techniques that do not work in their context.

Not everyone sees the problem in such stark terms as Hedges, Oakley and Hodgen, however. Sir Kevan Collins is the chief executive of the government-backed EEF, whose teaching and learning toolkit gives teacher-friendly summaries of the research evidence in key areas. “I genuinely don’t think ‘crisis’ is the right word. I think it’s a challenge,” he says.

Nevertheless, the challenge is serious enough for the EEF to be revisiting, one by one, the thousands of pieces of research that feed into those overall summaries.

“We are unzipping, if you like, all the individual studies in the meta-analysis, and recoding them individually, one by one, so that we can try and check the validity and security of those individual studies,” says Collins. “That’s a huge task, but it’s all in the quest of trying to make sure that the information is reliable. My main point is there are risks around replication, for sure, there are risks of quality.

“But I’m convinced that the evidence, notwithstanding those risks, outweighs the risk of just going by gut or by prejudice, which is the alternative. But the steps we are all taking are trying to address the challenge and questions, which should give people confidence.”

But doesn’t the fact that the EEF feels the need to carry out such a “huge task” demonstrate the seriousness of the problem? Collins’ response, while seeking to reassure, is not a denial. “You can be secure that that body of knowledge is currently the best of what we have in terms of identifying the best studies we can find since 1980, identifying studies that meet thresholds of quality and bringing those together in a sensible and coherent way,” he says.

The EEF is working with Hedges to develop an approach that tackles the challenge of replication, and Collins outlines the package of “world-leading” safeguards that it already has in place. For example, all EEF studies are delivered by people with no vested interest in the outcome; every result is published, whether negative, indifferent or positive; studies are carried out in large numbers of schools; and the toolkit is based on aggregates of studies, rather than relying on single pieces of research.

And although the EEF does not repeat its studies to ensure that the results are replicable before they are published (Collins cites the organisation’s limited resources), many of its studies are, in fact, large-scale replications of previous, smaller studies.

To acknowledge the replication crisis is one thing. To identify what research is in most urgent need of replication to ensure its validity is another.

For Hedges, “my criterion for that would probably be the ones that are most important and most costly if they are wrong. If we invest a huge amount of money in something, we owe it to society to be sure it’s right before we expend those resources.”

For Hodgen, research about feedback is a priority for replication. “There’s an area where we have got quite a lot of research but not a lot of direct replications,” he says. “We know that [feedback] is powerful. We know that it can be really positive, but it can also be negative. We could do with better described studies and then more replication, and we could do with studies on feedback to students but also feedback to teachers.”

What does all this mean at the chalkface, for the teachers actually educating our children day in, day out?

For some who are actively involved in research, the replication crisis is a concern, but not a paralysing one. They understand the importance of education research, but also its limitations.

Niki Kaiser is a chemistry teacher at Notre Dame High School in Norwich, a designated research school, and she is its network research lead. She thinks only teachers who are “very research-engaged” are likely to be aware of the replication crisis.

“Yes, of course we are concerned if it’s replicable,” she says. “We don’t want people to tell us to do things based on one study that somebody’s published and it’s got through because it had a positive result, and there are 50 people who did it and it didn’t get a positive result and didn’t get in a journal.”

But schools are complex places, each with a unique context, and research is just one thing that helps teachers when they make professional judgements, she adds.

“However robust [the research] is, you still, as a system leader or as a leader within a school or as a teacher, you still interrogate it to see how that would fit within your classroom. You are never going to get a simple enough message to plant something on someone and tell them to do it.”

What about the concerns that Hodgen has about the research on feedback? For Kaiser, this is an example of how academic research is not the only guide that teachers use.

“There is enough evidence out there that feedback can be a really positive thing. As a teacher, when you talk about teacher hunches and professional judgement, that’s one where you think, ‘Oh really? Of course it does.’ But I agree there needs to be more work on what that actually looks like.”

There is a similar message from Leon Walker, research school director at Meols Cop High School in Southport, Merseyside. It is leading a national trial of the Flash approach to marking, which the school’s head of English developed.

Delivering a trial has, he says, “really opened our eyes” to the difficulties involved in research and spreading something that works on a small, local scale to a much wider group. In particular, Walker now believes that schools need to think much more about the word “fidelity” - whether teachers are implementing any new initiative exactly as they should be.

“Evidence only takes us so far. It can help us make more informed choices, but it can’t give us final answers,” he says.

So, given the “existential crisis” in education research, what should teachers who are looking for solid evidence to guide them do?

Hodgen says that the EEF toolkit is a good place to start, with its padlock system of rating the strength of evidence being “quite a useful way of thinking about it”.

He commends the organisation’s more detailed guidance reports for the same reason, although he warns that “we are some way behind the replication crisis there”.

For Kaiser, busy teachers do not have the time to read through hundreds of individual research papers to assess their reliability.

“I guess we assume that if it gets fed through reputable organisations like the EEF or the Institute of Education then we would assume they are the gatekeepers that decide that it’s robust,” she says.

Collins says the key question is what the alternative is if teachers are not to rely on the research we currently have. “My view is you either allow yourself to be told what to do by someone else, you just follow your instinct or you turn to the best you can find, where people are trying to bring reason and evidence to bear,” he says.

Martin George is a reporter at Tes. He tweets @geomr

This article originally appeared in the 29 March 2019 issue under the headline “Evidence of a crisis in research”