The EEF view: What are the implications for education research?
Systematic reviews are important because they give us an objective overview of the evidence for a particular topic or research question. Even the most rigorous single study could be an outlier, giving a result that is either more negative or more positive than the true effect. By looking at an evidence base overall, we can limit the risk of making decisions based on results that are not representative of the likely effect in our schools.
An important consideration with systematic reviews is contrasting them with “literature reviews” or other evidence reviews that do not follow a systematic procedure. It’s possible to cherry pick evidence to support a pre-existing argument. Systematic reviews avoid this bias by setting out a process through which studies will be included and excluded at the start of the review.
Some important things to consider when looking at a systematic review are:
- Is it actually systematic or is it a literature review? Systematic reviews should have clear inclusion and exclusion criteria and should have gone through a process of comprehensively searching all of the evidence.
- Do the authors consider the quality of the studies they have included? Even if the review is systematic, it’s important that some consideration has been given to quality. Even the most systematic of reviews may be limited if only poor-quality studies have been conducted on the topic.
- Does the average impact across studies hide important messages? Sometimes systematic reviews may focus on a headline average impact through a meta-analysis but there are often important findings on the variation hidden behind this average. Maybe an approach is positive on average but has always had a negative impact when delivered to children in the early years, for example.
