Gold Standard? The Limitations of Cochrane Reviews
Dr Sean Wheatley, MSc, PhD; Science and Research Lead at X-PERT Health
Sean.Wheatley@xperthealth.org.uk
When it comes to research, Cochrane reviews have long been considered the absolute gold standard. The pinnacle of what can be produced. There are however reasons to question this, and whether they deserve their place on the pedestal many put them on.
Why are Cochrane Reviews Considered the Gold Standard?
Reasons include:
– The Cochrane Collaboration have, historically, been considered highly trustworthy and authoritative, almost to the point of being revered.
– Cochrane reviews include meta-analyses, a method of combining the results from different studies. Meta-analyses are at the top of the evidence pyramid, meaning they are considered the “best” form of research.
– The methods that must be followed for a meta-analysis to be published by the Cochrane Collaboration are clear and rigorous. This means that even amongst meta-analyses they are, in theory at least, particularly strong.
All that glitters is not gold though.
Cause for Concern no.1 – Reputational Damage
Firstly, there have been some doubts cast over whether the standards and management of the Cochrane Collaboration have been maintained at the expected level (see this, for example). It is beyond the scope of this blog to explore this in detail, but whether true or not these accusations do remove a little of the veneer from the organisation’s reputation.
Cause for Concern no. 2 – Study Quality
Even without concerns over organisational standards and leadership, it is still important to consider the pros and cons of any given piece of research. This is true wherever it sits on the evidence pyramid, and whoever carries it out, endorses it, and/or publishes it.
So, although the methodical requirements for Cochrane reviews are clear and rigorous, that does not mean the outcomes are always valid or useful. Having rigorous methods does not mean that the right questions are being asked or that the results are being represented fairly, for example. Indeed, there are multiple cases where reviews have been published by Cochrane that have major issues on these fronts.
Example 1: Garegnani et al. 2026 – Intermittent Fasting for Adults with Overweight or Obesity
One example of the above is a review of whether intermittent fasting helps people to lose weight. It can be accessed here. This was published earlier this year, and received plenty of media coverage (e.g., https://www.bbc.co.uk/news/articles/c4ge7n3pq62o). That there was so much media coverage emphasizes the potential impact of Cochrane reviews.
The general gist of most of the news articles was the same – a heavy implication that intermittent fasting did not work. As many people only see the headlines or skim the article, this likely influenced a lot of people’s opinions, and potentially their behaviours.
There were however some important limitations:
1) The article (arguably at least) asked the wrong questions.
As much as many people want to lose weight, which is what the review looked at, the numbers on the scale can be misleading. What is more important is body composition – specifically, whether there were changes in body fat (which we generally want to be lower) and lean mass (i.e., muscle, which we generally want to be maintained or increased). Weight loss is likely to be health promoting if it is because of a reduction in body fat, but may even be detrimental if it is because of a loss of muscle mass. Looking at weight loss without considering body composition is therefore not always useful.
The usefulness of any lifestyle intervention is also about more than just changes in health markers. It is important to consider whether people can and will follow it (and stick to it) in the real world too. The review did try to assess participant satisfaction with the different eating patterns, but there was not enough data in the studies that were included in the review for them to do so. It did not however fully consider other relevant practical considerations, such as whether participants found intermittent fasting easier to stick to, or whether they had their calories deliberately restricted (as opposed to any changes in how much they ate occurring naturally, because of changes in hunger for example). Considering the outcomes without considering such practical realities amounts to asking the wrong questions.
2) It misrepresents the outcomes.
Perhaps most importantly, and concerningly, the way the authors present their findings is misleading. For example, in a news article from the Cochrane Collaboration publicising the review the lead author is quoted as saying “intermittent fasting just doesn’t seem to work”. This is not true though – there was a statistically significant difference in the amount of weight that was lost through intermittent fasting compared to people who did not follow any intervention. This shows that intermittent fasting can and does work.
The same was true in the abstract of the paper itself, which states “Compared to no intervention or waiting list, intermittent fasting likely results in little to no difference in percentage weight loss from baseline”. Again, this is not a fair conclusion given that there was a statistically significant difference.
The conclusions were largely based on the fact that intermittent fasting did not perform better than “normal” calorie restriction (i.e., restricting calories more consistently, all the time). It is however important to note that intermittent fasting did not perform worse than calorie restriction. As calorie restriction is widely considered the “right” way to lose weight, and is the focus of most standard medical care and weight loss advice, intermittent fasting not being worse validates it as a suitable option (anything that is not worse than standard care should be considered a suitable option). This should have been the headline.
Example 2: Hooper et al. 2020 – Reduction in Saturated Fat Intake for Cardiovascular Disease
A second example, with similar issues, is a review on the impact of reducing saturated fat on cardiovascular disease. This is an important topic, and often a contentious one.
This review, which was originally published in 2015 before being updated in 2020, can be found here. It is highly influential, as it is often used to support guidance that people should reduce their saturated fat intake. This makes the limitations, which are discussed below, even more significant.
Key limitations include:
1) The article (arguably at least) asked the wrong questions.
The primary issue that keeps getting repeated in research and discussions on this topic – including in this Cochrane Review – is grouping all saturated fats together.
There are lots of different saturated fats, and lots of different foods that contain saturated fat. These foods have different structures, contain the different saturated fats in different quantities, and combine them in lots of different ways with lots of other different nutrients. To ask whether reducing saturated fat reduces the risk of X, Y or Z is therefore very much the wrong question. It is much more useful (and valid) to assess the impact of different foods that contain saturated fat separately, and ideally in the context of different dietary approaches.
There is already plenty of evidence that different saturated-fats (and/or foods that contain saturated fat) have different effects (e.g., this, this, this, this, this and this). This demonstrates the folly of continuing to carry out research on saturated fat as if it is a single entity.
2) It misrepresents the outcomes.
Even ignoring the significant flaw of grouping all saturated fats together, the results of this review are again misrepresented.
Firstly, the reporting focuses very heavily on a single analysis, the effect of reducing saturated fats on cardiovascular events. It largely ignores a range of other analyses for which no differences were found though. Importantly, this includes that there was no reduction in mortality in the groups who had less saturated fat. In fact, the “Authors’ conclusions”, which are all some people read, only talks about cardiovascular events, they do not mention mortality at all. This is despite the fact the assessment of mortality was one of the primary outcomes the review set out to assess.
Next, for the apparent reduction in cardiovascular events when people reduced saturated fat intake, there is insufficient acknowledgement of how fragile this finding was. Although the main analysis suggested there was a statistically significant impact, this did not stand up to further scrutiny. For example, if you only included studies with a low risk of bias, the outcome was no longer statistically significant. Or if you only included studies where there was actually a statistically significant reduction in saturated fat intake in the intervention group (something you might think is pretty important when you are evaluating the impact of reducing saturated fat intake), the outcome was no longer statistically significant.
The outcomes of these additional analyses are important, as they cast serious doubt on the conclusions of the review. They potentially even fundamentally change them. Despite this, they receive little to no coverage within the most prominent sections of the review. Indeed, you have to go to pages 154 and 155 of the paper to find these analyses*.
Garbage In, Garbage Out
The issues outlined above aren’t present in all Cochrane reviews, but they also aren’t the only issues we see. As noted before, there’s not enough space here to dig into other limitations. There is one other broad issue that often affects meta-analyses which is worth noting though, and that is the “garbage in, garbage out” principle.
“Garbage in, garbage out”, or “GIGO”, describes the fact that no matter how good the methods used for the meta-analysis are, if the studies which are included are poor (i.e., the information that is going IN to the analysis is “garbage”) then the results of the meta-analysis will not be useful (i.e., the results that are coming OUT will be “garbage”).
GIGO is tested to some degree during Cochrane reviews, as assessment of study quality and the risk of bias are required. However, the way these assessments are then used is often insufficient, as studies of a poor quality and/or with a high risk of bias are often still included in the analyses which inform the conclusions. Indeed, this was true of both the studies used as examples above:
1) In the intermittent fasting review, most of the studies were identified as having a high risk of bias, yet the overall results and conclusions were presented with limited acknowledgement of this.
2) In the saturated fat review, the main conclusions are based on analyses where studies with a high risk of bias were included, despite the fact that the main conclusion (that reducing saturated fat intake reduced the rate of cardiovascular events) was no longer true when analyses were limited to studies with a low risk of bias.
Presenting the results and drawing conclusions without adequately accounting for the quality of the studies undermines the point of assessing this in the first place. It also means that, to some degree at least, even Cochrane reviews often fall into the GIGO category.
Final Thoughts
There is always doubt when it comes to research findings, and it is important that this is adequately reflected in the conclusions. Nuance, acknowledgement or clarification that is buried within the paper or appendices is not good enough. Doing this lacks transparency. It is either lazy or deliberately misleading, neither of which should be accepted by Cochrane if they are truly the bastions of scientific rigour they claim to be.
Ultimately, many people will not read the full paper; not unreasonably, given they tend to be hundreds of pages long. It is therefore important that the more prominent sections (e.g., the abstract and the “Authors’ conclusions”) are clear, accurate and balanced. This is not the case in the examples noted above, and it is hard not to conclude that some bias is likely at play given the key issues favour current positions and penalise alternative approaches and messages.
So, What’s the Bottom Line?
Despite their reputation, there are multiple examples where reviews published by the Cochrane Collaboration fall below the standards that would be expected. Key issues include that the wrong questions are being asked, that outcomes are not being presented in a fair and reasonable way, and that poor quality studies are being allowed to influence the conclusions (falling into the “garbage in, garbage out” category).
The key point therefore is that we should not assume a study is useful and correct just because it is a meta-analysis, or just because it bears the Cochrane name.
* The impact of only including studies with a low risk of bias in the relevant analysis is mentioned in a footnote in the summary of findings table, with the “quality of evidence” being downgraded as a result. The overall conclusion remains the same though, and it is questionable whether this coverage is sufficient given that these results cast doubt over the main finding of the review.
