Concept article

 

Why meta-analysis can mislead: when the highest level of evidence fails in evidence-based medicine

 

Guilherme Rodrigues Oliveira1 (https://orcid.org/0009-0001-4372-7969)

Isabela Martins Vecchi2 (https://orcid.org/0009-0002-3369-2966)

 

1Corresponding contact. Universidade Estadual de Mato Grosso do Sul (Campo Grande). Mato Grosso do Sul, Brazil. [email protected]

2Universidade Estadual de Mato Grosso do Sul (Campo Grande). Mato Grosso do Sul, Brazil.

 

ABSTRACT | CONTEXT: Meta-analysis is widely positioned at the apex of the evidence hierarchy in evidence-based medicine (EBM), often serving as the foundation for clinical guidelines, policy decisions, and therapeutic recommendations. Their quantitative nature conveys an appearance of precision and objectivity that reinforces their authority in clinical reasoning. However, this perceived robustness may obscure important methodological and epistemological limitations. CONCEPTUALIZATION: This article critically examines the conditions under which meta-analysis may produce misleading inferences. We argue that the aggregation of evidence does not inherently generate validity, particularly in the presence of heterogeneity, publication bias, and variable study quality. By exploring structural limitations, including analytical flexibility, amplification of bias, and the misinterpretation of statistical significance, we demonstrate how meta-analysis can create an illusion of certainty. Rather than resolving uncertainty, they may repack it into quantitatively precise but conceptually fragile estimates. We propose that meta-analysis should be interpreted as conditional and context-dependent constructs, whose validity depends on rigorous methodological scrutiny and epistemological awareness. Reframing their role within EBM is essential to prevent overreliance and to promote more critical, responsible clinical decision-making.

 

KEYWORDS: Evidence-Based Medicine. Meta-Analysis. Bias. Reproducibility of Results. Clinical Decision-Making.

                                                          

How to cite this article: Oliveira GR, Vecchi IM. Why meta-analysis can mislead: when the highest level of evidence fails in evidence-based medicine. J Évid-Based Healthc. 2026;8:e6898. https://doi.org/10.17267/2675-021Xevidence.2026.e6898

 

Submitted Apr. 2, 2026, Accepted May 5, 2026, Published June 11, 2026

J. Évid-Based Healthc., Salvador, 2026;8:e6898

https://doi.org/10.17267/2675-021Xevidence.2026.e6898

ISSN: 2675-021X

Assigned editor: Luis Cláudio Correia 

 

1. Introduction

 

Evidence-Based Medicine (EBM) emerged as an intellectual movement aimed at reducing uncertainty in clinical decision-making through the integration of best available evidence, clinical expertise, and patient values1. Over time, however, this pragmatic framework evolved into a hierarchical structure in which meta-analysis of randomized controlled trials came to occupy a privileged epistemic position2.

This elevation has fostered an implicit but rarely examined assumption: that the quantitative aggregation of evidence necessarily produces more reliable knowledge. Meta-analysis, by virtue of their statistical sophistication and capacity to synthesize multiple studies, are often perceived as inherently superior forms of evidence. Their outputs, precise effect estimates, narrow confidence intervals, and visually compelling forest plots, convey an aura of objectivity that frequently escapes critical scrutiny3.

Yet, this perception rests on a fragile foundation. The assumption that aggregation leads to truth conflates precision with validity, overlooking the fact that statistical methods cannot correct for systematic bias, heterogeneity, or flawed study design4. As Ioannidis famously argued, the credibility of biomedical research is deeply constrained by pervasive biases, selective reporting, and methodological limitations5. Meta-analysis does not transcend these issues; rather, it may consolidate them under a veneer of methodological rigor.

Thus emerges a central paradox of modern EBM: the methodological tool designed to synthesize evidence may, under certain conditions, generate misleading conclusions while simultaneously reinforcing confidence in its outputs. Given the central role of meta-analysis in shaping clinical guidelines and healthcare policy, this paradox is not merely theoretical, it carries direct implications for patient care and public health.

 

2. The promise and the paradox of synthesis

 

Meta-analysis was developed as a response to the increasing fragmentation of scientific evidence, providing a formal mechanism for integrating results across studies3. By pooling data, it aims to increase statistical power, improve precision, and resolve inconsistencies among individual trials.

 

Within this framework, meta-analysis is often interpreted not merely as a methodological tool, but as a mechanism for resolving uncertainty. Conflicting findings are expected to converge into a single, more accurate estimate of effect, reinforcing the notion of cumulative scientific progress.

 

However, this promise is contingent upon a critical assumption of commensurability—that the studies being combined are sufficiently comparable in terms of populations, interventions, and methodological quality. In practice, this assumption is frequently violated.

 

Rather than eliminating variability, meta-analysis redistributes it within a statistical model. The resulting summary estimate may therefore represent not a clearer signal, but a mathematically stabilized expression of underlying heterogeneity6. In this sense, meta-analysis may produce precision without validity, a condition in which confidence intervals narrow while uncertainty persists at a conceptual level.

 

3. Structural sources of misleading inference

 

3.1 Heterogeneity as an epistemological constraint


Heterogeneity is often treated as a statistical inconvenience to be quantified and adjusted for. However, it represents a deeper epistemological limitation. Differences in patient populations, intervention protocols, outcome definitions, and study designs are not merely sources of noise but reflections of contextual variability that resist aggregation3.


Even when statistical heterogeneity appears low, clinically meaningful differences may persist. Pooling such studies may produce internally consistent yet externally ambiguous estimates, challenging their interpretability.

 

3.2 Amplification of bias


Publication bias remains a fundamental threat to the validity of meta-analysis. Studies with statistically significant findings are more likely to be published, cited, and included in systematic reviews7,8. This selective visibility distorts the available evidence base.


A landmark analysis of antidepressant trials submitted to the FDA demonstrated that published literature substantially overestimated treatment efficacy due to selective reporting9. Meta-analysis based on published data alone would therefore inherit and amplify this bias.


Small-study effects further contribute to distortion, as smaller trials often report larger treatment effects, frequently due to methodological limitations10.

 

3.3 Case study: rosiglitazone and cardiovascular risk


The meta-analysis by Nissen and Wolski, which suggested increased cardiovascular risk associated with rosiglitazone, illustrates how meta-analytic findings can rapidly influence clinical and regulatory decisions11. Despite its impact, the study was later criticized for methodological limitations, including sparse event data and reliance on aggregated outcomes12.


Subsequent analyses yielded conflicting results, highlighting how different analytical choices may lead to divergent conclusions from the same evidence base.

 

3.4 Epistemic fragility: aggregation of flawed evidence

 

The validity of a meta-analysis is fundamentally constrained by the quality of its included studies. As emphasized by Ioannidis, bias is not random error but systematic distortion5. When biased studies are aggregated, the resulting estimate may be more precise, but not more valid.


This principle, often summarized as “garbage in, garbage out”, remains one of the most enduring limitations of evidence synthesis.

 

3.5 Analytical flexibility and interpretative latitude


Meta-analyses involve numerous methodological decisions, including inclusion criteria, statistical models, and subgroup analyses. These decisions introduce degrees of freedom that may influence results13.

Such flexibility, while often necessary, creates opportunities for selective interpretation and reduces reproducibility.

 

3.6 Statistical significance and the illusion of evidence


The increased power of meta-analysis makes it particularly susceptible to detecting statistically significant results, even when effect sizes are small and clinically irrelevant12.


This reinforces a well-documented problem in biomedical research: the conflation of statistical significance with clinical importance. The overemphasis on p-values obscures the need for contextual interpretation of effect sizes and uncertainty.

 

4. The illusion of objectivity

 

Meta-analysis derives much of its authority from its quantitative nature. Statistical outputs, such as forest plots, pooled estimates, and confidence intervals, create a visual and mathematical representation of certainty.

 

However, this apparent objectivity masks the extent to which subjective decisions shape outcomes. Study selection, data extraction, and analytical modeling all involve interpretative judgment3. This produces what may be described as an illusion of objectivity, in which methodological formalization conceals underlying uncertainty and bias.

 

5. Implications for clinical practice

 

Meta-analyses exert substantial influence on clinical guidelines, policy decisions, and everyday medical practice. When flawed or misinterpreted, their impact is amplified. Overestimated treatment effects may lead to overtreatment, while underestimation of harm may compromise patient safety. These consequences underscore the need for critical appraisal beyond reliance on hierarchical classifications of evidence.

 

6. Toward a critical epistemology of evidence

 

Addressing these limitations requires both methodological refinement and epistemological awareness. Tools such as PRISMA and AMSTAR 2 improve transparency and appraisal standards14,15, but their effectiveness depends on their rigorous application.

 

More fundamentally, evidence must be understood as context-dependent and provisional, rather than hierarchical and absolute. Meta-analyses should be interpreted as conditional constructs shaped by their inputs and assumptions.

 

7. Conclusion

 

Meta-analyses have come to occupy a central and often unquestioned position within evidence-based medicine, shaping clinical guidelines, policy decisions, and everyday medical practice. However, their epistemic authority rests on assumptions that are frequently fragile, particularly in the presence of heterogeneity, bias, and methodological variability.

 

Rather than representing a definitive synthesis of truth, meta-analyses should be understood as constructed inferences, contingent upon the quality of their inputs, the assumptions of their models, and the interpretative decisions embedded throughout their design. Their capacity to produce precise estimates does not guarantee validity; on the contrary, it may conceal underlying uncertainty beneath a façade of statistical rigor.

 

The uncritical acceptance of meta-analytic findings reflects a broader tension within evidence-based medicine—the tendency to equate methodological sophistication with epistemological reliability. This conflation risks transforming tools of synthesis into sources of misplaced certainty, particularly when their limitations are insufficiently acknowledged.

 

Reframing the role of meta-analysis requires moving beyond rigid hierarchies of evidence toward a more critical and context-sensitive approach. Clinicians and researchers must engage not only with results, but with the processes that generate them, recognizing that evidence is not merely aggregated, but interpreted.

 

Ultimately, the strength of meta-analysis lies not in its position at the top of an evidence hierarchy, but in the rigor and transparency with which it is conducted and critically appraised. Preserving its value within modern medicine depends on resisting the illusion of certainty it can create and embracing, instead, a more nuanced understanding of evidence as inherently provisional, conditional, and open to revision.

 

Authors’ contributions

 

The authors declared that they have made substantial contributions to the work in terms of the conception or design of the research; the acquisition, analysis or interpretation of data for the work; and the writing or critical review for relevant intellectual content. All authors approved the final version to be published and agreed to take public responsibility for all aspects of the study.

 

Competing interests

 

No financial, legal, or political conflicts involving third parties (government, private companies, and foundations, etc.) were declared for any aspect of the submitted work (including but not limited to grants and funding, advisory board participation, study design, manuscript preparation, statistical analysis, etc.).

 

Indexers

 

The Journal of Evidence-Based Healthcare is indexed by DOAJ and EBSCO.

 

References

 

1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312(7023):71–2. https://doi.org/10.1136/bmj.312.7023.71

 

2. Guyatt G, Rennie D, Meade MO, Cook DJ. Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. 3rd ed. New York: McGraw-Hill Professional Publishing; 2015.  

 

3. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane Handbook for Systematic Reviews of Interventions. 2nd ed. Chichester: John Wiley & Sons; 2019.  

4. Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power. Eur J Epidemiol. 2016;31(4):337–50. https://doi.org/10.1007/s10654-016-0149-3

 

5. Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2(8):e124. https://doi.org/10.1371/journal.pmed.0020124

6. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-Analysis. Chichester: John Wiley & Sons; 2009.

7. Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629–34. https://doi.org/10.1136/bmj.315.7109.629

8. Sterne J A C, Sutton A J, Ioannidis J P A, Terrin N, Jones D R, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002. https://doi.org/10.1136/bmj.d4002  

9. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and Its Influence on Apparent Efficacy. N Engl J Med. 2008;358(3):252–60. https://doi.org/10.1056/NEJMsa065779

10. Turner RM, Bird SM, Higgins JPT. The impact of study size on Meta-analyses: Examination of Underpowered Studies in Cochrane Reviews. Plos One. 2013;8(3):e59202. https://doi.org/10.1371/journal.pone.0059202

11. Nissen SE, Wolski K. Effect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes. N Engl J Med. 2007;356(24):2457–71. https://doi.org/10.1056/NEJMoa072761

12. Diamond GA, Bax L, Kaul S. Uncertain effects of rosiglitazone on the risk for myocardial infarction and cardiovascular death. Ann Intern Med. 2007;147(8):578–81. https://doi.org/10.7326/0003-4819-147-8-200710160-00182

 

13. Simmons JP, Nelson LD, Simonsohn U. False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychol Sci. 2011;22(11):1359–66. https://doi.org/10.1177/0956797611417632  

14. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. https://doi.org/10.1136/bmj.n71  

15. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. https://doi.org/10.1136/bmj.j4008