Methodologic quality and risk-of-bias in systematic reviews of healthcare interventions: a review of methods
DOI:
https://doi.org/10.17267/2675-021Xevidence.2022.e4067Keywords:
Systematic Review, Healthcare, Methodologic Quality, Risk of bias, Critical appraisalAbstract
OBJECTIVE: To compare the characteristics of systematic reviews of healthcare interventions that assessed or did not methodologic quality/risk-of-bias of included studies. Additionally, to analyze methodologic features of those assessing the methodologic quality/risk-of-bias. METHODS: PubMed database was searched. From 25,571 systematic reviews retrieved, a random sample of 1,025 was screened. Frequencies were used to describe outcomes. Unadjusted and adjusted logistic regressions were performed to test the associations with methodologic quality/risk-of-bias results assessment. In a second analysis, systematic reviews that assessed methodologic quality/risk of bias were dichotomized according to the design of included studies (randomized clinical trials-only versus non-randomized studies of intervention or a combination of both). RESULTS: 303 systematic reviews were fully analyzed. Methodologic quality/risk of bias was assessed by 278 (92%). Methodologic quality/risk-of-bias assessment was associated with a higher number of databases searched (>4, P= 0.008), the presence of meta-analysis (P= 0.005), and the design of included studies (randomized clinical trials-only, P= 0.042). The chance of using a suitable tool and a tool designed for risk-of-bias assessment rather than methodologic quality was higher for randomized clinical trials-only systematic reviews (P< 0.05). The most used tool was Cochrane’s RoB Tool without a clear studies’ overall risk classification system. CONCLUSION: methodologic quality/risk-of-bias assessment was associated with included studies’ design (randomized clinical trials-only), a meta-analysis of data, and the number of databases searched (>4). The most used tool was Cochrane’s RoB Tool, with no clearly defined rating system. Methodologic quality/risk-of-bias assessment methods description, results, and impacts on meta-analysis, the certainty of evidence, and systematic reviews’ results are still to be consistently addressed.
Downloads
References
Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312(7023):71-2. https://doi.org/10.1136/bmj.312.7023.71
Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097
Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. https://doi.org/10.1136/bmj.j4008
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-6. https://doi.org/10.1136/bmj.39489.470347.ad
Lasserson TJ, Thomas J, Higgins JPT. Chapter 1: Starting a review [Internet]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). London: Cochrane; 2020. Available from: www.training.cochrane.org/handbook
Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). London: Cochrane; 2020. Available from: www.training.cochrane.org/handbook
Page MJ, Higgins JP, Clayton G, Sterne JA, Hrobjartsson A, Savovic J. Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies. PloS one. 2016;11(7):e0159267. https://doi.org/10.1371/journal.pone.0159267
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. https://doi.org/10.1136/bmj.n71
Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, et al. Systematic review adherence to methodological or reporting quality. Syst Rev. 2017;6(1):131. https://doi.org/10.1186/s13643-017-0527-2
Saltaji H, Ospina MB, Armijo-Olivo S, Agarwal S, Cummings GG, Amin M, et al. Evaluation of risk-of-bias assessment of trials in systematic reviews of oral health interventions, 1991-2014: A methodology study. J Am Dent Assoc. 2016;147(9):720-8.e1. https://doi.org/10.1016/j.adaj.2016.03.017
Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol. 2018;18(1):5. https://doi.org/10.1186/s12874-017-0468-4
Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011] [Internet]. London: The Cochrane Collaboration; 2011. Available from: http://handbook-5-1.cochrane.org/
Martinic MK, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203. https://doi.org/10.1186/s12874-019-0855-0
Stefani CM, Rezende LVML, Honnef LR, Oliveira LB, Lopes NC, Oliveira JMD, et al. Methodologic quality and risk of bias in Systematic Reviews of Healthcare Interventions: a methodologic review. 2021. https://doi.org/10.17605/OSF.IO/CH5T9
Green S, Higgins JPT, Alderson P, et al. Cochrane handbook: cochrane reviews: Ch 8: assessing risk of bias in included studies. Cochr Handbook Syst Rev Intervent. 2011:3–10. https://training.cochrane.org/handbook/archive/v5.1/
Wells G, Shea B, O'Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomized studies in meta-analysis, 2000 [Internet]. Available from: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp
Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5. Rating the quality of evidence--publication bias. J Clin Epidemiol. 2011;64(12):1277-82. https://doi.org/10.1016/j.jclinepi.2011.01.011
Santos MBF, Agostini BA, Bassani R, Pereira GKR, Sarkis-Onofre R. Protocol registration improves reporting quality of systematic reviews in dentistry. BMC Med Res Methodol. 2020;20(1):57. https://doi.org/10.1186/s12874-020-00939-7
Higgins JPT, Sterne JAC, Savovi? J, Page MJ, Hróbjartsson A, Boutron I, et al. A revised tool for assessing risk of bias in randomized trials In: Chandler J, McKenzie J, Boutron I, Welch V, editors. [Internet] Cochrane Methods. Cochrane Database of Systematic Reviews. 2016. https://doi.org/10.1002/14651858.CD201601
Sterne JAC, Savovi? J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:4898. https://doi.org/10.1136/bmj.l4898
Higgins JPT, Savovi? J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial [Internet]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). London: Cochrane; 2022. Available from: www.training.cochrane.org/handbook
Reeves BC, Deeks JJ, Higgins JPT, Shea B, Tugwell P, Wells GA. Chapter 24: Including non-randomized studies on intervention effects. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). London: Cochrane; 2020. Available from: www.training.cochrane.org/handbook
Sterne JAC, Hernán MA, Reeves BC, Savovi? J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions. BMJ 2016;355:i4919. https://doi.org/10.1136/bmj.i4919
Sterne JAC, Hernán MA, McAleenan A, Reeves BC, Higgins JPT. Chapter 25: Assessing risk of bias in a non-randomized study. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). London: Cochrane; 2020. Available from: www.training.cochrane.org/handbook
Sterne J, Higgins J, Reeves B, Savovi? J, Turner L. Cochrane Scientific Committee. Recommendation statement/report. Review of the development of the risk of bias tool for nonrandomised studies for interventions – ROBINS-I [Internet]. Available from: https://methods.cochrane.org/sites/default/files/public/uploads/scientific_committee_statement_report_robins_i_fin.pdf
Coleman BD, Khan KM, Maffulli N, Cook JL, Wark JD. Studies of surgical outcome after patellar tendinopathy: clinical significance of methodological deficiencies and guidelines for future studies. Victorian Institute of Sport Tendon Study Group. Scand J Med Sci Sports 2000;10(1):2–11. https://doi.org/10.1034/j.1600-0838.2000.010001002.x
Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52(6):377-84. https://doi.org/10.1136/jech.52.6.377
Tufanaru C, Munn Z, Aromataris E, Campbell J, Hopp L. Chapter 3: Systematic reviews of effectiveness. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. https://doi.org/10.46658/JBIMES-20-04
Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. https://doi.org/10.46658/JBIMES-20-01
Downloads
Published
Issue
Section
License
Copyright (c) 2022 Cristine Miron Stefani, Liliana Vicente Melo de Lucas Rezende , Lia Rosana Honnef, Luciana Butini Oliveira , Nathan da Cruz Lopes , Carla Massignan, Graziela De Luca Canto, Júlia Meller Dias de Oliveira
This work is licensed under a Creative Commons Attribution 4.0 International License.
The authors retain copyrights, transferring to the Journal of Evidence-Based Healthcare only the right of first publication. This work is licensed under a Creative Commons Attribution 4.0 International License.
