Risk of bias analysis in diabetic retinopathy randomized clinical trials evaluated by RoB-1 tool from Cochrane systematic reviews

| The objective of clinical trials is to answer about intervention in the real-world, for which they must be properly designed and executed by presenting the results reliably with the findings and in a clear way. OBJECTIVES: To identify the risk of bias in clinical trials about interventions for diabetic retinopathy and/or diabetic macular edema from Cochrane systematic reviews. METHODS: A sensitive search strategy was designed to search Cochrane systematic reviews of interventions in diabetic retinopathy and diabetic macular edema. The assessment of the risk of bias was captured as presented by the author. FINDINGS: We found eight SR and one meta-analysis network totaling 116 randomized clinical trials. Our sample revealed that among the domains randomization, allocation secret, masking of participants and personnel, incomplete outcomes, selective outcomes and others, the risk of bias assessed as low ranged from 30.4 to 49.1%; unclear risk between 22 to 56% and high risk from 1 to 21.7%. CONCLUSIONS: The risk of bias in diabetic retinopathy randomized clinical trials exists in high frequency and the reader must be aware of it.


Introduction
Diabetic retinopathy (DR) is the leading cause of blindness between the ages of 20 and 74 1 , which represents the economically active population. It is a disabling disease with a high negative impact on public health, on the social security system and on the self-esteem of patients and their families. It has shown an increase in incidence and prevalence 2 due to increased life expectancy and inadequate experiences imposed by life in urban centers such as excessive intake of processed foods with high carbohydrate content and reduced physical activity. 3 As a result of the increased incidence, the demand for more effective and lower-cost therapeutic measures has been increasing, and, consequently, the need for studies to prove the effectiveness of these new interventions. However, these studies must present reliable and clear results for safer decision-making.
Randomized clinical trials (RCT) are the appropriate study to assess the impact of an intervention in practice, because it is the study design that makes fair comparisons of interventions to projecting them into the real world. The number of publications in DR increased from 857 in 2010 to 1573 in 2019 4 but only 2.15% of all publications in ophthalmology are randomized controlled trials (RCT). 5 RCT are very well designed studies that must meet the guiding parameters that define it in order to achieve the intended purpose. 6 There is evidence that failure to respect these parameters can lead to bias. Bias is a systematic error, or deviation from the truth 7 , perhaps, if expressed more correctly, bias is a systematic error that can lead to a deviation from the truth. We must consider the risk of bias when reading RCT, because they can lead to errors in the interpretation of outcomes. Although, how much bias affected the results of a study 8 remains uncertain.
According to the Cochrane Handbook 9 , a tool used to guide systematic reviews (SR), the evaluation of the risk of bias 10 is mandatory to assess possible biases in the clinical trials presented in the sample and, thus, allow the reader to analyze them in the most suitable way. SR that do not have risk of bias may be unreliable and may overestimate the benefits of an intervention.
It is known that some characteristics of clinical trials can lead to exaggerated effects of outcomes such as funding status 11 , number of centers participating in a trial 12 , early discontinuation of a trial 13 and developing country situation. 14 Studies with more positive results have a greater tendency to be published and are much more attractive to the decision maker. Given the above, there is an urgent need to demand methodological rigor in scientific productions so that they fulfill their greatest purpose as a science, the well-being of human beings.

Objective
To identify and analyze qualitatively the risk of bias using RoB-1 tool 15 reported by Cochrane systematic reviews about interventions for diabetic retinopathy and/or diabetic macular edema.

Study design and setting
This was a cross-sectional analysis of literature developed at developed at Department of Ophthalmology, Universidade Federal de São Paulo (UNIFESP), São Paulo, Brasil.

Inclusion criteria
To obtain a representative sample of risk of bias from intervention RCT, we considered all Cochrane systematic reviews and Cochrane network meta analysis that considered only RCT about interventions for diabetic retinopathy and /or diabetic macular edema that included any clinical trial and presented an assessment of the risk of bias using the RoB-1 tool. 9

Searching for reviews
We conducted systematic searches in the Cochrane Database of Systematic Reviews using a sensitive search strategy (Table 1) on April, 23 2021.

Selecting reviews
Two authors (VM and VB) independently selected the SR obtained from the search strategy according to the inclusion criteria. The differences were resolved through a consensus.

Selecting and presenting the results
Two authors (VM and VB) captured the risk of bias analysis as presented by the author of SR in the respective review, all SR in this sample evaluated the risk of bias according to the RoB-1 tool. 9 Repeat clinical trials were withdrawn, before being removed the results were compared to the RoB evaluation and in case of disparity, the clinical trial was read in its entirety and re-evaluated for tiebreaker. 9 The analysis of the Cochrane RoB-1 tool 9 follows its own path and provides for the evaluation of seven domains according to the following criteria:

RoB-1 analysis
Low risk: when the process has been properly described in detail and leaves no doubt of its execution.
Unclear: when the description of the process does not exist although the author claims to have occurred or if the description was unclear, causing doubts to the reader.
High risk: Clear absence or misrepresentation in the domain execution process.

The domains:
Random sequence generation: Proper randomization responds to the principles of randomness ("alea") and equal chances for all randomized participants.
Allocation concealment: To guarantee randomization, it is necessary to ensure concealment with appropriate techniques, such as dark and sealed opaque envelopes, which guarantee the team's non-access to which group the participant was allocated.
Blinding of participants and personnel: The blindness of the participants and personnel ensures that the results are free from any induction. However, in some cases, blinding is impossible in studies with laser application and invasive procedures. In evaluations of oral drug interventions, correct blinding should describe that the control group received drugs in the same quantity, color, smell and appearance as the intervention group. In all cases, the control procedure must be described in as much detail as possible.
Masking of outcomes assessment: The blinding of the outcome evaluators must be clearly presented with the precise description that the evaluators did not have access to randomization; that the allocation secret was opened before the whole team at the end of the study, that the evaluators had restricted access to the images and results without access to the participant or the other team members. Incomplete outcome data: Occurs when participants drop out or are withdrawn from the study after randomization or do not attend an orientation meeting or should have been measured or unable to complete diaries or questionnaires or they cannot be located (lost for follow-up) or are improperly removed or lost records.
Selective reporting: It occurs when, due to the lack of publication of the study record or protocol, it disables the confrontation of the outcomes with the original interest of the author.
Others: Project-specific polarization risks; Baseline imbalance when there is an imbalance in the characteristics of the populations between the intervention and control arms; Blocked randomization in non-blinded trials; Differential diagnosis activity; The conduct of the study is affected by interim results (for example, recruitment of additional participants from a subgroup that shows more benefits); There is a deviation from the protocol; Inadequate administration of an intervention; Contamination through the use of other interventions that may influence; overly broad inclusion criteria for participants; use of an inadequate instrument to measure results that may lead to an underestimation of outcomes.

Results
The search strategy found 195 SR, 185 were excluded by selecting the title and abstract. After reading the full text of the text, one RS was excluded for considering quasi randomized clinical trials reaching a sample of eight Cochrane systematic reviews [15][16][17][18][19][20][21][22] and one systematic review with network meta-analysis 23 according inclusion criteria as seen in Figure 1. Four addressed anti-VEGF injection 15,17,19,23 , one addressed steroid injection intra vitreous 20 three assessed laser application 16,18,21 and one assessed systemic intervention 22 as seen in summary of characteristics of SR in Table 2. 127 RCT were found, 11 repeated studies were removed totaling our sample of 116 RCT published between 1977 and 2018. Two SR 16,22 evaluated the risk of bias for primary and secondary outcomes separately, in this work we will consider only the risk of bias assessed in the primary outcomes. One RS 20 analyzed for the domain incomplete outcomes results per protocol and intention to treat, in our study we will only consider intention to treat. The total risk of bias can be seen in Table 3 and the summary of risk of bias according to RS are presented in Table 4.

Discussion
Bias is a systematic error that can lead to underestimation or overestimation of outcomes.
According to the Cochrane Manual, the risk of bias assessment is mandatory for SR submission. This assessment should be performed by at least two independent authors and a third one for tiebreakers, in order to increase the reliability of the assessment. Although the parameters of RoB-1 tool are objective and clear, with items selected for theoretical and empirical foundations 24 , proposed by clinical research methodologists, they probably still need refinement, as they still depend on the reader's judgment and understanding, making it a subjective assessment.
Randomization is the first of the domains described by the RoB-1 tool 24 , it is anchored in the principles of randomness and equal chances among the participants. Maintaining your integrity and unpredictability avoids the distortion of results and the influence of known or unknown prognostic factors. The non-guarantee of randomization provides for the risk of a selection intervention according to the author's interests. In our sample 62 (53.4%) RCT was classified as unclear in randomization, although 104 (89.7%) of all samples was published after 1996, when the first CONSORT 25 was published, already predicting the mandatory description of randomization in clinical trials. One clinical trial of the sample received a high risk at randomization, although in the title of this trial was described as a randomized study in the description of the text we were able to observe the non-guarantee of unpredictability 26 which occurred due to naming the right eye for a certain type of intervention and the left for another intervention if the participant could enter both eyes in the study.
The allocation concealment sequence is the guarantee of maintaining the confidentiality of randomization, nonetheless, misallocation generation is causally linked to bias or is an indirect marker of other factors associated with bias. 24 In our sample the allocation showed numbers close to those of randomization, with the criticism of not justifying its presence since the majority occurred after the introduction of first CONSORT. 25 Blinding, or masking, has the objective of not contaminating the results by suggestion or suggestive influence of the participants or evaluators, although it is very difficult in some interventions. As is known, there are studies in which masking becomes infeasible. 27 Masking the outcome assessors is not very difficult, as we can capture data and results and present them to independent experts, the difficulty lies in masking the participant and the team in some surgical and intervention procedures, such as surgery, intravitreal injections and evaluations with lasers. The lack of masking in randomized trials has been shown to be associated with more exaggerated estimated intervention effects, by 9% on average. 28 In our sample, the author of one of the SR 16 reports that knowledge about the result of photocoagulation was notorious among the included studies and, therefore, could affect the evaluation by assigning a high risk of masking participants and evaluators. Another SR 18 attributed a low risk of bias to masking participants in one RCT when comparing different laser techniques, in this case it was possible because they all received some photocoagulation. One SR 22 did not present the masking of the participants in the risk of bias table, but in the writing she only reported the bias of the evaluators. This same review did not assess the masking of an SR and did not explain why. The lack of data for the judgment is benefited by doubt and evaluated as unclear by RoB-1, the high risk is when it is explicit that the process did not occur or occurred in a mistaken way.
Depending on the type of laser evaluated, the blinding of the participants is difficult due to the multiple symptoms that the procedure can cause and for the evaluator it is also difficult to mask it because according to the laser it promotes scar on the retina. In the case of intra-vitreous injections, the masking of the participants can be observed adequately in several RCTs with the manipulation of the eyes in the operating room simulating the intervention, but with conjunctival injection of placebo.
Incomplete outcome data also known as missing outcome data, due to friction (dropout) during the study or exclusions from the analysis is also responsible for causing bias in outcomes. To conclude that there is no missing outcome data, the reader must be sure that the randomized participants were all included in the analysis, in this case it is considered low risk.
An intention-to-treat (ITT) analysis is often recommended as the least biased way to estimate the effects of intervention in randomized trials. It is based on three basic principles of keeping the participant in the intervention group to which he was randomized, measuring results in all participants and including all participants in the sample are obviously mandatory conducts, but the assessment of outcomes according to the expected follow-up is not always it is possible due to friction, so few works can perform a true ITT clinical trial. 29 In this work, we can observe an SR15 in which the author considered the RCT author's recognition of having had losses in the study as a low risk, although it was above acceptable, although not adequate with the definition provided for in this work.
When there is uncertainty in lost data, it is classified as unclear risk and high risk is when losses above 20% occur. Some small losses are out of control and are acceptable. The 'rule 5 and 20' (ie, if> 20% of missing data, then the study is highly biased; if <5%, then, low risk of bias) exists to help the reader understand the missing size. 8,30 Selective reports are not only incomplete reports, but their omission makes the results biased and inconsistent and may overestimate the benefits of an intervention. 31 In addition, statistically significant effectiveness results are more likely to be published than ineffective results 32 , therefore, the publication of the study protocol to face the interest of the original research is fundamental to the publication's credibility. 33 it has no economic cost and no difficulty justifying its absence and appears to be a practice observed in the diabetic retinopathy literature. 34 In our sample, the lack of publication of the confrontation protocol was considered to be an unclear risk of bias. A high risk of bias in selective reports was considered in cases in which, when confronted with the protocol, results were omitted or unexpected data were published. Although the lack of registration of the protocol is not justified, since no fee is charged, it presents no difficulties and since 1997 it is mandatory, according to the first federal law of the United States, to require registration of the trial (Modernization of Food and Drug Administration Act 1997 (FDAMA) in Congress approves the law (FDAMA)). 35 In our sample three SR 15,19,21 reported low risk of bias without mentioning the comparison with the protocol, considering only what was foreseen in the description of the study methodology with the published results, not being in accordance with what was foreseen by the RoB-1 tool.
In evaluating other forms of risk of bias, some authors considered the pharmaceutical industry sponsorship to be high risk of bias 21,22 , although it is a conduct that should be evaluated with great caution as well as the sponsorship of some authors by the pharmaceutical industry. Post-randomization exclusions 15 , lack of baseline balance 23 , baseline registration after study start, exclusion of analysis participants who discontinued study medication, representatives of the pharmaceutical company that funded the study as members 21,22 , safety monitoring committee and early study interruption 16,23 , evaluation of the other eye as a control using medication that may have systemic absorption 19,23 were also considered high risk in risk of bias. The lack of conflict of interest report was considered unclear in risk of bias. The imbalance of the arms at the baseline was considered to be of high risk, the inclusion of both eyes was considered an uncertain risk, as well as the initial uncertainty of the study in one SR. 23 In our sample we found four (3.4%) RCT qualified as "low" in all domains of Rob-1, in the literature this figure is 6%. In the literature at least one risk of bias as unclear appeared in 89% of RCTs, this sample showed 96.6% (112). In the literature at least one risk of bias as unclear appeared in 89% of RCTs, this sample showed 96.6% (112) in a date from 2016 study 36 these discrepancies may have occurred if the sample compared had very old clinical trials or pre-CONSORT.
It is noticed that other sources of bias may arise, in addition to those predicted, according to the study's intervention, requiring analytical technical guidance to pay attention to possible new sources of bias. In our sample, we found an example of this in intravitreal injection when the other eye was evaluated as a control, in this case the RS author mentions that there may have been systemic drug absorption. 19, 23 RCT are sources likely to present biases that can negatively impact the results and conclusions. Since the beginnings of Cochrane, its creators were concerned about the risk of bias, as scorning it can seriously pervert the result of an SR. 24 In this study, the sample was judged to be at high risk of bias and may be higher than the real one since the unclear and subjective assessment is very high and may be bearing many high risks that were not properly classified because it is not clear how the domain was developed. Rob-1 tool was presented in this analysis because it was presented by all SRs in the sample, the objective was to present the tool used by each SR, whatever it was. This is likely to have occurred because all SRs date back to 2011, when the introduction of the new tool took place. 24 Although there were some doubts and failures in the judgment of the risk of bias as predicted by the RoB-1 tool 24 , the number of risk of bias found in this work was still very high. This work alerts the reader that the risk of bias in RCT in diabetic retinopathy exists and he must be aware of the methodology used in the study for the analysis of the results and, although of clear importance, there are still SRs that resist presenting them. 37 It also showed that 89.7% of the included trials were performed after the introduction of the first CONSORT in 1996, that is, lack of adherence to the testing targeting instruments.
The limitation of this work is that the sample was restricted to Cochrane systematic reviews, which, although it is a high standard in the execution of the SR with multiple authors participating in each phase of execution, reducing the subjectivity inherent in the process and reducing inconsistencies 38 , we failed to evaluate other SR. If we increase the sample by evaluating the bias of other RCTs, we can find a different number in the quantity and quality of these biases. 39 In the literature, we can observe that the lack of adherence to the CONSORT rules 25 in diabetic retinopathy is clearly presented 40 as well as in other study designs in diabetic retinopathy 41 , which may lead the reader to wrong conclusions and this is the advantage of this work, alerting the reader to carefully read the methodology of scientific publications on diabetic retinopathy.

Conclusion
There is a high risk of bias in diabetic retinopathy RCTs, requiring a careful reading of the methodology so that the results are taken into account for all facets. It is clear that strict respect by authors to the rules dictated by CONSORT 25 is mandatory to minimize bias, but it is also essential that editors and their peer reviewers are more demanding in exacting compliance with these rules.

Author's contribution
All listed authors planned and conducted the review, wrote the report and approved the final version.

Conflicts of interest
No financial, legal or political conflicts involving third parties (government, corporations and private foundations, etc.) have been declared for any aspect of the submitted work (including, but not limited to grants and funding, advisory board participation, study design, preparation of manuscript, statistical analysis, etc.).