Assessment of the reporting quality of randomized controlled trials related to the pharmacotherapy of COVID-19 based on the CONSORT 2010 checklist: a systematic review

Abstract

In the coronavirus disease 2019 (COVID-19) pandemic, thousands of randomized controlled trials (RCTs) on COVID-19 treatment have been conducted over a short period; 6332 trials have been registered with clinicaltrials.gov to date [[1]National Institutes of HealthU.S. National Library of Medicine [Internet].https://clinicaltrials.gov/ct2/results?cond=COVID-19Date: 2021Google Scholar], but research on the reporting quality of COVID-19 RCTs has been insufficient. Incomplete reporting of RCTs can be harmful in clinical settings [[2]Ioannidis J.P. Greenland S. Hlatky M.A. Khoury M.J. Macleod M.R. Moher D. et al.Increasing value and reducing waste in research design, conduct, and analysis.Lancet. 2014; 383: 166-175Google Scholar], so assessing the reporting quality of RCTs related to pharmacotherapy of COVID-19 may be an important way to identify effective treatments and end the pandemic. However, to our knowledge, only one study has evaluated the reporting quality of RCTs related to COVID-19 interventions to date, and that study evaluated only 40 abstracts [[3]Wang D. Chen L. Wang L. Hua F. Li J. Li Y. et al.Abstracts for reports of randomised trials of COVID-19 interventions had low quality and high spin.J Clin Epidemiol. 2021; 139: 107-120Google Scholar]. Full reports of RCTs related to COVID-19 are freely available in public health databases and have information relevant to clinical therapeutic applications. Evaluations of the quality of full reports and identification of RCT characteristics are needed. This study aimed to evaluate the overall quality of full reports of RCTs related to pharmacotherapy of COVID-19 based on the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement [[4]Schulz K.F. Altman D.G. Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials.Ann Intern Med. 2010; 152: 726-732Google Scholar] and to identify factors associated with better reporting quality. The 10 704 studies identified from PubMed, the National Institutes of Health, EMBASE, Cochrane, and the Web of Science were searched for RCTs published from inception to 31 December, 2020, and 87 RCTs were included in the final analysis (Fig. 1A, Table S1). Details about the included RCTs and CONSORT scoring are summarized in Tables S2–S4.Fig. 1(A) Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram of the study selection process. We performed a database search using five search engines to identify all COVID-19 intervention RCTs from inception to 31 December, 2020, using search terms such as “COVID-19,” “2019 nCoV,” “Coronavirus Disease 19,” “SARS-CoV-2,” “Randomised controlled trial,” “Controlled Clinical Trial,” and “Comparative study”. All RCTs involving human subjects related to the treatment of COVID-19 published in English were included. RCTs not directly related to COVID-19 treatment (e.g. for the purpose of prophylaxis, treatment of complications of COVID-19), other study designs (e.g. nonrandomized clinical trials, observation studies, research letters, brief reports, and protocols), and studies not involving humans (e.g. diagnostic test accuracy, laboratory studies) were excluded. (B) The adherence rate (%) of 87 RCTs for each item of the CONSORT 2010 checklist. The CONSORT 2010 checklist includes 25 items (37 subitems). The checklist items pertain to the content of the title and abstract, introduction, methods, results, discussion, and other information. (C) Factors associated with overall reporting quality score in 87 RCTs. Linear regression analyses were performed to identify potential reporting quality factors. Univariable analyses were performed first, and then all factors without multicollinearity were entered into multivariable analyses. During multivariable modelling, variable selection was performed through a backward process based on the Akaike information criterion value. For the multivariable analyses, adjusted R2 = 0.57; p < .001. CI, confidence interval; CONSORT, Consolidated Standards of Reporting Trials; COVID-19, coronavirus disease 2019; RCT, randomized controlled trial.View Large Image Figure ViewerDownload Hi-res image Download (PPT)Fig. 1(A) Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram of the study selection process. We performed a database search using five search engines to identify all COVID-19 intervention RCTs from inception to 31 December, 2020, using search terms such as “COVID-19,” “2019 nCoV,” “Coronavirus Disease 19,” “SARS-CoV-2,” “Randomised controlled trial,” “Controlled Clinical Trial,” and “Comparative study”. All RCTs involving human subjects related to the treatment of COVID-19 published in English were included. RCTs not directly related to COVID-19 treatment (e.g. for the purpose of prophylaxis, treatment of complications of COVID-19), other study designs (e.g. nonrandomized clinical trials, observation studies, research letters, brief reports, and protocols), and studies not involving humans (e.g. diagnostic test accuracy, laboratory studies) were excluded. (B) The adherence rate (%) of 87 RCTs for each item of the CONSORT 2010 checklist. The CONSORT 2010 checklist includes 25 items (37 subitems). The checklist items pertain to the content of the title and abstract, introduction, methods, results, discussion, and other information. (C) Factors associated with overall reporting quality score in 87 RCTs. Linear regression analyses were performed to identify potential reporting quality factors. Univariable analyses were performed first, and then all factors without multicollinearity were entered into multivariable analyses. During multivariable modelling, variable selection was performed through a backward process based on the Akaike information criterion value. For the multivariable analyses, adjusted R2 = 0.57; p < .001. CI, confidence interval; CONSORT, Consolidated Standards of Reporting Trials; COVID-19, coronavirus disease 2019; RCT, randomized controlled trial.View Large Image Figure ViewerDownload Hi-res image Download (PPT) The average overall reporting quality score (OQS) of the 87 RCTs was 19.4 of 25 (range, 12–24.5; 95% confidence interval (CI), 18.3–20.4). The intraclass correlation coefficients for the scoring of interobserver agreement for the OQS was 0.96. The mean CONSORT adherence rate, 77.4% (95% CI, 69.4%–85.4%), was similar to or higher than that of other RCTs conducted before the COVID-19 pandemic [[5]Ziogas D.C. Zintzaras E. Analysis of the quality of reporting of randomized controlled trials in acute and chronic myeloid leukemia, and myelodysplastic syndromes as governed by the CONSORT statement.Ann Epidemiol. 2009; 19: 494-500Google Scholar,[6]Ngah V.D. Mazingisa A.V. Zunza M. Wiysonge C.S. A review of adherence and predictors of adherence to the CONSORT statement in the reporting of tuberculosis vaccine trials.Vaccines. 2020; 8: 770Google Scholar]. Of the 37 checklist items, 25 were addressed in ≥75% (Fig. 1B). The methods domain rate was lower (69.7%), consistent with previous findings [7Beneki E. Vrysis C. Zintzaras E. Doxani C. Analysis of the quality of reporting of randomized controlled trials in anticoagulant versus antiplatelet medication for venous thromboembolism prophylaxis as governed by the CONSORT statement.J Thromb Thrombolysis. 2021; 52: 138-147Google Scholar, 8Mozetic V. Leonel L. Leite Pacheco R. de Oliveira Cruz Latorraca C. Guimarães T. Logullo P. et al.Reporting quality and adherence of randomized controlled trials about statins and/or fibrates for diabetic retinopathy to the CONSORT checklist.Trials. 2019; 20: 729Google Scholar, 9Rikos D. Dardiotis E. Aloizou A.M. Siokas V. Zintzaras E. Hadjigeorgiou G.M. Reporting quality of randomized controlled trials in restless legs syndrome based on the CONSORT statement.Tremor Other Hyperkinet Mov (N Y). 2019; 9Google Scholar], but it is important to report methodological items for validity and applicability of trial results. To identify the factors associated with reporting quality, linear regression modelling revealed that 5 years of journal impact factors (IFs) between 10 and 50, higher abstract word count, and sample size >100 were associated with better reporting quality, whereas first author from an Africa region and more than one primary outcome were associated with poor reporting quality. The other categories were not associated with OQS (Fig. 1C). This study showed that journals with an IF between 10 and 50 (β = 1.6; 95% CI, 0.05–3.14) were associated with better OQS, whereas an IF > 50 was not significant. This finding is consistent with that by Hays et al. [[10]Hays M. Andrews M. Wilson R. Callender D. O’Malley P.G. Douglas K. Reporting quality of randomized controlled trial abstracts among high-impact general medical journals: a review and analysis.BMJ Open. 2016; 6e011082Google Scholar], suggesting that reviewers may be aware of the limited impact of journal IF on reporting quality when evaluating RCT results. A higher abstract word count (β = 0.01; 95% CI, 0.004–0.02) was associated with better OQS. We noted that abstract word count was associated with better reporting quality, not only in the abstract but also in the main text of RCTs, which may be helpful when evaluating RCTs with limited time and resources. Song et al. suggested that authors need a sufficient number of words to correctly report all necessary information [[11]Song S.Y. Kim B. Kim I. Kim S. Kwon M. Han C. et al.Assessing reporting quality of randomized controlled trial abstracts in psychiatry: adherence to CONSORT for abstracts: a systematic review.PLoS One. 2017; 12e0187807Google Scholar]. Sample size >100 (β = 2.13; 95% CI, 0.59–3.67) was associated with better OQS, suggesting that the more patients enrolled, the higher the reporting quality of RCTs [[12]Tardy M.P. Gal J. Chamorey E. Almairac F. Vandenbos F. Bondiau P.Y. et al.Quality of randomized controlled trials reporting in the treatment of adult patients with high-grade gliomas.Oncologist. 2018; 23: 337-345Google Scholar]. RCTs with more than one primary outcome (β = −1.32; 95% CI, −2.5 to −0.15) were associated with lower OQS, which may be because inclusion of more than one primary outcome in a study makes it difficult for authors to comprehensively describe all outcomes owing to space limitations or is likely to yield only one statistically significant outcome among several outcomes. There were some limitations. First, only RCTs published in English were included. However, five public databases were the most used among clinical researchers. Second, the not applicable (N/A) category was not used, which could have led to falsely low scores for some items. However, it was difficult to distinguish between N/A or incomplete reporting, so evaluating the quality of reporting without using N/A could be more objective. In summary, the reporting quality of RCTs related to COVID-19 pharmacotherapy was found to be adequate, except for some optional items, even though the RCTs have been conducted over a limited period. This finding suggests that health care providers have been making efforts to share as much knowledge and validation of results related to COVID-19 treatment as possible, even in the presence of high workloads while treating patients to overcome the COVID-19 pandemic as soon as possible. This study may provide a standard for the selection related to RCTs of COVID-19 to guide clinical practice and public policy choices designed to end the pandemic. Institutional review board approval was waived at the Veterans Health Service Medical Center (BOHUN 2021-05-032). Y.J., Y.O., S.P., H.J., and E.K. contributed to the conception and design of the study and acquisition, analysis, or interpretation of data. Y.J., Y.O., and E.K. drafted the manuscript. Y.J. and Y.O. share the first authorship. Critical revision of the manuscript for important intellectual content was by Y.J., Y.O., S.P., H.J., and E.K. Supervision was by E.K. All authors meet the criteria for authorship. All authors declare that they have no competing interests and no financial relationships with any organization that might have an interest in this work. This work was funded by a project of the Veterans Health Service , Veterans Medical Research Institute , Veterans Health Service Medical Center, Seoul, South Korea (grant number VHSMC 21044 ), by a grant from the Korean government, South Korea (Ministry of Science and ICT, MICT; NRF-2021R1F1A1062044 ), and by the Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Education, South Korea (Grant Number 2021R1A6A1A03044296 ). The funder had no role in the trial design, data collection, data interpretation, or report preparation. Youn-Joo Jung, Yunkyoung Oh, and EunYoung Kim had full access to all the data in the study and take responsibility for the integrity of the data and accuracy of the data analysis. The authors thank Soohyun Yang, MD, PhD; Young Lee; and Jaeok Park of the Veterans Health Service, Veterans Medical Research Institute, Veterans Health Service Medical Center, Seoul, South Korea. The following are the Supplementary data to this article: Download .docx (.04 MB) Help with docx files Multimedia component 1 Download .docx (.06 MB) Help with docx files Multimedia component 2 Download .docx (.04 MB) Help with docx files Multimedia component 3 Download .docx (.03 MB) Help with docx files Multimedia component 4