Chest Computed Tomography for Detection of Coronavirus Disease 2019 (COVID-19): Don't Rush the Science.

Abstract

Ideas and Opinions21 July 2020Chest Computed Tomography for Detection of Coronavirus Disease 2019 (COVID-19): Don't Rush the ScienceFREEMichael D. Hope, MD, Constantine A. Raptis, MD, and Travis S. Henry, MDMichael D. Hope, MDUniversity of California, San Francisco, and San Francisco Veterans Affairs Medical Center, San Francisco, California (M.D.H.)Search for more papers by this author, Constantine A. Raptis, MDMallinckrodt Institute of Radiology, Washington University School of Medicine in St. Louis, St. Louis, Missouri (C.A.R.)Search for more papers by this author, and Travis S. Henry, MDUniversity of California, San Francisco, San Francisco, California (T.S.H.)Search for more papers by this authorAuthor, Article, and Disclosure Informationhttps://doi.org/10.7326/M20-1382 SectionsAboutVisual AbstractPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail Amidst the coronavirus disease 2019 (COVID-19) pandemic, there is great pressure on physicians to provide clarity and answers. Good science, however, takes time and careful consideration to prove the value of advancements in diagnosis and treatment. We would like to share what we believe is a classic arc of events for a new imaging indication in the radiology literature: A rush to publish positive results leads to their overinterpretation and, consequently, the dissemination of premature conclusions with broad implications. Although this has occurred before with imaging, our recent experience is unique in that the implications are far-reaching and potentially of immediate importance.The shortage of rapid and highly sensitive reverse transcriptase polymerase chain reaction (RT-PCR) tests for the diagnosis of COVID-19 has led many in the health care community to consider a screening or diagnostic role for imaging. Publications from China during the outbreak there suggest a central role for computed tomography (CT). Fang and colleagues reported CT findings of pneumonia in 50 of 51 patients with RT-PCR–proven COVID-19 (1). Ai and colleagues then reported CT findings of pneumonia in 580 of 601 patients with RT-PCR–proven COVID-19 (2). Together, these publications ostensibly present a compelling story for CT, with sensitivities for the diagnosis of COVID-19 reported as 98% and 97%, respectively. Ai and colleagues concluded that chest CT may be used as a primary tool for detecting COVID-19 in epidemic areas.Concurrent with these publications, the journal that published them described an “ultra-rapid peer review” process (3). An expert panel was formed for review of the many manuscripts the journal received about COVID-19 imaging research, with the expectation that panel members would review articles within a 24-hour turnaround time. The commendable goal of this process is to publish key results as fast as possible. But does this ultra-rapid process allow enough time and consideration to ensure that only high-quality research is published?We believe that the answer is no (4, 5). In reviewing these 2 publications in detail, as well as others that support the use of CT for the diagnosis of COVID-19, we have found that many problems, such as faulty research design, incomplete methods sections with little description of likely biased patient cohorts, absence of a valid gold standard, multiple confounding variables, and scant discussion, limit the generalizability of the results and call into question the broad conclusions that are made. The findings of COVID-19 pneumonia that were used (for example, consolidation and ground-glass opacity) are not specific to the disease; rather, they are commonly seen in a range of infectious and noninfectious conditions. Consequently, positive CT results are only believable if the pretest probability of COVID-19 is high.Interestingly, a later publication attempted to show that COVID-19 can be differentiated from other viral pneumonias (6). Using 219 cases of COVID-19 pneumonia from China and 205 proven viral pneumonias (not COVID-19) from the United States, the authors asked blinded readers to score the cases as COVID-19 or not. They reported reasonable sensitivities and high specificities for both Chinese and U.S. radiologists and concluded that “Radiologists in China and the United States distinguished COVID-19 from viral pneumonia on chest CT with high specificity but moderate sensitivity.” On careful review, we found many methodological flaws (5). Clear differences between the Chinese and U.S. cohorts, which could be obvious by imaging and potentially guide a blinded reviewer, are present, including differences in age (45 vs. 65 years), prevalence of cardiovascular disease (12% vs. 60%), and possibly disease severity. In addition, important and common diseases with imaging appearances that overlap with COVID-19 pneumonia were not included. Moreover, the radiologist's gestalt, and not specific imaging findings, was used to “diagnose” COVID-19 pneumonia.We acknowledge that these are extraordinary times that place great pressure on the scientific community to produce answers and treatments. This is precisely why we need to rely on a thorough peer review process to scrutinize submissions and make sure that data are carefully collected, results are judiciously analyzed, and conclusions are fair and appropriate. We believe that a 24-hour turnaround time for peer review is likely not adequate.Although the intention of the literature promoting the use of CT for the diagnosis of COVID-19 is admirable—that is, faster diagnosis—it has caused confusion within the radiology community. One of the repercussions of using CT in the diagnosis of COVID-19, which is not discussed in the radiology literature, is that safely performing imaging is problematic. At the very least, droplet precautions with appropriate protective gear (now in short supply) need to be followed, CT scan rooms must be thoroughly cleaned, and the air needs to be recirculated given that COVID-19 is an airborne disease. Even if all protocols are followed, there is a risk that COVID-19 infection may be passed to other patients or staff in imaging departments. The American College of Radiology helped to resolve this confusion with guidelines for the use of imaging for suspected COVID-19 infection in mid-March (last updated March 22) (7). Their guidance is sound: “The findings on chest imaging in COVID-19 are not specific and overlap with other infections, including influenza, H1N1 [influenza], [severe acute respiratory syndrome], and [Middle East respiratory syndrome]” and “CT should not be used to screen for or as a first-line test to diagnose COVID-19.”This is a cautionary tale from the radiology community about the consequences of rushing the scientific review process. The best intentions can lead to unforeseen consequences. This may become more relevant as we push forward with potential treatments and vaccines for COVID-19.References1. Fang Y, Zhang H, Xie J, et al. Sensitivity of chest CT for COVID-19: comparison to RT-PCR. Radiology. 2020:200432. [PMID: 32073353] doi:10.1148/radiol.2020200432 CrossrefMedlineGoogle Scholar2. Ai T, Yang Z, Hou H, et al. Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases. Radiology. 2020:200642. [PMID: 32101510] doi:10.1148/radiol.2020200642 CrossrefMedlineGoogle Scholar3. Moy L, Bluemke D. The radiology scientific expert panel. Radiology. 2020:204005. [PMID: 32105563] doi:10.1148/radiol.2020204005 CrossrefMedlineGoogle Scholar4. Hope MD, Raptis CA, Shah A, et al; six signatories. A role for CT in COVID-19? What data really tell us so far [Letter]. Lancet. 2020. [PMID: 32224299] doi:10.1016/S0140-6736(20)30728-5 CrossrefMedlineGoogle Scholar5. Raptis CA, Hammer MM, Short RG, et al. Chest CT and COVID-19: a critical review of the literature to date. AJR Am J Roentgenol. 2020. [Forthcoming]. CrossrefGoogle Scholar6. Bai HX, Hsieh B, Xiong Z, et al. Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT. Radiology. 2020:200823. [PMID: 32155105] doi:10.1148/radiol.2020200823 CrossrefMedlineGoogle Scholar7. American College of Radiology. ACR recommendations for the use of chest radiography and computed tomography (CT) for suspected COVID-19 infection. Accessed at www.acr.org/Advocacy-and-Economics/ACR-Position-Statements/Recommendations-for-Chest-Radiography-and-CT-for-Suspected-COVID19-Infection on 5 April 2020. Google Scholar Comments0 CommentsSign In to Submit A Comment Harrison X. Bai, 2,3; Ben Hsieh,2; Wei-Hua Liao,11. Xiangya Hospital Central South University 2. Rhode Island Hospital 3. Brown University23 April 2020 Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT We thank Hope et al. for responding (1) to our comment (2). We agree with Hope et al. that the study by Bai et al. (3) has certain limitations in experimental design and the data presented but we disagree with the comment that "the conclusions lack clinical utility and are unjustified." First, diseases other than other pneumonia with overlapping imaging features with COVID-19 may not be difficult to distinguish for a radiologist on chest CT, if clinical history is given. For example, if a patient has acute lung injury or exposure to drugs, there is usually a clear history of that which gives the radiologist the clue to the diagnosis. The study by Bai et al. focused on the most difficult differential diagnosis and was designed to test diagnostic performance in this very specific context of a CT scan where the main differential diagnosis was pneumonia.Second, readers distinguished COVID-19 from other viral pneumonia "by gestalt" because that is how a radiologist evaluates these scans in real life. To properly control for these other confounding factors, Bai et al. age-matched the two groups in the test cohort where the 7 radiologists interpreted the study. The scan thickness was randomized between the two groups and other factors that could have biased the evaluation were eliminated to the best of ability. We agree that patients in the non-COVID group may have more aortic or coronary calcium than those in the COVID group as shown in the demographics table, but patients in the COVID group were also much older as well. It is possible that patients in the non-COVID group had more severe disease but we do not believe that disease severity is a factor that we should have controlled for (although it should have been evaluated for the non-COVID group) because 1. It is a natural reflection of how the two diseases present on chest CT 2. There is likely much overlap between the two groups on disease severity 3. The readers were blinded to this information as well so they could not base their evaluation solely on this.Finally, we disagree that the results presented in Table 5 are of little practical use. Although these differences shown by themselves are not "discriminating factors" per say and we did not attempt to build a model based on a combination of these factors that can distinguish COVID-19 from other viral pneumonia, the fact that these differences existed between the COVID and non-COVID group supports that there is something on CT that can be picked up by a radiologist "by gestalt", which is the usual way a radiologist evaluates these finding on CT when the main differential diagnosis is pneumonia.1. Hope MD, Raptis CA, Henry TS. Authors’ Response. In: Comments: Chest Computed Tomography for Detection of Coronavirus Disease 2019 (COVID-19): Don't Rush the Science [Internet]. Pennsylvania: Annals of Internal Medicine; 2020 April 15 [cited 2020 April 22].2. Bai HX, Hsieh B, Liao W. Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT. In: Comments: Chest Computed Tomography for Detection of Coronavirus Disease 2019 (COVID-19): Don't Rush the Science [Internet]. Pennsylvania: Annals of Internal Medicine; 2020 April 13 [cited 2020 April 22].3. Bai HX, Hsieh B, Xiong Z, Halsey K, Choi JW, Tran TML, et al. Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT. Radiology. 2020:200823-. Disclosures: Harrison X Bai, Ben Hsieh, and Wei-Hua Liao are authors of the manuscript “Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT” Michael D. Hope, MD,1,2; Constantine A. Raptis, MD,3; Travis S. Henry, MD,11. University of California San Francisco 2. San Francisco Veterans Affairs Medical Center 3. Washington University School of Medicine in Saint Louis15 April 2020 Authors' Response We thank Bai and colleagues for their comment on our recent article. Their attempt to study CT findings that are specific for COVID-19 is laudable, but we believe their conclusions lack clinical utility and are unjustified based on the experimental design and data presented. Determining specificity for a diagnostic test requires a cohort representative of the population where the test would be applied. In a retrospective study, this is best done with a cross-sectional cohort. Case-control studies, such as the one described by Bai et al., are limited for this purpose. They create a population with an overrepresentation of the disease being tested, and an artificial binary distinction between COVID-19 and other viral pneumonias. The exclusion of noninfectious diseases with overlapping imaging features with COVID-19 such acute lung injury or organizing pneumonia of other causes (e.g. drugs, exposure, cryptogenic) is another concern. Given these limitations, the specificity reported in this case-control design cannot be applied to a general population. Bai et al. provide no objective imaging criteria for how readers distinguished COVID-19 from other viral pneumonias. We are left to assume that this was done by gestalt, and readers clearly used different diagnostic strategies. For example, “Reviewer 3” called most cases COVID-19 (85% versus 40% and 43%). Furthermore, radiologists were blinded to the RT-PCR status, but not the overall goal of the study. Setting aside differences in age between the cohorts, other important clues could have been followed. Differences in scan parameters such as reconstruction field of view (supplemental table 2) could distinguish the patient populations. Imaging findings like aortic or coronary calcium may have been the discriminator; these findings reflect cardiovascular disease, which was five times more common in the non-COVID-19 cohort. Most importantly, what if disease severity were the discriminator? No data is provided about disease severity for the non-COVID-19 cohort. It is impossible to know if identifying COVID-19 was as straightforward as identifying more severe pneumonia. Even if these limitations are disregarded, the results presented in Table 5 are of little practical use. For example, what is the clinical applicability of the observation that ground glass opacity is seen in 91% versus 68% of patients with versus without COVID-19? Viral pneumonia commonly shows ground glass. This is not a discriminating factor. The paper offers no finding or constellation of findings to reliably distinguish COVID-19 from other pneumonias, or the myriad processes with similar imaging appearances. Harrison X Bai, Ben Hsieh, Wei-Hua LiaoRhode Island Hospital, Rhode Island Hospital, Xiangya Hospital Central South University14 April 2020 Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT We read with interest the Ideas and Opinions article by Hope et al. titled "Chest Computed Tomography for Detection of Coronavirus Disease 2019 (COVID-19): Don't Rush the Science" (1). We agree with the authors that chest CT should not be used in upfront diagnostic setting and the results of previously published studies should be interpreted with caution. However, we disagree with the authors' criticisms of paper by Bai et al. (2) The inherent differences in baseline characteristics between COVID-19 and non-COVID-19 pneumonia groups are consistent with several other studies (3, 4) and likely just reflect a younger patient population that COVID-19 is targeting. However, the two groups were age-matched in the test set interpreted by all seven radiologists. Since CT is not considered a first line diagnostic tool for COVID-19, one can argue that the focus should be on the differentiation between COVID-19 and other viral pneumonia since this is likely of the greatest clinical value. Finally, Bai et al. demonstrated many differences in specific imaging findings between the two groups in Table 5. COVID-19 patients were more likely to have a peripheral distribution of abnormal findings (80% vs. 57%, p<0.001), ground-glass opacity (91% vs. 68%, p<0.001), fine reticular opacity (56% vs. 22%, p<0.001), vascular thickening (59% vs. 22%, p<0.001) and reverse halo sign (11% vs. 1%, p=0.005) and less likely to have a central+peripheral distribution of abnormal findings (14% vs. 35%, p<0.001), air bronchogram (14% vs. 23%, p=0.014), pleural thickening (15 vs. 33%, p<0.001), pleural effusion (4 vs. 39%, p<0.001) and lymphadenopathy (2.7% vs. 10.2%, p<0.001) on chest CT when compared to the non-COVID-19 patients.1. Hope MD, Raptis CA, Henry TS. Chest Computed Tomography for Detection of Coronavirus Disease 2019 (COVID-19): Don't Rush the Science. Annals of Internal Medicine.2. Bai HX, Hsieh B, Xiong Z, Halsey K, Choi JW, Tran TML, et al. Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT. Radiology. 2020:200823-.3. Jain S, Self WH, Wunderink RG, Fakhran S, Balk R, Bramley AM, et al. Community-acquired pneumonia requiring hospitalization among US adults. New England Journal of Medicine. 2015;373(5):415-27.4. Shi H, Han X, Jiang N, Cao Y, Alwalid O, Gu J, et al. Radiological findings from 81 patients with COVID-19 pneumonia in Wuhan, China: a descriptive study. The Lancet Infectious Diseases. 2020. Disclosures: Harrison X Bai, Ben Hsieh, and Wei-Hua Liao are authors of the manuscript “Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT” Joseph Fraiman, MDLallie Kemp Regional Medical Center27 April 2020 Response to Author's Comments The authors of the original opinion piece are kind to reply. However their reference to a recent meta-analysis1 as proof of high sensitivity for the PCR test is revealing. Despite using flawed methodology as the backbone of their argument against CT, they appear not to have subjected their own citation to equivalent scrutiny, or perhaps any at all.This meta-analysis, theoretically evaluating the diagnostic performance of PCR for Covid-19, defines Covid-19 disease based on PCR testing—seriously. In a breathtaking model of tautological thinking, the review uses PCR testing as its own gold standard. How this passed peer review is unclear, but it explains why no individual study has ever found PCR sensitivity to be higher than roughly 70%, while this ‘meta-analysis’ somehow finds an aggregate sensitivity of 89%. In addition to pooling case reports, case series, and other studies where no reference standard is applied, the authors even ignore each study’s calculations for sensitivity, instead applying their own novel method (one that makes things simple: the PCR is right if it says so).To cite such a paper, and to quote their finding, after performing detailed critical appraisal as a means for questioning CT sensitivity, feels at best intellectually disingenuous, particularly while asserting “higher quality literature is needed.” Moreover, citing Chinese diagnostic criteria as support against CT fails the smell test. I invite anyone to examine the consensus position of Chinese professional societies on CT use for Covid-19.2A fundamental truth remains: the best available literature, flawed as it may be, shows PCR sensitivity is poor, CT sensitivity is better, and the combination of both is best—by far. We can journal club all we want. China flattened their curve using routine CT-plus-PCR for symptomatic patients. Here in the U.S. we can continue setting records for cases and fatalities from Covid-19, having swiftly overtaken the rest of the world, or we can admit our mistakes and move forward. Many, many American lives hang in the balance. 1) Kim H, Hong H, Yoon SH. Diagnostic Performance of CT and Reverse Transcriptase-Polymerase Chain Reaction for Coronavirus Disease 2019: A Meta-Analysis Radiology. 2020 Apr 17:201343. doi: 10.1148/radiol.2020201343.2) Yang Q, Liu Q, Xu H, et al. Imaging of Coronavirus Disease 2019: A Chinese Expert Consensus Statement [published online ahead of print, 2020 Apr 18]. Eur J Radiol. 2020;109008. doi:10.1016/j.ejrad.2020.109008 Michael D Hope, MD 1,2; Constantine A Raptis, MD 3; Amar Shah, MD, MPA 4; Mark M Hammer, MD 5; Travis S Henry, MD 11.University of California San Francisco 2.San Francisco Veterans Affairs Medical Center 3.Washington University School of Medicine in Saint Louis 4.Zucker School of Medicine at Hofstra/Northwell 5.Br24 April 2020 Authors' response to Fraiman We thank Dr. Fraiman for his impassioned and colorful response to our recent Ideas and Opinions piece. We thought it would be useful to share some recent data that relates to the issues raised. Recent meta-analysis shows that the sensitivity of reverse transcriptase-polymerase chain reaction (RT-PCR) is higher than originally reported in radiology literature: pooled sensitivity of 89%. Furthermore, the positive predictive value of RT-PCR is estimated to be over ten times that of CT in low prevalence countries (1). The National Health Care Commission of China has removed chest CT from the recent versions of their diagnostic criteria for COVID-19 (2). Major US medical centers in epidemic areas have also used CT more sparingly as they gain experience managing the disease (3,4). The New York University (NYU) experience is particularly telling: “Once one embraces that COVID-19 can look like anything and that CT findings may even be completely normal for the first several days of the symptomatic stage of infection, the value of imaging by CT approaches zero.” Academic literature is not just of academic value. It can have crucial policy implications. Our analysis shows that much of the early radiology literature regarding CT and COVID-19 is fundamentally flawed (5). Higher quality data is needed for informed decision making as we collectively learn how best to manage this ongoing pandemic.