External validation of Machine Learning models for COVID-19 detection based on Complete Blood Count

Andrea Campagner,Federico Cabitza,Anna Carobene

doi:10.1007/s13755-021-00167-3

Abstract

PurposeThe rRT-PCR for COVID-19 diagnosis is affected by long turnaround time, potential shortage of reagents, high false-negative rates and high costs. Routine hematochemical tests are a faster and less expensive alternative for diagnosis. Thus, Machine Learning (ML) has been applied to hematological parameters to develop diagnostic tools and help clinicians in promptly managing positive patients. However, few ML models have been externally validated, making their real-world applicability unclear.MethodsWe externally validate 6 state-of-the-art diagnostic ML models, based on Complete Blood Count (CBC) and trained on a dataset encompassing 816 COVID-19 positive cases. The external validation was performed based on two datasets, collected at two different hospitals in northern Italy and encompassing 163 and 104 COVID-19 positive cases, in terms of both error rate and calibration.Results and ConclusionWe report an average AUC of 95% and average Brier score of 0.11, out-performing existing ML methods, and showing good cross-site transportability. The best performing model (SVM) reported an average AUC of 97.5% (Sensitivity: 87.5%, Specificity: 94%), comparable with the performance of RT-PCR, and was also the best calibrated. The validated models can be useful in the early identification of potential COVID-19 patients, due to the rapid availability of CBC exams, and in multiple test settings.

Highlights

Since its initial spread in January 2020, the COVID-19 pandemic has so far affected more than 180 million people and caused more than 3 million deaths worldwide.The reverse polymerase chain reaction (PCR) and the reverse transcriptase-PCR are the gold standard tests for the detection of SARS-CoV-2 coronavirus, causative of COVID-19
In order to address this gap in the literature, and to extend the work presented in [5, 9], in this contribution we present the validation process of 6 Machine Learning (ML) models that are based on the complete Blood Count (CBC) data
In this article, we reported about the external validation of 6 state-of-the-art ML models for COVID-19 diagnosis based on routine hematochemical parameters

Summary

Introduction

The reverse polymerase chain reaction (PCR) and the reverse transcriptase-PCR (rRT-PCR) are the gold standard tests for the detection of SARS-CoV-2 coronavirus, causative of COVID-19 Both present known shortcomings such as long turnaround time, high costs, high false-negative rates (up to 15%) [12], the need for specialized equipment, and the associated shortage of reagents [13]. For these reasons, Machine Learning (ML) have been applied to hematological parameters [22, 27, 36] for a more rapid and cost-effective detection of the COVID-19 disease [13]. Even if we assume the performance of those models can be replicated [3], they are associated with much higher transaction costs than routine blood exams (including logistics and patient handling), and with lower safety, due to the high amount of radiation doses of CT procedures, and to the risk of contamination of the radiology suites [16]

Objectives

Methods

Results

Conclusion