Detection and quantification of pulmonary embolism with artificial intelligence: The SFR 2022 artificial intelligence data challenge

Abstract

PurposeIn 2022, the French Society of Radiology together with the French Society of Thoracic Imaging and CentraleSupelec organized their 13th data challenge. The aim was to aid in the diagnosis of pulmonary embolism, by identifying the presence of pulmonary embolism and by estimating the ratio between right and left ventricular (RV/LV) diameters, and an arterial obstruction index (Qanadli's score) using artificial intelligence. Materials and methodsThe data challenge was composed of three tasks: the detection of pulmonary embolism, the RV/LV diameter ratio, and Qanadli's score. Sixteen centers all over France participated in the inclusion of the cases. A health data hosting certified web platform was established to facilitate the inclusion process of the anonymized CT examinations in compliance with general data protection regulation. CT pulmonary angiography images were collected. Each center provided the CT examinations with their annotations. A randomization process was established to pool the scans from different centers. Each team was required to have at least a radiologist, a data scientist, and an engineer. Data were provided in three batches to the teams, two for training and one for evaluation. The evaluation of the results was determined to rank the participants on the three tasks. ResultsA total of 1268 CT examinations were collected from the 16 centers following the inclusion criteria. The dataset was split into three batches of 310, 580 and 378 C T examinations provided to the participants respectively on September 5, 2022, October 7, 2022 and October 9, 2022. Seventy percent of the data from each center were used for training, and 30% for the evaluation. Seven teams with a total of 48 participants including data scientists, researchers, radiologists and engineering students were registered for participation. The metrics chosen for evaluation included areas under receiver operating characteristic curves, specificity and sensitivity for the classification task, and the coefficient of determination r2 for the regression tasks. The winning team achieved an overall score of 0.784. ConclusionThis multicenter study suggests that the use of artificial intelligence for the diagnosis of pulmonary embolism is possible on real data. Moreover, providing quantitative measures is mandatory for the interpretability of the results, and is of great aid to the radiologists especially in emergency settings.