Abstract 3373: Deep learning for circulating tumor cell (CTC) identification with the CELLSEARCH system: achieving top human-level performance

Abstract

Abstract Background: Enumeration of CTCs in blood samples with the CELLSEARCH® system is a prognostic biomarker in metastatic breast, prostate and colorectal cancer (full intended use documents.cellsearchctc.com). Currently, the final CTC identification is performed by human reviewers through visual assessment, which is a time-consuming procedure that could be affected by subjective interpretations. Modern Artificial Intelligence (AI) methods can provide a solution by automating the process of CTC identification, thus eliminating subjectivity and producing faster and more reproducible results. We have recently presented an automated algorithm (for research use only) based on deep learning models for cell segmentation and classification which allows the automated identification of CTCs in CELLSEARCH® fluorescent images (Ansaloni et al., EACR 2022). The purpose of this study was to assess inter-rater variability among human reviewers and to compare the AI performance against them. Methods: Blood samples derived from metastatic cancer patients were processed with CELLTRACKS® AUTOPREP®. The AI algorithm was trained on fluorescent images of 7255 CTCs and 32876 non-CTC events (training set) acquired by CELLTRACKS ANALYZER II® (CTAII) from 90 breast, 122 prostate and 54 colorectal cancer samples. 10 human reviewers, qualified for CTC image analysis, performed blind labeling of a separate dataset independently (test set: 55 samples taken from 27 breast and 28 prostate cancer). We studied inter-rater agreement and variability using Fleiss Kappa and Coefficient of Variation (CV). The labels provided by 5 experienced reviewers (average experience ≅ 15 years) were used to generate the ground truth (GT) by majority voting. The CTC detection performance of the AI was ranked against the other 5 reviewers (average experience ≅ 5 years). Results: In the test dataset, 2990 out of 9554 events were identified as CTCs by the GT review on CTAII images. In the task of classifying each event as CTC or non-CTC, the inter-rater agreement among the 10 human reviewers measured by Fleiss Kappa was 0.85. These classification results provided the number of CTCs per patient counted by each reviewer. The inter-rater variability on the CTC enumeration among the 10 reviewers was evaluated for each sample (median CV = 17.9% and interquartile range = 17.6% , excluding 27 samples with mode = 0 CTC). Measured against the GT classification, the AI performance (accuracy = 94.7%; F1 = 91.6%) was at the top of the human reviewers’ ranking (accuracy = 92.4 - 93.6 - 93.8 - 94.0 - 95.8%; F1 = 86.8 - 89.2 - 89.2 - 89.5 - 93.1%). Conclusion: The AI algorithm reached top-ranking performance in CTC identification, surpassing 4 out of 5 expert reviewers. These results show the applicability of AI to cell classification to remove human subjectivity from the review process and to maximize standardization among different research centers. Citation Format: Luca Biasiolli, Pietro Ansaloni, Nicolò Gentili, Ramona Miserendino, Chiara Rossi, Giulio Signorini, Gianni Medoro. Deep learning for circulating tumor cell (CTC) identification with the CELLSEARCH system: achieving top human-level performance [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 3373.