Generalizability challenges of a machine learning model for classification of indeterminate biliary strictures

Abstract

We have read with great interest the article by Saraiva et al,1Saraiva M.M. Ribeiro T. Ferreira J.P.S. et al.Artificial intelligence for automatic diagnosis of biliary stricture malignancy status in single-operator cholangioscopy: a pilot study.Gastrointest Endosc. 2022; 95: 339-348Abstract Full Text Full Text PDF PubMed Scopus (6) Google Scholar which developed a deep learning algorithm for the classification of malignant and benign biliary strictures with the use of cholangioscopy images. We also agree with the important role that artificial intelligence can play in improving the characterization of indeterminate biliary strictures.2Ghandour B. Hsieh H.-W. Akshintala V. et al.Machine learning for classification of indeterminate biliary strictures during cholangioscopy.Am J Gastroenterol. 2021; 116: S1Crossref PubMed Google Scholar We would, however, appreciate some clarification regarding the methods used to develop the neural network, including the following:•Did the training of the neural network include hyperparameter selection? If yes, then what data were used for this purpose? If the same data partition was used for hyperparameter selection and for estimation of measures of performance, then the estimates will be overly optimistic.•What was the reason behind performing both a 5-fold and a split-sample cross-validation?•What method was used for estimating the confidence intervals? Furthermore, we believe that the reported estimates of area under the receiving operating characteristic curve—accuracy, sensitivity, and specificity of 0.988, 94.9%, 94.7%, and 92.1%, respectively—are overly optimistic. Despite patient-based grouping, where data from a single patient were restricted to a single fold, the authors did not consider the correlation between images from the same patient. When the test fold contains multiple images from the same patient, and when within-individual correlations are ignored in the estimation of measures of performance, the estimates will be overly optimistic. In conclusion, the study by Saraiva et al1Saraiva M.M. Ribeiro T. Ferreira J.P.S. et al.Artificial intelligence for automatic diagnosis of biliary stricture malignancy status in single-operator cholangioscopy: a pilot study.Gastrointest Endosc. 2022; 95: 339-348Abstract Full Text Full Text PDF PubMed Scopus (6) Google Scholar confirms the usefulness and value of machine learning in improving the clinical outcomes in patients with indeterminate biliary strictures through accurate diagnosis of their disease. We agree with the authors that further studies are required for external validation of the neural network developed in their work. Dr Khashab is a consultant for Boston Scientific, Olympus America, Medtronic, GI Supply, Apollo Endosurgery, and Pentax and the recipient of royalties from UpToDate and Elsevier. Dr Akshintala is a co-founder of Origin Endoscopy Inc. The other authors disclosed no financial relationships. Artificial intelligence for automatic diagnosis of biliary stricture malignancy status in single-operator cholangioscopy: a pilot studyGastrointestinal EndoscopyVol. 95Issue 2PreviewThe diagnosis and characterization of biliary strictures (BSs) is challenging. The introduction of digital single-operator cholangioscopy (DSOC) that allows direct visual inspection of the lesion and targeted biopsy sampling significantly improved the diagnostic yield in patients with indeterminate BSs. However, the diagnostic efficiency of DSOC remains suboptimal. Convolutional neural networks (CNNs) have shown great potential for the interpretation of medical images. We aimed to develop a CNN-based system for automatic detection of malignant BSs in DSOC images. Full-Text PDF ResponseGastrointestinal EndoscopyVol. 95Issue 6PreviewWe are extremely grateful to Ghandour et al1 for the important comments on our pilot study.2 Full-Text PDF