Development and validation of the artificial intelligence (AI)-based diagnostic model for bronchial lumen identification

Abstract

Bronchoscopy is a key step in the diagnosis and treatment of respiratory diseases. However, the level of expertise varies among different bronchoscopists. Artificial intelligence (AI) may help them identify bronchial lumens. Thus, a bronchoscopy quality-control system based on AI was built to improve the performance of bronchoscopists. This single-center observational study consecutively collected bronchoscopy videos from Shanghai Chest Hospital and segmented each video into 31 different anatomical locations to develop an AI-assisted system based on a convolutional neural network (CNN) model. We then designed a single-center trial to compare the accuracy of lumen recognition by bronchoscopists with and without the assistance of the AI system. A total of 28,441 qualified images of bronchial lumen were used to train the CNNs. In the cross-validation set, the optimal accuracy of the six models was between 91.83% and 96.62%. In the test set, the visual geometry group 16 (VGG-16) achieved optimal performance with an accuracy of 91.88%, and an area under the curve of 0.995. In the clinical evaluation, the accuracy rate of the AI system alone was 54.30% (202/372). For the identification of bronchi except for segmental bronchi, the accuracy was 82.69% (129/156). In group 1, the recognition accuracy rates of doctors A, B, a and b alone were 42.47%, 34.68%, 28.76%, and 29.57%, respectively, but increased to 57.53%, 54.57%, 54.57%, and 46.24% respectively when combined with the AI system. Similarly, in group 2, the recognition accuracy rates of doctors C, D, c, and d were 37.90%, 41.40%, 30.91%, and 33.60% respectively, but increased to 51.61%, 47.85%, 53.49%, and 54.30% respectively, when combined with the AI system. Except for doctor D, the accuracy of doctors in recognizing lumen was significantly higher with AI assistance than without AI assistance, regardless of their experience (P<0.001). Our AI system could better recognize bronchial lumen and reduce differences in the operation levels of different bronchoscopists. It could be used to improve the quality of everyday bronchoscopies.