Artificial Intelligence Diagnostic System Research Articles

Development of an Artificial Intelligence Diagnostic System Using Linked Color Imaging for Barrett's Esophagus.

Background: Barrett's esophagus and esophageal adenocarcinoma cases are increasing as gastroesophageal reflux disease increases. Using artificial intelligence (AI) and linked color imaging (LCI), our aim was to establish a method of diagnosis for short-segment Barrett's esophagus (SSBE). Methods: We retrospectively selected 624 consecutive patients in total at our hospital, treated between May 2017 and March 2020, who experienced an esophagogastroduodenoscopy with white light imaging (WLI) and LCI. Images were randomly chosen as data for learning from WLI: 542 (SSBE+/- 348/194) of 696 (SSBE+/- 444/252); and LCI: 643 (SSBE+/- 446/197) of 805 (SSBE+/- 543/262). Using a Vision Transformer (Vit-B/16-384) to diagnose SSBE, we established two AI systems for WLI and LCI. Finally, 126 WLI (SSBE+/- 77/49) and 137 LCI (SSBE+/- 81/56) images were used for verification purposes. The accuracy of six endoscopists in making diagnoses was compared to that of AI. Results: Study participants were 68.2 ± 12.3 years, M/F 330/294, SSBE+/- 409/215. The accuracy/sensitivity/specificity (%) of AI were 84.1/89.6/75.5 for WLI and 90.5/90.1/91.1/for LCI, and those of experts and trainees were 88.6/88.7/88.4, 85.7/87.0/83.7 for WLI and 93.4/92.6/94.6, 84.7/88.1/79.8 for LCI, respectively. Conclusions: Using AI to diagnose SSBE was similar in accuracy to using a specialist. Our finding may aid the diagnosis of SSBE in the clinic.

Diagnostic decisions of specialist optometrists exposed to ambiguous deep-learning outputs

Artificial intelligence (AI) has great potential in ophthalmology. We investigated how ambiguous outputs from an AI diagnostic support system (AI-DSS) affected diagnostic responses from optometrists when assessing cases of suspected retinal disease. Thirty optometrists (15 more experienced, 15 less) assessed 30 clinical cases. For ten, participants saw an optical coherence tomography (OCT) scan, basic clinical information and retinal photography (‘no AI’). For another ten, they were also given AI-generated OCT-based probabilistic diagnoses (‘AI diagnosis’); and for ten, both AI-diagnosis and AI-generated OCT segmentations (‘AI diagnosis + segmentation’) were provided. Cases were matched across the three types of presentation and were selected to include 40% ambiguous and 20% incorrect AI outputs. Optometrist diagnostic agreement with the predefined reference standard was lowest for ‘AI diagnosis + segmentation’ (204/300, 68%) compared to ‘AI diagnosis’ (224/300, 75% p = 0.010), and ‘no Al’ (242/300, 81%, p = < 0.001). Agreement with AI diagnosis consistent with the reference standard decreased (174/210 vs 199/210, p = 0.003), but participants trusted the AI more (p = 0.029) with segmentations. Practitioner experience did not affect diagnostic responses (p = 0.24). More experienced participants were more confident (p = 0.012) and trusted the AI less (p = 0.038). Our findings also highlight issues around reference standard definition.

Role of the artificial intelligence in the management of T1 colorectal cancer

Approximately 10% of submucosal invasive (T1) colorectal cancers demonstrate extraintestinal lymph node metastasis, necessitating surgical intervention with lymph node dissection. The ability to identify T1b (submucosal invasion depth ≥ 1000 µm) as a risk factor for lymph node metastasis via pre-treatment endoscopy is crucial in guiding treatment strategies. Accurately distinguishing T1b from T1a (submucosal invasion depth < 1000 µm) or dysplasia remains a significant challenge for artificial intelligence (AI) systems, which require high and consistent diagnostic capabilities. Moreover, as endoscopic therapies like endoscopic full-thickness resection and endoscopic intermuscular dissection evolve, and the focus on reducing unnecessary surgeries intensifies, the initial management of T1 colorectal cancers via endoscopic treatment is anticipated to increase. Consequently, the development of highly accurate and reliable AI systems is essential, not only for pre-treatment depth assessment but also for post-treatment risk stratification of lymph node metastasis. While such AI diagnostic systems are still under development, significant advancements are expected in the near future to improve decision-making in T1 colorectal cancer management.

Value of CT examination combined with serum tumor markers assisted with artificial intelligence diagnostic system in the diagnosis of benign and malignant pulmonary nodules

Value of CT examination combined with serum tumor markers assisted with artificial intelligence diagnostic system in the diagnosis of benign and malignant pulmonary nodules

Real-time detection of laryngopharyngeal cancer using an artificial intelligence-assisted system with multimodal data

BackgroundLaryngopharyngeal cancer (LPC) includes laryngeal and hypopharyngeal cancer, whose early diagnosis can significantly improve the prognosis and quality of life of patients. Pathological biopsy of suspicious cancerous tissue under the guidance of laryngoscopy is the gold standard for diagnosing LPC. However, this subjective examination largely depends on the skills and experience of laryngologists, which increases the possibility of missed diagnoses and repeated unnecessary biopsies. We aimed to develop and validate a deep convolutional neural network-based Laryngopharyngeal Artificial Intelligence Diagnostic System (LPAIDS) for real-time automatically identifying LPC in both laryngoscopy white-light imaging (WLI) and narrow-band imaging (NBI) images to improve the diagnostic accuracy of LPC by reducing diagnostic variation among on-expert laryngologists.MethodsAll 31,543 laryngoscopic images from 2382 patients were categorised into training, verification, and test sets to develop, validate, and internal test LPAIDS. Another 25,063 images from five other hospitals were used as external tests. Overall, 551 videos were used to evaluate the real-time performance of the system, and 200 randomly selected videos were used to compare the diagnostic performance of the LPAIDS with that of laryngologists. Two deep-learning models using either WLI (model W) or NBI (model N) images were constructed to compare with LPAIDS.ResultsLPAIDS had a higher diagnostic performance than models W and N, with accuracies of 0·956 and 0·949 in the internal image and video tests, respectively. The robustness and stability of LPAIDS were validated in external sets with the area under the receiver operating characteristic curve values of 0·965–0·987. In the laryngologist-machine competition, LPAIDS achieved an accuracy of 0·940, which was comparable to expert laryngologists and outperformed other laryngologists with varying qualifications.ConclusionsLPAIDS provided high accuracy and stability in detecting LPC in real-time, which showed great potential for using LPAIDS to improve the diagnostic accuracy of LPC by reducing diagnostic variation among on-expert laryngologists.

Artificial intelligence system-based histogram analysis of computed tomography features to predict tumor invasiveness of ground-glass nodules.

The use of an artificial intelligence (AI)-based diagnostic system can significantly aid in analyzing the histogram of pulmonary nodules. The aim of our study was to evaluate the value of computed tomography (CT) histogram indicators analyzed by AI in predicting the tumor invasiveness of ground-glass nodules (GGNs) and to determine the added value of contrast-enhanced CT (CECT) compared with nonenhanced CT (NECT) in this prediction. This study enrolled patients with persistent GGNs who underwent preoperative NECT and CECT scanning. AI-based histogram analysis was performed for pathologically confirmed GGNs, which was followed by screening invasiveness-related factors via univariable analysis. Multivariable logistic models were developed based on candidate CT histogram indicators measured on either NECT or CECT. Receiver operating characteristic (ROC) curve and precision-recall (PR) curve were used to evaluate the models' performance. A total of 116 patients comprising 121 GGNs were included and divided into the precancerous lesion and adenocarcinoma groups based on invasiveness. In the AI-based histogram analysis, the mean CT value [NECT: odds ratio (OR) =1.009; 95% confidence interval (CI): 1.004-1.013; P<0.001] and solid component volume (NECT: OR =1.005; 95% CI: 1.000-1.010; P=0.032) were associated with the adenocarcinoma and used for multivariable logistic modeling. The area under ROC curve (AUC) and PR curve (AUPR) were not significantly different between the NECT model (AUC =0.765, 95% CI: 0.679-0.837; AUPR =0.907, 95% CI: 0.825-0.953) and the optimal CECT model (delayed phase: AUC =0.772, 95% CI: 0.687-0.843; AUPR =0.895, 95% CI: 0.812-0.944). No significantly different metrics were observed between the NECT and CECT models (precision: 0.707 vs. 0.742; P=0.616). The AI diagnostic system can help in the diagnosis of GGNs. The system displayed decent performance in GGN detection and alert to malignancy. Mean CT value and solid component volume were independent predictors of tumor invasiveness. CECT provided no additional improvement in diagnostic performance as compared with NECT.

US of thyroid nodules: can AI-assisted diagnostic system compete with fine needle aspiration?

Artificial intelligence (AI) systems can diagnose thyroid nodules with similar or better performance than radiologists. Little is known about how this performance compares with that achieved through fine needle aspiration (FNA). This study aims to compare the diagnostic yields of FNA cytopathology alone and combined with BRAFV600E mutation analysis and an AI diagnostic system. The ultrasound images of 637 thyroid nodules were collected in three hospitals. The diagnostic efficacies of an AI diagnostic system, FNA-based cytopathology, and BRAFV600E mutation analysis were evaluated in terms of sensitivity, specificity, accuracy, and the κ coefficient with respect to the gold standard, defined by postsurgical pathology and consistent benign outcomes from two combined FNA and mutation analysis examinations performed with a half-year interval. The malignancy threshold for the AI system was selected according to the Youden index from a retrospective cohort of 346 nodules and then applied to a prospective cohort of 291 nodules. The combination of FNA cytopathology according to the Bethesda criteria and BRAFV600E mutation analysis showed no significant difference from the AI system in terms of accuracy for either cohort in our multicenter study. In addition, for 45 included indeterminate Bethesda category III and IV nodules, the accuracy, sensitivity, and specificity of the AI system were 84.44%, 95.45%, and 73.91%, respectively. The AI diagnostic system showed similar diagnostic performance to FNA cytopathology combined with BRAFV600E mutation analysis. Given its advantages in terms of operability, time efficiency, non-invasiveness, and the wide availability of ultrasonography, it provides a new alternative for thyroid nodule diagnosis. Thyroid ultrasonic artificial intelligence shows statistically equivalent performance for thyroid nodule diagnosis to FNA cytopathology combined with BRAFV600E mutation analysis. It can be widely applied in hospitals and clinics to assist radiologists in thyroid nodule screening and is expected to reduce the need for relatively invasive FNA biopsies. • In a retrospective cohort of 346 nodules, the evaluated artificial intelligence (AI) system did not significantly differ from fine needle aspiration (FNA) cytopathology alone and combined with gene mutation analysis in accuracy. • In a prospective multicenter cohort of 291 nodules, the accuracy of the AI diagnostic system was not significantly different from that of FNA cytopathology either alone or combined with gene mutation analysis. • For 45 indeterminate Bethesda category III and IV nodules, the AI system did not perform significantly differently from BRAFV600E mutation analysis.

Preoperative prediction of sinonasal papilloma by artificial intelligence using nasal video endoscopy: a retrospective study

Sinonasal inverted papilloma (IP) is at risk of recurrence and malignancy, and early diagnosis using nasal endoscopy is essential. We thus developed a diagnostic system using artificial intelligence (AI) to identify nasal sinus papilloma. Endoscopic surgery videos of 53 patients undergoing endoscopic sinus surgery were edited to train and evaluate deep neural network models and then a diagnostic system was developed. The correct diagnosis rate based on visual examination by otolaryngologists was also evaluated using the same videos and compared with that of the AI diagnostic system patients. Main outcomes evaluated included the percentage of correct diagnoses compared to AI diagnosis and the correct diagnosis rate for otolaryngologists based on years of practice experience. The diagnostic system had an area under the curve of 0.874, accuracy of 0.843, false positive rate of 0.124, and false negative rate of 0.191. The average correct diagnosis rate among otolaryngologists was 69.4%, indicating that the AI was highly accurate. Evidently, although the number of cases was small, a highly accurate diagnostic system was created. Future studies with larger samples to improve the accuracy of the system and expand the range of diseases that can be detected for more clinical applications are warranted.

Artificial intelligence for the diagnosis of clinically significant prostate cancer based on multimodal data: a multicenter study

BackgroundThe introduction of multiparameter MRI and novel biomarkers has greatly improved the prediction of clinically significant prostate cancer (csPCa). However, decision-making regarding prostate biopsy and prebiopsy examinations is still difficult. We aimed to establish a quick and economic tool to improve the detection of csPCa based on routinely performed clinical examinations through an automated machine learning platform (AutoML).MethodsThis study included a multicenter retrospective cohort and two prospective cohorts with 4747 cases from 9 hospitals across China. The multimodal data, including demographics, clinical characteristics, laboratory tests, and ultrasound reports, of consecutive participants were retrieved using extract-transform-load tools. AutoML was applied to explore potential data processing patterns and the most suitable algorithm to build the Prostate Cancer Artificial Intelligence Diagnostic System (PCAIDS). The diagnostic performance was determined by the receiver operating characteristic curve (ROC) for discriminating csPCa from insignificant prostate cancer (PCa) and benign disease. The clinical utility was evaluated by decision curve analysis (DCA) and waterfall plots.ResultsThe random forest algorithm was applied in the feature selection, and the AutoML algorithm was applied for model establishment. The area under the curve (AUC) value in identifying csPCa was 0.853 in the training cohort, 0.820 in the validation cohort, 0.807 in the Changhai prospective cohort, and 0.850 in the Zhongda prospective cohort. DCA showed that the PCAIDS was superior to PSA or fPSA/tPSA for diagnosing csPCa with a higher net benefit for all threshold probabilities in all cohorts. Setting a fixed sensitivity of 95%, a total of 32.2%, 17.6%, and 26.3% of unnecessary biopsies could be avoided with less than 5% of csPCa missed in the validation cohort, Changhai and Zhongda prospective cohorts, respectively.ConclusionsThe PCAIDS was an effective tool to inform decision-making regarding the need for prostate biopsy and prebiopsy examinations such as mpMRI. Further prospective and international studies are warranted to validate the findings of this study.Trial registrationChinese Clinical Trial Registry ChiCTR2100048428. Registered on 06 July 2021.

Characteristics of uroflowmetry patterns in men with detrusor underactivity revealed by artificial intelligence.

To elucidate the characteristics of uroflowmetry (UFM) observed in men with detrusor underactivity (DU) using our developed artificial intelligence (AI) diagnostic algorithm to distinguish between DU and bladder outlet obstruction (BOO). Subjective and objective parameters, including four UFM parameters (first peak flow rate, time to first peak, gradient to first peak, and the ratio of first peak flow rate to maximum flow rate [Qmax ]) selected by analyzing the judgment basis of the AI diagnostic system, were compared in 266 treatment-naive men with lower urinary tract symptoms (LUTS). Patients were divided into the DU (70; 26.32%) and non-DU (196; 73.68%) groups, and the UFM parameters for predicting the presence of DU were determined by multivariate analysis and receiver operating characteristic (ROC) curve analysis. Detrusor underactivity was defined as a bladder contractility index <100 and a BOO index <40. Most parameters on the first peak flow of UFM were significantly lower in the DU group. On multivariate analysis, lower first peak flow rate and lower ratio of first peak flow rate to Qmax were significant parameters to predict DU. In the ROC analysis, the ratio of the first peak flow rate to Qmax showed the highest area under the curve (0.848) and yielded sensitivities of 76% and specificities of 83% for DU diagnosis, with cutoff values of 0.8. Parameters on the first peak flow of UFM, especially the ratio of the first peak flow rate to Qmax , can diagnose DU with high accuracy in men with LUTS.

Abstract P1-05-06: Establishment of the breast ultrasound support system using deep-learning system

Abstract Although the categorization of ultrasound using the Breast Imaging Reporting and Data System (BI-RADS) has become widespread worldwide, the problem of inter-observer variability remains. To maintain uniformity in diagnostic accuracy, we have developed a novel artificial intelligence (AI) system in which AI can distinguish whether a static image obtained using a breast ultrasound represents BI-RADS3 or lower, or BI-RADS4a or higher, to determine the medical management that should be performed on a patient whose breast ultrasound shows abnormalities. To establish and validate the AI system, a training dataset consisting of 4,028 images containing 5,014 lesions and a test dataset consisting of 3,166 images containing 3,656 lesions were collected and annotated. We selected a setting that maximized the area under the curve (AUC) and minimized the difference in sensitivity and specificity by adjusting the internal parameters of the AI system, achieving an AUC, sensitivity, and specificity of 0.95, 90.0%, and 88.5%, respectively. Furthermore, based on 30 images extracted from the test data, the diagnostic accuracy of 20 clinicians and the AI system was compared, and the AI system was found to be significantly superior to the clinicians (McNemar test, p &amp;lt; 0.001). Then, we conducted a trial to introduce the system for use in clinical practice. Physicians reviewed the images and determined whether they were BI-RADS3 or lower, or BI-RADS4a or higher. Next, the classification was performed again for the same images concerning the AI diagnosis. At this time, the initial judgment was allowed to be overturned. We checked whether there was any difference in the diagnostic accuracy, sensitivity, and specificity before and after reviewing to the AI diagnosis. Reviews by 24 physicians were evaluated: 4 Japanese Breast Cancer Society breast specialists, 5 non-specialists and physicians with experience treating more than 40 cases of breast cancer, and 15 non-specialists and physicians with no experience treating more than 40 cases of breast cancer. The average rate of accuracy before confirming the AI diagnosis increased to 73.1% after confirming the AI diagnosis (p=0.00548), compared to 69.3% on average before the AI diagnosis. Compared to practice experience, the accuracy increased from an average of 77.1% to 79.6% for the 9 physicians who were breast specialists or who had treated 40 or more cases of breast cancer. For the 15 physicians with less than 40 breast cancer cases, the average rate of accuracy increased from 64.7% to 69.2%. Furthermore, sensitivity increased significantly to an average of 99.7% after reviewing of the AI diagnosis from an average of 88.8% prior to reviewing the AI-diagnosis.(p&amp;lt; 0.01). Specificity increased from an average of 62.4% to 63.8% (p=0.433) after reviewing AI diagnosis. We showed that our AI system, when applied to clinical practice and used by physicians, contributes to the improvement of diagnostic accuracy. Our results indicated that our AI diagnostic system was sufficiently accurate to be used in the clinical practice. Citation Format: Erina Odani, Tetsu Hayashida, masayuki kikuchi, Aiko Nagayama, tomoko seki, maiko takahashi, Akiko Matsumoto, Takeshi Murata, Rurina Watanuki, Takamichi Yokoe, Ayako Nakashoji, Hinako Maeda, Tatsuya Onishi, Sota Asaga, Takashi Hojo, Hiromitsu Jinno, Keiichi Sotome, Akira Matsui, Akihiko Suto, Shigeru Imoto, Yuko Kitagawa. Establishment of the breast ultrasound support system using deep-learning system [abstract]. In: Proceedings of the 2022 San Antonio Breast Cancer Symposium; 2022 Dec 6-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2023;83(5 Suppl):Abstract nr P1-05-06.

The combination of CD138/MUM1 dual-staining and artificial intelligence for plasma cell counting in the diagnosis of chronic endometritis.

To investigate the utility of combination of CD138/MUM1 dual-staining (DS) and artificial intelligence (AI) for plasma cell (PC) counting in the diagnosis of chronic endometritis (CE). Two hundred ninety-eight infertile women underwent endometrial biopsy were included. In 100 women, three successivesections were cut from each paraffin-embedded tissue block for CD138 immunohistochemical (IHC) single-staining (SS), MUM1 SS and CD138/MUM1 DS. The prevalence of CE and the sensitivity/specificity in the diagnosis of CE with different methods was studied. These sections diagnosed as CE with DS were collected to train artificial intelligence (AI) diagnostic system. In other 198 women, their tissue sections stained with CD138/MUM1 DS were used to test the AI system in the diagnosis of CE. CD138/MUM1 DS revealed that the cell membranes and nuclei of PCs were simultaneously labelled by CD138 and MUM1, respectively. The positive rate of ECs identified by CD138/MUM1 DS (38%, 38/100) was lower than CD138 SS (52%, 52/100) and MUM1 SS (62%, 62/100) (p<.05). The sensitivity, specificity and accuracy of CD138/MUM1 DS in the diagnosis of ECs reached 100%. The sensitivity, specificity and accuracy rates of AI diagnostic system of ECs were 100%, 83.3% and 91.4%, respectively. The 17 cases over-diagnosed as EC with the AI were corrected quickly by pathologists reviewing these false PC pictures listed by the AI. The combination of CD138/MUM1 DS and AI is a promising method to improve the accuracy and efficiency of CE diagnosis.

Improving the efficiency of identifying malignant pulmonary nodules before surgery via a combination of artificial intelligence CT image recognition and serum autoantibodies.

To construct a new pulmonary nodule diagnostic model with high diagnostic efficiency, non-invasive and simple to measure. This study included 424 patients with radioactive pulmonary nodules who underwent preoperative 7-autoantibody (7-AAB) panel testing, CT-based AI diagnosis, and pathological diagnosis by surgical resection. The patients were randomly divided into a training set(n = 212) and avalidation set(n = 212). The nomogram was developed through forward stepwise logistic regression based on the predictive factors identified by univariate and multivariate analyses in the training set and was verified internally in the verification set. A diagnostic nomogram was constructed based on the statistically significant variables of age as well as CT-based AI diagnostic, 7-AAB panel, and CEA test results. In the validation set, the sensitivity, specificity, positive predictive value, and AUC were 82.29%, 90.48%, 97.24%, and 0.899 (95%[CI], 0.851-0.936), respectively. The nomogram showed significantly higher sensitivity than the 7-AAB panel test result (82.29% vs. 35.88%, p < 0.001) and CEA (82.29% vs. 18.82%, p < 0.001); it also had a significantly higher specificity than AI diagnosis (90.48% vs. 69.04%, p = 0.022). For lesions with a diameter of ≤ 2 cm, the specificity of the Nomogram was higher than that of the AI diagnostic system (90.00% vs. 67.50%, p = 0.022). Based on the combination of a 7-AAB panel, an AI diagnostic system, and other clinical features, our Nomogram demonstrated good diagnostic performance in distinguishing lung nodules, especially those with ≤ 2 cm diameters. • A novel diagnostic model of lung nodules was constructed by combining high-specific tumor markers with a high-sensitivity artificial intelligence diagnostic system. • The diagnostic model has good diagnostic performance in distinguishing malignant and benign pulmonary nodules, especially for nodules smaller than 2 cm. • The diagnostic model can assist the clinical decision-making of pulmonary nodules, with the advantages of high diagnostic efficiency, noninvasive, and simple measurement.

Establishment of a deep-learning system to diagnose BI-RADS4a or higher using breast ultrasound for clinical application.

Although the categorization of ultrasound using the Breast Imaging Reporting and Data System (BI‐RADS) has become widespread worldwide, the problem of inter‐observer variability remains. To maintain uniformity in diagnostic accuracy, we have developed a system in which artificial intelligence (AI) can distinguish whether a static image obtained using a breast ultrasound represents BI‐RADS3 or lower or BI‐RADS4a or higher to determine the medical management that should be performed on a patient whose breast ultrasound shows abnormalities. To establish and validate the AI system, a training dataset consisting of 4028 images containing 5014 lesions and a test dataset consisting of 3166 images containing 3656 lesions were collected and annotated. We selected a setting that maximized the area under the curve (AUC) and minimized the difference in sensitivity and specificity by adjusting the internal parameters of the AI system, achieving an AUC, sensitivity, and specificity of 0.95, 91.2%, and 90.7%, respectively. Furthermore, based on 30 images extracted from the test data, the diagnostic accuracy of 20 clinicians and the AI system was compared, and the AI system was found to be significantly superior to the clinicians (McNemar test, p < 0.001). Although deep‐learning methods to categorize benign and malignant tumors using breast ultrasound have been extensively reported, our work represents the first attempt to establish an AI system to classify BI‐RADS3 or lower and BI‐RADS4a or higher successfully, providing important implications for clinical actions. These results suggest that the AI diagnostic system is sufficient to proceed to the next stage of clinical application.

What Are the Users’ Needs? Design of a User-Centered Explainable Artificial Intelligence Diagnostic System

The application of artificial intelligence (AI) systems has surged in the high-risk area of medicine, and these systems must explain their decisions to different users. However, existing explainable AI (XAI) design practices in the medical domain are mostly focused on domain experts, such as physicians, and there is a lack of XAI design practices for consumer users. Therefore, we developed a library of XAI user needs in the medical domain, which can be used as an auxiliary tool for the development of user-centered XAI design solutions in this domain. Moreover, through empirical research, based on our XAI user Needs Library, we designed an XAI-based electrocardiogram diagnostic system prototype for consumer users and conducted a user evaluation. The results provide the empirical experience of the design space of XAI and promote consumer user-centered XAI practices.

Performance of a Chest Radiograph AI Diagnostic Tool for COVID-19: A Prospective Observational Study

PurposeTo conduct a prospective observational study across 12 U.S. hospitals to evaluate real-time performance of an interpretable artificial intelligence (AI) model to detect COVID-19 on chest radiographs.Materials and MethodsA total of 95 363 chest radiographs were included in model training, external validation, and real-time validation. The model was deployed as a clinical decision support system, and performance was prospectively evaluated. There were 5335 total real-time predictions and a COVID-19 prevalence of 4.8% (258 of 5335). Model performance was assessed with use of receiver operating characteristic analysis, precision-recall curves, and F1 score. Logistic regression was used to evaluate the association of race and sex with AI model diagnostic accuracy. To compare model accuracy with the performance of board-certified radiologists, a third dataset of 1638 images was read independently by two radiologists.ResultsParticipants positive for COVID-19 had higher COVID-19 diagnostic scores than participants negative for COVID-19 (median, 0.1 [IQR, 0.0–0.8] vs 0.0 [IQR, 0.0–0.1], respectively; P < .001). Real-time model performance was unchanged over 19 weeks of implementation (area under the receiver operating characteristic curve, 0.70; 95% CI: 0.66, 0.73). Model sensitivity was higher in men than women (P = .01), whereas model specificity was higher in women (P = .001). Sensitivity was higher for Asian (P = .002) and Black (P = .046) participants compared with White participants. The COVID-19 AI diagnostic system had worse accuracy (63.5% correct) compared with radiologist predictions (radiologist 1 = 67.8% correct, radiologist 2 = 68.6% correct; McNemar P < .001 for both).ConclusionAI-based tools have not yet reached full diagnostic potential for COVID-19 and underperform compared with radiologist prediction.Keywords: Diagnosis, Classification, Application Domain, Infection, Lung Supplemental material is available for this article.. © RSNA, 2022

Validation of artificial intelligence prediction models for skin cancer diagnosis using dermoscopy images: the 2019 International Skin Imaging Collaboration Grand Challenge

BackgroundPrevious studies of artificial intelligence (AI) applied to dermatology have shown AI to have higher diagnostic classification accuracy than expert dermatologists; however, these studies did not adequately assess clinically realistic scenarios, such as how AI systems behave when presented with images of disease categories that are not included in the training dataset or images drawn from statistical distributions with significant shifts from training distributions. We aimed to simulate these real-world scenarios and evaluate the effects of image source institution, diagnoses outside of the training set, and other image artifacts on classification accuracy, with the goal of informing clinicians and regulatory agencies about safety and real-world accuracy.MethodsWe designed a large dermoscopic image classification challenge to quantify the performance of machine learning algorithms for the task of skin cancer classification from dermoscopic images, and how this performance is affected by shifts in statistical distributions of data, disease categories not represented in training datasets, and imaging or lesion artifacts. Factors that might be beneficial to performance, such as clinical metadata and external training data collected by challenge participants, were also evaluated. 25 331 training images collected from two datasets (in Vienna [HAM10000] and Barcelona [BCN20000]) between Jan 1, 2000, and Dec 31, 2018, across eight skin diseases, were provided to challenge participants to design appropriate algorithms. The trained algorithms were then tested for balanced accuracy against the HAM10000 and BCN20000 test datasets and data from countries not included in the training dataset (Turkey, New Zealand, Sweden, and Argentina). Test datasets contained images of all diagnostic categories available in training plus other diagnoses not included in training data (not trained category). We compared the performance of the algorithms against that of 18 dermatologists in a simulated setting that reflected intended clinical use.Findings64 teams submitted 129 state-of-the-art algorithm predictions on a test set of 8238 images. The best performing algorithm achieved 58·8% balanced accuracy on the BCN20000 data, which was designed to better reflect realistic clinical scenarios, compared with 82·0% balanced accuracy on HAM10000, which was used in a previously published benchmark. Shifted statistical distributions and disease categories not included in training data contributed to decreases in accuracy. Image artifacts, including hair, pen markings, ulceration, and imaging source institution, decreased accuracy in a complex manner that varied based on the underlying diagnosis. When comparing algorithms to expert dermatologists (2460 ratings on 1269 images), algorithms performed better than experts in most categories, except for actinic keratoses (similar accuracy on average) and images from categories not included in training data (26% correct for experts vs 6% correct for algorithms, p<0·0001). For the top 25 submitted algorithms, 47·1% of the images from categories not included in training data were misclassified as malignant diagnoses, which would lead to a substantial number of unnecessary biopsies if current state-of-the-art AI technologies were clinically deployed.InterpretationWe have identified specific deficiencies and safety issues in AI diagnostic systems for skin cancer that should be addressed in future diagnostic evaluation protocols to improve safety and reliability in clinical practice.FundingMelanoma Research Alliance and La Marató de TV3.

Diagnostic value of circulating genetically abnormal cells to support computed tomography for benign and malignant pulmonary nodules

BackgroundThe accuracy of CT and tumour markers in screening lung cancer needs to be improved. Computer-aided diagnosis has been reported to effectively improve the diagnostic accuracy of imaging data, and recent studies have shown that circulating genetically abnormal cell (CAC) has the potential to become a novel marker of lung cancer. The purpose of this research is explore new ways of lung cancer screening.MethodsFrom May 2020 to April 2021, patients with pulmonary nodules who had received CAC examination within one week before surgery or biopsy at First Affiliated Hospital of Zhengzhou University were enrolled. CAC counts, CT scan images, serum tumour marker (CEA, CYFRA21–1, NSE) levels and demographic characteristics of the patients were collected for analysis. CT were uploaded to the Pulmonary Nodules Artificial Intelligence Diagnostic System (PNAIDS) to assess the malignancy probability of nodules. We compared diagnosis based on PNAIDS, CAC, Mayo Clinic Model, tumour markers alone and their combination. The combination models were built through logistic regression, and was compared through the area under (AUC) the ROC curve.ResultsA total of 93 of 111 patients were included. The AUC of PNAIDS was 0.696, which increased to 0.847 when combined with CAC. The sensitivity (SE), specificity (SP), and positive (PPV) and negative (NPV) predictive values of the combined model were 61.0%, 94.1%, 94.7% and 58.2%, respectively. In addition, we evaluated the diagnostic value of CAC, which showed an AUC of 0.779, an SE of 76.3%, an SP of 64.7%, a PPV of 78.9%, and an NPV of 61.1%, higher than those of any single serum tumour marker and Mayo Clinic Model. The combination of PNAIDS and CAC exhibited significantly higher AUC values than the PNAIDS (P = 0.009) or the CAC (P = 0.047) indicator alone. However, including additional tumour markers did not significantly alter the performance of CAC and PNAIDS.ConclusionsCAC had a higher diagnostic value than traditional tumour markers in early-stage lung cancer and a supportive value for PNAIDS in the diagnosis of cancer based on lung nodules. The results of this study offer a new mode of screening for early-stage lung cancer using lung nodules.

Implementation of artificial intelligence in the histological assessment of pulmonary subsolid nodules.

BackgroundClinical management of subsolid nodules (SSNs) is defined by the suspicion of tumor invasiveness. We sought to develop an artificial intelligent (AI) algorithm for invasiveness assessment of lung adenocarcinoma manifesting as radiological SSNs. We investigated the performance of this algorithm in classification of SSNs related to invasiveness.MethodsA retrospective chest computed tomography (CT) dataset of 1,589 SSNs was constructed to develop (85%) and internally test (15%) the proposed AI diagnostic tool, SSNet. Diagnostic performance was evaluated in the hold-out test set and was further tested in an external cohort of 102 SSNs. Three thoracic surgeons and three radiologists were required to evaluate the invasiveness of SSNs on both test datasets to investigate the clinical utility of the proposed SSNet.ResultsIn the differentiation of invasive adenocarcinoma (IA), SSNet achieved a similar area under the curve [AUC; 0.914, 95% confidence interval (CI): 0.813–0.987] with that of the 6 doctors (0.900, 95% CI: 0.867–0.922). When interpreting with the assistance of SSNet, the sensitivity of junior doctors, specificity of senior doctor, and their accuracy were significantly improved. In the external test, SSNet (AUC: 0.949, 95% CI: 0.884–1.000) achieved a better AUC than doctors (AUC: 0.883, 95% CI: 0.826–0.939) whose AUC increased (AUC: 0.908, 95% CI: 0.847–0.982) with SSNet assistance. In the histological subtype classifications, SSNet achieved better performance than practicing doctors. The AUCs of doctors were significantly improved with the assistance of SSNet in both 4-category and 3-category classifications to 0.836 (95% CI: 0.811–0.862) and 0.852 (95% CI: 0.825–0.882), respectively.ConclusionsThe AI diagnostic system achieved non-inferior performance to doctors, and will potentially improve diagnostic performance and efficiency in SSN evaluation.

An Artificial Intelligence System for the Detection of Bladder Cancer via Cystoscopy: A Multicenter Diagnostic Study.

Cystoscopy plays an important role in bladder cancer (BCa) diagnosis and treatment, but its sensitivity needs improvement. Artificial intelligence has shown promise in endoscopy, but few cystoscopic applications have been reported. We report a Cystoscopy Artificial Intelligence Diagnostic System (CAIDS) for BCa diagnosis. In total, 69 204 images from 10 729 consecutive patients from 6 hospitals were collected and divided into training, internal validation, and external validation sets. The CAIDS was built using a pyramid scene parsing network and transfer learning. A subset (n = 260) of the validation sets was used for a performance comparison between the CAIDS and urologists for complex lesion detection. The diagnostic accuracy, sensitivity, specificity, and positive and negative predictive values and 95% confidence intervals (CIs) were calculated using the Clopper-Pearson method. The diagnostic accuracies of the CAIDS were 0.977 (95% CI = 0.974 to 0.979) in the internal validation set and 0.990 (95% CI = 0.979 to 0.996), 0.982 (95% CI = 0.974 to 0.988), 0.978 (95% CI = 0.959 to 0.989), and 0.991 (95% CI = 0.987 to 0.994) in different external validation sets. In the CAIDS vs urologists' comparisons, the CAIDS showed high accuracy and sensitivity (accuracy = 0.939, 95% CI = 0.902 to 0.964; sensitivity = 0.954, 95% CI = 0.902 to 0.983) with a short latency of 12 seconds, much more accurate and quicker than the expert urologists. The CAIDS achieved accurate BCa detection with a short latency. The CAIDS may provide many clinical benefits, from increasing the diagnostic accuracy for BCa, even for commonly misdiagnosed cases such as flat cancerous tissue (carcinoma in situ), to reducing the operation time for cystoscopy.