Area Under The Receiver Operating Characteristic Curve For Model Research Articles

Intravoxel incoherent motion diffusion-weighted imaging for predicting kidney allograft function decline: comparison with clinical parameters

ObjectiveTo evaluate the added benefit of diffusion-weighted imaging (DWI) over clinical parameters in predicting kidney allograft function decline.MethodsData from 97 patients with DWI of the kidney allograft were retrospectively analyzed. The DWI signals were analyzed with both the mono-exponential and bi-exponential models, yielding total apparent diffusion coefficient (ADCT), true diffusion (D), pseudo-diffusion (D*), and perfusion fraction (fp). Three predictive models were constructed: Model 1 with clinical parameters, Model 2 with DWI parameters, and Model 3 with both clinical and DWI parameters. The predictive capability of each model was compared by calculating the area under the receiver-operating characteristic curve (AUROC).ResultsForty-five patients experienced kidney allograft function decline during a median follow-up of 98 months. The AUROC for Model 1 gradually decreased with follow-up time > 40 months, whereas Model 2 and Model 3 maintained relatively stable AUROCs. The AUROCs of Model 1 and Model 2 were not statistically significant. Multivariable analysis showed that the Model 3 included cortical D (HR = 3.93, p = 0.001) and cortical fp (HR = 2.85, p = 0.006), in addition to baseline estimated glomerular filtration rate (eGFR) and proteinuria. The AUROCs for Model 3 were significantly higher than those for Model 1 at 60-month (0.91 vs 0.86, p = 0.02) and 84-month (0.90 vs 0.83, p = 0.007) follow-up.ConclusionsDWI parameters were comparable to clinical parameters in predicting kidney allograft function decline. Integrating cortical D and fp into the clinical model with baseline eGFR and proteinuria may add prognostic value for long-term allograft function decline.Critical relevance statementOur findings suggested that cortical D and fp derived from IVIM-DWI increased the performance to predict long-term kidney allograft function decline. This preliminary study provided basis for the utility of multi-b DWI for managing patients with a kidney transplant.Key points• Both clinical and multi-b DWI parameters could predict kidney allograft function decline.• The ability to predict kidney allograft function decline was similar between DWI and clinical parameters.• Cortical D and fp derived from IVIM-DWI increased the performance to predict long-term kidney allograft function decline.Graphical

Abstract PO-084: Automated detection of pancreatic ductal adenocarcinoma (PDAC) on CT scans using artificial intelligence (AI): Impact of inclusion of automated pancreas segmentation on the accuracy of 3D-convolutional neural network (CNN)

Abstract Purpose: Around 30% of PDAC less than 2-cm tend to go undetected on CT due to their subtle imaging signatures. Automated detection of PDAC using AI represents an opportunity to augment physician expertise and to improve outcomes through early detection of PDAC. Our purpose was to develop a 3D-CNN for fully automated detection of PDAC and to further evaluate the impact of inclusion of pancreas segmentation on the accuracy of this 3D-CNN. Methods: A Medical Imaging Data Readiness Scale (MIDaR) level A dataset (portal venous phase CTs, slice thickness ≤ 3.75 mm) of 466 treatment-naïve biopsy-proven PDAC and 1994 subjects with normal pancreas was created after exclusion of CTs with suboptimal image quality or biliary stents. Volumetric pancreas and tumor segmentations on CTs were done by two radiologists using 3D Slicer. A total of 370 CTs with PDAC and 370 CTs with normal pancreas were randomly selected for separate training and validation sets, and 396 CTs (96 CTs with PDAC and 300 CTs with normal pancreas) were utilized for testing. Two separate 3D-CNNs were trained. A three-stage bounding-box-only model (A): stage 1 was based on a UNET-like architecture and localized the pancreas on CT with a bounding box; stage 2 utilized an Inception ResNet architecture and classified each slice through the pancreas into PDAC vs. normal; and stage 3 utilized the output of stage 2 to generate final classification for a given CT. Conversely, a four-stage pancreas segmentation-based model (B) included stage 1 of model A followed by an additional stage of automated pancreas and tumor segmentation (stage 2), classification of each slice through the pancreas into PDAC vs. normal (stage 3) and, finally, generation of final classification score (stage 4) for a given CT. Area under the receiver operating characteristic curve (AUROC) of the two models were compared on the test set. Results: Mean (SD) PDAC diameter in the test set was 1.1 (0.43) cm. Model A (three-stage bounding-box-only) correctly classified 305 (77%) out of 396 CTs from the test set into PDAC vs. normal. It incorrectly classified 12/96 (12.5%) CTs with PDAC as normal and 79/300 (26%) normal CTs as PDAC. AUROC for model A was 0.85. Model B (four-stage pancreas segmentation-based) correctly classified 351 (88%) out of 396 CTs. It incorrectly classified 13/96 (13.5%) CTs with PDAC as normal and 32/300 (10.7%) normal CTs as PDAC. AUROC for model B was 0.94. AUROC for model B was significantly higher than model A (p<0.005). Conclusion: A 3D-CNN can detect small PDAC with high accuracy using automated localization of pancreas with a bounding box without relying on separate pancreas segmentation. Inclusion of an additional automated pancreas segmentation step reduced false positives with consequent incremental gain in the model’s accuracy. Prospective validation and subsequent integration of such models into clinical workflows has the potential to reduce inadvertent errors in detection of subtle or small PDAC on standard-of-care CT scans. Citation Format: Anurima Patra, Korfiatis Panagiotis, Garima Suman, Ananya Panda, Sushil Kumar Garg, Ajit Goenka. Automated detection of pancreatic ductal adenocarcinoma (PDAC) on CT scans using artificial intelligence (AI): Impact of inclusion of automated pancreas segmentation on the accuracy of 3D-convolutional neural network (CNN) [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-084.

Development and validation of a procedure-based organ failure assessment model for patients in the intensive care unit: an administrative database study.

AimTo develop a procedure‐based organ failure assessment model for intensive care unit (ICU) patients and to examine the ability of this model to predict in‐hospital mortality, with reference to the Sequential Organ Failure Assessment (SOFA) score.MethodsUsing the Japanese nationwide Diagnosis Procedure Combination database, we identified patients aged ≥15 years who were admitted to the ICUs April 2018–March 2019. Since April 2018, Japanese health care providers have been required to input ICU patients' SOFA scores into this database. We extracted data on the following procedures on ICU admission: oxygen supplementation, invasive mechanical ventilation, blood transfusions, catecholamines, chest compression, extracorporeal membrane oxygenation, and renal replacement therapy. A procedure‐based organ failure assessment model (Model 1) for in‐hospital mortality was developed using therapeutic procedures for organ failure on the day of ICU admission in the derivation cohort. We also constructed a model using the SOFA score (Model 2). Discriminatory ability was assessed using area under the receiver operating characteristic curve (AUROC) in the validation cohort, and the discriminatory abilities of the models were compared.ResultsIn total, 69,019 patients were included. Overall in‐hospital mortality was 7.2%. The AUROCs for Model 1 (0.810) and Model 2 (0.817) in the validation cohort did not show a statistically significant difference (P = 0.20).ConclusionThe models established using procedure‐based organ failure assessment showed no statistically significant differences from those using the SOFA score, suggesting that procedure records in administrative databases can be used for risk adjustment in clinical studies on ICU mortality.

Artificial Intelligence-Based Multimodal Risk Assessment Model for Surgical Site Infection (AMRAMS): Development and Validation Study.

BackgroundSurgical site infection (SSI) is one of the most common types of health care–associated infections. It increases mortality, prolongs hospital length of stay, and raises health care costs. Many institutions developed risk assessment models for SSI to help surgeons preoperatively identify high-risk patients and guide clinical intervention. However, most of these models had low accuracies.ObjectiveWe aimed to provide a solution in the form of an Artificial intelligence–based Multimodal Risk Assessment Model for Surgical site infection (AMRAMS) for inpatients undergoing operations, using routinely collected clinical data. We internally and externally validated the discriminations of the models, which combined various machine learning and natural language processing techniques, and compared them with the National Nosocomial Infections Surveillance (NNIS) risk index.MethodsWe retrieved inpatient records between January 1, 2014, and June 30, 2019, from the electronic medical record (EMR) system of Rui Jin Hospital, Luwan Branch, Shanghai, China. We used data from before July 1, 2018, as the development set for internal validation and the remaining data as the test set for external validation. We included patient demographics, preoperative lab results, and free-text preoperative notes as our features. We used word-embedding techniques to encode text information, and we trained the LASSO (least absolute shrinkage and selection operator) model, random forest model, gradient boosting decision tree (GBDT) model, convolutional neural network (CNN) model, and self-attention network model using the combined data. Surgeons manually scored the NNIS risk index values.ResultsFor internal bootstrapping validation, CNN yielded the highest mean area under the receiver operating characteristic curve (AUROC) of 0.889 (95% CI 0.886-0.892), and the paired-sample t test revealed statistically significant advantages as compared with other models (P<.001). The self-attention network yielded the second-highest mean AUROC of 0.882 (95% CI 0.878-0.886), but the AUROC was only numerically higher than the AUROC of the third-best model, GBDT with text embeddings (mean AUROC 0.881, 95% CI 0.878-0.884, P=.47). The AUROCs of LASSO, random forest, and GBDT models using text embeddings were statistically higher than the AUROCs of models not using text embeddings (P<.001). For external validation, the self-attention network yielded the highest AUROC of 0.879. CNN was the second-best model (AUROC 0.878), and GBDT with text embeddings was the third-best model (AUROC 0.872). The NNIS risk index scored by surgeons had an AUROC of 0.651.ConclusionsOur AMRAMS based on EMR data and deep learning methods—CNN and self-attention network—had significant advantages in terms of accuracy compared with other conventional machine learning methods and the NNIS risk index. Moreover, the semantic embeddings of preoperative notes improved the model performance further. Our models could replace the NNIS risk index to provide personalized guidance for the preoperative intervention of SSIs. Through this case, we offered an easy-to-implement solution for building multimodal RAMs for other similar scenarios.

Development of a predictive risk model for school readiness at age 3 years using the UK Millennium Cohort Study

ObjectivesThe aim of this study is to develop a predictive risk model (PRM) for school readiness measured at age 3 years using perinatal and early infancy data.Design and participantsThis paper...