Average AUROC Research Articles

Evaluating Machine Learning Models for Stroke Prognosis and Prediction in Atrial Fibrillation Patients: A Comprehensive Meta-Analysis

Background/Objective: Atrial fibrillation (AF) complicates the management of acute ischemic stroke (AIS), necessitating precise predictive models to enhance clinical outcomes. This meta-analysis evaluates the efficacy of machine learning (ML) models in three key areas: stroke prognosis in AF patients, stroke prediction in AF patients, and AF prediction in stroke patients. The study aims to assess the accuracy and variability of ML models in forecasting AIS outcomes and detecting AF in stroke patients, while exploring the clinical benefits and limitations of integrating these models into practice. Methods: We conducted a systematic search of PubMed, Embase, and Cochrane databases up to June 2024, selecting studies that evaluated ML accuracy in stroke prognosis and prediction in AF patients and AF prediction in stroke patients. Data extraction and quality assessment were performed independently by two reviewers, with random-effects modeling applied to estimate pooled accuracy metrics. Results: The meta-analysis included twenty-four studies comprising 7,391,645 patients, categorized into groups for stroke prognosis in AF patients (eight studies), stroke prediction in AF patients (thirteen studies), and AF prediction in stroke patients (three studies). The pooled AUROC was 0.79 for stroke prognosis and 0.68 for stroke prediction in AF, with higher accuracy noted in short-term predictions. The mean AUROC across studies was 0.75, with models such as Extreme Gradient Boosting (XGB) and Random Forest (RF) showing superior performance. For stroke prognosis in AF, the mean AUROC was 0.78, whereas stroke prediction yielded a mean AUROC of 0.73. AF prediction post-stroke had an average AUROC of 0.75. These findings indicate moderate predictive capability of ML models, underscoring the need for further refinement and standardization. The absence of comprehensive sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) metrics limited the ability to conduct full meta-analytic modeling. Conclusions: While ML models demonstrate potential for enhancing stroke prognosis and AF prediction, they have yet to meet the clinical standards required for widespread adoption. Future efforts should focus on refining these models and validating them across diverse populations to improve their clinical utility.

ScCaT: An explainable capsulating architecture for sepsis diagnosis transferring from single-cell RNA sequencing.

Sepsis is a life-threatening condition characterized by an exaggerated immune response to pathogens, leading to organ damage and high mortality rates in the intensive care unit. Although deep learning has achieved impressive performance on prediction and classification tasks in medicine, it requires large amounts of data and lacks explainability, which hinder its application to sepsis diagnosis. We introduce a deep learning framework, called scCaT, which blends the capsulating architecture with Transformer to develop a sepsis diagnostic model using single-cell RNA sequencing data and transfers it to bulk RNA data. The capsulating architecture effectively groups genes into capsules based on biological functions, which provides explainability in encoding gene expressions. The Transformer serves as a decoder to classify sepsis patients and controls. Our model achieves high accuracy with an AUROC of 0.93 on the single-cell test set and an average AUROC of 0.98 on seven bulk RNA cohorts. Additionally, the capsules can recognize different cell types and distinguish sepsis from control samples based on their biological pathways. This study presents a novel approach for learning gene modules and transferring the model to other data types, offering potential benefits in diagnosing rare diseases with limited subjects.

Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders

Deep neural networks are increasingly used in medical imaging for tasks such as pathological classification, but they face challenges due to the scarcity of high-quality, expert-labeled training data. Recent efforts have utilized pre-trained contrastive image-text models like CLIP, adapting them for medical use by fine-tuning the model with chest X-ray images and corresponding reports for zero-shot pathology classification, thus eliminating the need for pathology-specific annotations. However, most studies continue to use the same contrastive learning objectives as in the general domain, overlooking the multi-labeled nature of medical image-report pairs. In this paper, we propose a new fine-tuning strategy that includes positive-pair loss relaxation and random sentence sampling. We aim to improve the performance of zero-shot pathology classification without relying on external knowledge. Our method can be applied to any pre-trained contrastive image-text encoder and easily transferred to out-of-domain datasets without further training, as it does not use external data. Our approach consistently improves overall zero-shot pathology classification across four chest X-ray datasets and three pre-trained models, with an average macro AUROC increase of 4.3%. Additionally, our method outperforms the state-of-the-art and marginally surpasses board-certified radiologists in zero-shot classification for the five competition pathologies in the CheXpert dataset.

A flexible deep learning framework for liver tumor diagnosis using variable multi-phase contrast-enhanced CT scans

BackgroundLiver cancer is a significant cause of cancer-related mortality worldwide and requires tailored treatment strategies for different types. However, preoperative accurate diagnosis of the type presents a challenge. This study aims to develop an automatic diagnostic model based on multi-phase contrast-enhanced CT (CECT) images to distinguish between hepatocellular carcinoma (HCC), intrahepatic cholangiocarcinoma (ICC), and normal individuals.MethodsWe designed a Hierarchical Long Short-Term Memory (H-LSTM) model, whose core components consist of a shared image feature extractor across phases, an internal LSTM for each phase, and an external LSTM across phases. The internal LSTM aggregates features from different layers of 2D CECT images, while the external LSTM aggregates features across different phases. H-LSTM can handle incomplete phases and varying numbers of CECT image layers, making it suitable for real-world decision support scenarios. Additionally, we applied phase augmentation techniques to process multi-phase CECT images, improving the model’s robustness.ResultsThe H-LSTM model achieved an overall average AUROC of 0.93 (0.90, 1.00) on the test dataset, with AUROC for HCC classification reaching 0.97 (0.93, 1.00) and for ICC classification reaching 0.90 (0.78, 1.00). Comprehensive validation in scenarios with incomplete phases was performed, with the H-LSTM model consistently achieving AUROC values over 0.9.ConclusionThe proposed H-LSTM model can be employed for classification tasks involving incomplete phases of CECT images in real-world scenarios, demonstrating high performance. This highlights the potential of AI-assisted systems in achieving accurate diagnosis and treatment of liver cancer. H-LSTM offers an effective solution for processing multi-phase data and provides practical value for clinical diagnostics.

Age added to MELD or ACLF predicts survival in patients with alcohol-associated hepatitis declined for liver transplantation.

Severe alcohol-associated hepatitis (AH) that is nonresponsive to corticosteroids is associated with high mortality, particularly with concomitant acute-on-chronic liver failure (ACLF). Most patients will not be candidates for liver transplantation (LT) and their outcomes are largely unknown. Our aim was to determine the outcomes of these declined candidates and to derive practical prediction models for transplant-free survival applicable at the time of the waitlist decision. We analyzed a database of patients with severe AH who were hospitalized at a LT center from January 2012 to July 2021, using the National Death Index for those lacking follow-up. Clinical variables were analyzed based on the endpoints of mortality at 30, 60, 90, and 180 days. Logistic and Cox regression analyses were used for model derivation. Over 9.5 years, 206 patients with severe AH were declined for LT, mostly for unfavorable psychosocial profiles, with a mean MELD of 33 (±8), and 61% with ACLF. Over a median follow-up of 521 (17.5-1368) days, 58% (119/206) died at a median of 21 (9-124) days. Of 32 variables, only age added prognostic value to MELD and ACLF grade. CLIF-C ACLF score and 2 new models, MELD-Age and ACLF-Age, had similar predictability (AUROC: 0.73, 0.73, 0.72, respectively), outperforming Lille and Maddrey's (AUROC: 0.63, 0.62). In internal cross-validation, the average AUROC was 0.74. ACLF grade ≥2, MELD score >35, and age >45 years were useful cutoffs for predicting increased 90-day mortality from waitlist decision. Only two patients initially declined for LT for AH subsequently underwent LT (1%). Patients with severe AH declined for LT have high short-term mortality and rare rates of subsequent LT. Age added to MELD or ACLF grade enhances survival prediction at the time of waitlist decision in patients with severe AH declined for LT.

MAPRS: An intelligent approach for post-prescription review based on multi-label learning

Antimicrobial resistance (AMR) is a major threat to public health worldwide. It is a promising way to improve appropriate prescription by the review and stewardship of antimicrobials, and Post-Prescription Review (PPR) is currently the main tool used in hospitals. Existing methods of PPR typically focus on the dichotomy of antimicrobial prescription based on binary classification which, however, is usually a multi-label classification problem. Moreover, previous research did not explain the causes beneath the inappropriate antimicrobial used in the clinical setting, which could be practically important for problem location and decision improvement. In this paper, we collected antimicrobial prescriptions and related data from clean surgery in a hospital in northeastern China, and proposed a Multi-label Antimicrobial Post-Prescription Review System (MAPRS). MAPRS first uses NLP techniques to process unstructured data in prescriptions and explores the value of clinical record text for solving medical problems. Then, Classifier Chains are used to deal with multi-label problems and fused with machine learning algorithms to construct a classifier. At last, a SHAP explanation module is introduced to explain the inappropriate prescriptions. The experimental results show that MAPRS could achieve great performance in a challenging six-category multi-label task, with a subset accuracy of 90.7 % and an average AUROC of 94.3 %. Our results can help hospitals to perform intelligent prescription review and improve the antimicrobial stewardship.

When an extra rejection class meets out-of-distribution detection in long-tailed image classification

Detecting Out-of-Distribution (OOD) inputs is essential for reliable deep learning in the open world. However, most existing OOD detection methods have been developed based on training sets that exhibit balanced class distributions, making them susceptible when confronted with training sets following a long-tailed distribution. To alleviate this problem, we propose an effective three-branch training framework, which demonstrates the efficacy of incorporating an extra rejection class along with auxiliary outlier training data for effective OOD detection in long-tailed image classification. In our proposed framework, all outlier training samples are assigned the label of the rejection class. We employ an inlier loss, an outlier loss, and a Tail-class prototype induced Supervised Contrastive Loss (TSCL) to train both the in-distribution classifier and OOD detector within one network. During inference, the OOD detector is constructed using the rejection class. Extensive experimental results demonstrate that the superior OOD detection performance of our proposed method in long-tailed image classification. For example, in the more challenging case where CIFAR100-LT is used as in-distribution, our method improves the average AUROC by 1.23% and reduces the average FPR95 by 3.18% compared to the baseline method utilizing Outlier Exposure (OE). Code is available at github.

EPI-Trans: an effective transformer-based deep learning model for enhancer promoter interaction prediction

BackgroundRecognition of enhancer–promoter Interactions (EPIs) is crucial for human development. EPIs in the genome play a key role in regulating transcription. However, experimental approaches for classifying EPIs are too expensive in terms of effort, time, and resources. Therefore, more and more studies are being done on developing computational techniques, particularly using deep learning and other machine learning techniques, to address such problems. Unfortunately, the majority of current computational methods are based on convolutional neural networks, recurrent neural networks, or a combination of them, which don’t take into consideration contextual details and the long-range interactions between the enhancer and promoter sequences. A new transformer-based model called EPI-Trans is presented in this study to overcome the aforementioned limitations. The multi-head attention mechanism in the transformer model automatically learns features that represent the long interrelationships between enhancer and promoter sequences. Furthermore, a generic model is created with transferability that can be utilized as a pre-trained model for various cell lines. Moreover, the parameters of the generic model are fine-tuned using a particular cell line dataset to improve performance.ResultsBased on the results obtained from six benchmark cell lines, the average AUROC for the specific, generic, and best models is 94.2%, 95%, and 95.7%, while the average AUPR is 80.5%, 66.1%, and 79.6% respectively.ConclusionsThis study proposed a transformer-based deep learning model for EPI prediction. The comparative results on certain cell lines show that EPI-Trans outperforms other cutting-edge techniques and can provide superior performance on the challenge of recognizing EPI.

An efficient deep neural network for automatic classification of acute intracranial hemorrhages in brain CT scans

BackgroundRecent advancements in deep learning models have demonstrated their potential in the field of medical imaging, achieving remarkable performance surpassing human capabilities in tasks such as classification and segmentation. However, these modern state-of-the-art network architectures often demand substantial computational resources, which limits their practical application in resource-constrained settings. This study aims to propose an efficient diagnostic deep learning model specifically designed for the classification of intracranial hemorrhage in brain CT scans. MethodOur proposed model utilizes a combination of depthwise separable convolutions and a multi-receptive field mechanism to achieve a trade-off between performance and computational efficiency. The model was trained on RSNA datasets and validated on CQ500 dataset and PhysioNet dataset. ResultThrough a comprehensive comparison with state-of-the-art models, our model achieves an average AUROC score of 0.952 on RSNA datasets and exhibits robust generalization capabilities, comparable to SE-ResNeXt, across other open datasets. Furthermore, the parameter count of our model is just 3 % of that of MobileNet V3. ConclusionThis study presents a diagnostic deep-learning model that is optimized for classifying intracranial hemorrhages in brain CT scans. The efficient characteristics make our proposed model highly promising for broader applications in medical settings.

Drug repositioning based on tripartite cross-network embedding and graph convolutional network

Drug-disease association prediction is an important part of drug discovery, which can help researchers uncover potential drug candidates and disease targets more accurately to deal with the drawbacks of inefficiency and high cost of traditional methods. In contrast, past drug-disease prediction processes often overlooked the associations between drugs and side effects and the impact of proteins on diseases. These issues hindered models from fully learning the potential information within the network. Hence, a method capable of comprehensively learning from multi-source heterogeneous information and extracting high-level features of biological associations is needed. To overcome these challenges, proposes a drug repositioning method based on a tripartite cross-network embedding graph convolutional network, called TGCNDR, to predict the associations between drugs and diseases. Firstly, we construct a tripartite cross-network of drug-disease associations, drug-protein associations, and drug-side effect associations. TGCNDR employs graph convolutional networks to capture information from different nodes to learn drug embedding. Subsequently, TGCNDR uses anchor links for knowledge transfer, and different networks use the Self-attention mechanism to represent drug importance. Experiments on benchmark datasets demonstrate that TGCNDR exhibits good performance (the average AUROC is 0.922, the average AUPR is 0.916, and the average F1-score is 0.860) compared with state-of-the-art methods. In the case study, we perform molecular docking experiments on the predicted drug candidates and identify newly approved drugs for osteosarcoma, which suggests that TGCNDR is an effective tool for drug repositioning.

Toward confident prostate cancer detection using ultrasound: a multi-center study.

Deep learning-based analysis of micro-ultrasound images to detect cancerous lesions is a promising tool for improving prostate cancer (PCa) diagnosis. An ideal model should confidently identify cancer while responding with appropriate uncertainty when presented with out-of-distribution inputs that arise during deployment due to imaging artifacts and the biological heterogeneity of patients and prostatic tissue. Using micro-ultrasound data from 693 patients across 5 clinical centers who underwent micro-ultrasound guided prostate biopsy, we train and evaluate convolutional neural network models for PCa detection. To improve robustness to out-of-distribution inputs, we employ and comprehensively benchmark several state-of-the-art uncertainty estimation methods. PCa detection models achieve performance scores up to average AUROC with a 10-fold cross validation setup. Models with uncertainty estimation obtain expected calibration error scores as low as , indicating that confident predictions are very likely to be correct. Visualizations of the model output demonstrate that the model correctly identifies healthy versus malignant tissue. Deep learning models have been developed to confidently detect PCa lesions from micro-ultrasound. The performance of these models, determined from a large and diverse dataset, is competitive with visual analysis of magnetic resonance imaging, the clinical benchmark to identify PCa lesions for targeted biopsy. Deep learning with micro-ultrasound should be further studied as an avenue for targeted prostate biopsy.

CAPTURE: Comprehensive anti-cancer peptide predictor with a unique amino acid sequence encoder

Anticancer peptides (ACPs) key properties including bioactivity, high efficacy, low toxicity, and lack of drug resistance make them ideal candidates for cancer therapies. To deeply explore the potential of ACPs and accelerate development of cancer therapies, although 53 Artificial Intelligence supported computational predictors have been developed for ACPs and non ACPs classification but only one predictor has been developed for ACPs functional types annotations. Moreover, these predictors extract amino acids distribution patterns to transform peptides sequences into statistical vectors that are further fed to classifiers for discriminating peptides sequences and annotating peptides functional classes. Overall, these predictors remain fail in extracting diverse types of amino acids distribution patterns from peptide sequences. The paper in hand presents a unique CARE encoder that transforms peptides sequences into statistical vectors by extracting 4 different types of distribution patterns including correlation, distribution, composition, and transition. Across public benchmark dataset, proposed encoder potential is explored under two different evaluation settings namely; intrinsic and extrinsic. Extrinsic evaluation indicates that 12 different machine learning classifiers achieve superior performance with the proposed encoder as compared to 55 existing encoders. Furthermore, an intrinsic evaluation reveals that, unlike existing encoders, the proposed encoder generates more discriminative clusters for ACPs and non-ACPs classes. Across 8 public benchmark ACPs and non-ACPs classification datasets, proposed encoder and Adaboost classifier based CAPTURE predictor outperforms existing predictors with an average accuracy, recall and MCC score of 1%, 4%, and 2% respectively. In generalizeability evaluation case study, across 7 benchmark anti-microbial peptides classification datasets, CAPTURE surpasses existing predictors by an average AU-ROC of 2%. CAPTURE predictive pipeline along with label powerset method outperforms state-of-the-art ACPs functional types predictor by 5%, 5%, 5%, 6%, and 3% in terms of average accuracy, subset accuracy, precision, recall, and F1 respectively. CAPTURE web application is available at https://sds_genetic_analysis.opendfki.de/CAPTURE.

A prediction method of interaction based on Bilinear Attention Networks for designing polyphenol-protein complexes delivery systems

Polyphenol-protein complexes delivery systems are gaining attention for their potential health benefits and food industry development. However, creating an ideal delivery system requires extensive wet-lab experimentation. To address this, we collected 525 ligand-protein interaction data pairs and established an interaction prediction model using Bilinear Attention Networks. We utilized 10-fold cross validation to address potential overfitting issues in the model, resulting in showed higher average AUROC (0.8443), AUPRC (0.7872), and F1 (0.8164). The optimal threshold (0.3739) was selected for the model to be used for subsequent analysis. Based on the model prediction results and optimal threshold, by verifying experimental analysis, the interaction of paeonol with the following proteins was obtained, including bovine serum albumin (lgKa = 6.2759), bovine β-lactoglobulin (lgKa = 6.7479), egg ovalbumin (lgKa = 5.1806), zein (lgKa = 6.0122), bovine α-lactalbumin (lgKa = 3.9170), bovine lactoferrin (lgKa = 4.5380), the first four proteins are consistent with the predicted results of the model, with lgKa >5. The established model can accurately and rapidly predict the interaction of polyphenol-protein complexes. This study is the first to combine open ligand-protein interaction experiments with Deep Learning algorithms in the food industry, greatly improving research efficiency and providing a novel perspective for future complex delivery system construction.

An Unsupervised Method for Industrial Image Anomaly Detection with Vision Transformer-Based Autoencoder.

Existing industrial image anomaly detection techniques predominantly utilize codecs based on convolutional neural networks (CNNs). However, traditional convolutional autoencoders are limited to local features, struggling to assimilate global feature information. CNNs' generalizability enables the reconstruction of certain anomalous regions. This is particularly evident when normal and abnormal regions, despite having similar pixel values, contain different semantic information, leading to ineffective anomaly detection. Furthermore, collecting abnormal image samples during actual industrial production poses challenges, often resulting in data imbalance. To mitigate these issues, this study proposes an unsupervised anomaly detection model employing the Vision Transformer (ViT) architecture, incorporating a Transformer structure to understand the global context between image blocks, thereby extracting a superior representation of feature information. It integrates a memory module to catalog normal sample features, both to counteract anomaly reconstruction issues and bolster feature representation, and additionally introduces a coordinate attention (CA) mechanism to intensify focus on image features at both spatial and channel dimensions, minimizing feature information loss and thereby enabling more precise anomaly identification and localization. Experiments conducted on two public datasets, MVTec AD and BeanTech AD, substantiate the method's effectiveness, demonstrating an approximate 20% improvement in average AUROC% at the image level over traditional convolutional encoders.

M6A-TCPred: a web server to predict tissue-conserved human m6A sites using machine learning approach

BackgroundN6-methyladenosine (m6A) is the most prevalent post-transcriptional modification in eukaryotic cells that plays a crucial role in regulating various biological processes, and dysregulation of m6A status is involved in multiple human diseases including cancer contexts. A number of prediction frameworks have been proposed for high-accuracy identification of putative m6A sites, however, none have targeted for direct prediction of tissue-conserved m6A modified residues from non-conserved ones at base-resolution level.ResultsWe report here m6A-TCPred, a computational tool for predicting tissue-conserved m6A residues using m6A profiling data from 23 human tissues. By taking advantage of the traditional sequence-based characteristics and additional genome-derived information, m6A-TCPred successfully captured distinct patterns between potentially tissue-conserved m6A modifications and non-conserved ones, with an average AUROC of 0.871 and 0.879 tested on cross-validation and independent datasets, respectively.ConclusionOur results have been integrated into an online platform: a database holding 268,115 high confidence m6A sites with their conserved information across 23 human tissues; and a web server to predict the conserved status of user-provided m6A collections. The web interface of m6A-TCPred is freely accessible at: www.rnamd.org/m6ATCPred.

CASE: Exploiting Intra-class Compactness and Inter-class Separability of Feature Embeddings for Out-of-Distribution Detection

Detecting out-of-distribution (OOD) inputs is critical for reliable machine learning, but deep neural networks often make overconfident predictions, even for OOD inputs that deviate from the distribution of training data. Prior methods relied on the widely used softmax cross-entropy (CE) loss that is adequate for classifying in-distribution (ID) samples but not optimally designed for OOD detection. To address this issue, we propose CASE, a simple and effective OOD detection method by explicitly improving intra-class Compactness And inter-class Separability of feature Embeddings. To enhance the separation between ID and OOD samples, CASE uses a dual-loss framework, which includes a separability loss that maximizes the inter-class Euclidean distance to promote separability among different class centers, along with a compactness loss that minimizes the intra-class Euclidean distance to encourage samples to be close to their class centers. In particular, the class centers are defined as a free optimization parameter of the model and updated by gradient descent, which is simple and further enhances the OOD detection performance. Extensive experiments demonstrate the superiority of CASE, which reduces the average FPR95 by 37.11% and improves the average AUROC by 15.89% compared to the baseline method using a softmax confidence score on the more challenging CIFAR-100 model.

Building large-scale registries from unstructured clinical notes using a low-resource natural language processing pipeline

Building clinical registries is an important step in clinical research and improvement of patient care quality. Natural Language Processing (NLP) methods have shown promising results in extracting valuable information from unstructured clinical notes. However, the structure and nature of clinical notes are very different from regular text that state-of-the-art NLP models are trained and tested on, and they have their own set of challenges. In this study, we propose Sentence Extractor with Keywords (SE-K), an efficient and interpretable classification approach for extracting information from clinical notes and show that it outperforms more computationally expensive methods in text classification. Following the Institutional Review Board (IRB) approval, we used SE-K and two embedding based NLP approaches (Sentence Extractor with Embeddings (SE-E) and Bidirectional Encoder Representations from Transformers (BERT)) to develop comprehensive registry of anterior cruciate ligament surgeries from 20 years of unstructured clinical data at a multi-site tertiary-care regional children's hospital. The low-resource approach (SE-K) had better performance (average AUROC of 0.94 ± 0.04) than the embedding-based approaches (SE-E: 0.93 ± 0.04 and BERT: 0.87 ± 0.09) for out of sample validation, in addition to minimum performance drop between test and out-of-sample validation. Moreover, the SE-K approach was at least six times faster (on CPU) than SE-E (on CPU) and BERT (on GPU) and provides interpretability. Our proposed approach, SE-K, can be effectively used to extract relevant variables from clinic notes to build large-scale registries, with consistently better performance compared to the more resource-intensive approaches (e.g., BERT). Such approaches can facilitate information extraction from unstructured notes for registry building, quality improvement and adverse event monitoring.

Systematic evaluation of common natural language processing techniques to codify clinical notes.

Proper codification of medical diagnoses and procedures is essential for optimized health care management, quality improvement, research, and reimbursement tasks within large healthcare systems. Assignment of diagnostic or procedure codes is a tedious manual process, often prone to human error. Natural Language Processing (NLP) has been suggested to facilitate this manual codification process. Yet, little is known on best practices to utilize NLP for such applications. With Large Language Models (LLMs) becoming more ubiquitous in daily life, it is critical to remember, not every task requires that level of resource and effort. Here we comprehensively assessed the performance of common NLP techniques to predict current procedural terminology (CPT) from operative notes. CPT codes are commonly used to track surgical procedures and interventions and are the primary means for reimbursement. Our analysis of 100 most common musculoskeletal CPT codes suggest that traditional approaches can outperform more resource intensive approaches like BERT significantly (P-value = 4.4e-17) with average AUROC of 0.96 and accuracy of 0.97, in addition to providing interpretability which can be very helpful and even crucial in the clinical domain. We also proposed a complexity measure to quantify the complexity of a classification task and how this measure could influence the effect of dataset size on model's performance. Finally, we provide preliminary evidence that NLP can help minimize the codification error, including mislabeling due to human error.

TCGA-Reports: A machine-readable pathology report resource for benchmarking text-based AI models

In cancer research, pathology report text is a largely untapped data source. Pathology reports are routinely generated, more nuanced than structured data, and contain added insight from pathologists. However, there are no publicly available datasets for benchmarking report-based models. Two recent advances suggest the urgent need for a benchmark dataset. First, improved optical character recognition (OCR) techniques will make it possible to access older pathology reports in an automated way, increasing the data available for analysis. Second, recent improvements in natural language processing (NLP) techniques using artificial intelligence (AI) allow more accurate prediction of clinical targets from text. We apply state-of-the-art OCR and customized post-processing to report PDFs from The Cancer Genome Atlas, generating a machine-readable corpus of 9,523 reports. Finally, we perform a proof-of-principle cancer-type classification across 32 tissues, achieving 0.992 average AU-ROC. This dataset will be useful to researchers across specialties, including research clinicians, clinical trial investigators, and clinical NLP researchers.

Development of an image-based Random Forest classifier for prediction of surgery duration of laparoscopic sigmoid resections

PurposeSigmoid diverticulitis is a disease with a high socioeconomic burden, accounting for a high number of left-sided colonic resections worldwide. Modern surgical scheduling relies on accurate prediction of operation times to enhance patient care and optimize healthcare resources. This study aims to develop a predictive model for surgery duration in laparoscopic sigmoid resections, based on preoperative CT biometric and demographic patient data.MethodsThis retrospective single-center cohort study included 85 patients who underwent laparoscopic sigmoid resection for diverticular disease. Potentially relevant procedure-specific anatomical parameters recommended by a surgical expert were measured in preoperative CT imaging. After random split into training and test set (75% / 25%) multiclass logistic regression was performed and a Random Forest classifier was trained on CT imaging parameters, patient age, and sex in the training cohort to predict categorized surgery duration. The models were evaluated in the test cohort using established performance metrics including receiver operating characteristics area under the curve (AUROC).ResultsThe Random Forest model achieved a good average AUROC of 0.78. It allowed a very good prediction of long (AUROC = 0.89; specificity 0.71; sensitivity 1.0) and short (AUROC = 0.81; specificity 0.77; sensitivity 0.56) procedures. It clearly outperformed the multiclass logistic regression model (AUROC: average = 0.33; short = 0.31; long = 0.22).ConclusionA Random Forest classifier trained on demographic and CT imaging biometric patient data could predict procedure duration outliers of laparoscopic sigmoid resections. Pending validation in a multicenter study, this approach could potentially improve procedure scheduling in visceral surgery and be scaled to other procedures.