Hold-out Set Research Articles

Development of a Perioperative Risk Mortality Calculator for Humanitarian Surgical Care.

Risk models to predict perioperative mortality rates (POMR) are critical to surgical quality improvement yet are not widely adapted for use in humanitarian and low-resource settings (LRS). We developed a POMR and corresponding nomogram and calculator for use in humanitarian surgical care. Electronic health record data from a high-income academic medical center from 2015 to 2019 were retrospectively extracted, selecting variables and operations specific to LRS. This development dataset was used to create a logistic regression POMR model, which was then prospectively validated using data from 2022 to 2023 from the same institution. EHR from a total of 49,277 patients were used. The model fitted eight variables feasibly obtainable in LRS: age >65years (OR=4.05 and 95% CI: 3.71-4.43), male sex (OR=1.32 and 95% CI: 1.25-1.40), GCS < 13 (OR=5.20 and 95% CI: 4.73-5.73), glucose > 200mg/dL (OR=2.19 and 95% CI: 2.01-2.38), Hgb ≤ 11g/dL (OR=2.65 and 95% CI: 2.43-2.89), HR>120 bpm (OR=2.49 and 95% CI: 2.35-2.64), T>38 degrees Celsius (OR=1.32 and 95% CI: 1.19-1.45), and SBP > 180mmHg (OR=1.18 and 95% CI: 1.02-1.37). The model demonstrated a high area under the curve (0.847, 0.867, and 0.925), sensitivity (0.739, 0.886, and 0.844), specificity (0.807, 0.780, and 0.864), and negative predictive value (0.750, 0.997, and 0.999) on training, holdout, and prospective validation sets. We validated a POMR model for use in LRS using eight variables that are readily available in the target environment. This model's predictors and accompanying clinical tools of an Excel calculator and nomogram make it simultaneously comprehensive and accessible in LRS.

Accurate Classification of Human CD4+ T, CD8+ T, and CD19+ B Cells Isolated from Splenocytes by Cross-Polarized Diffraction Image Pairs.

Diffraction imaging of cells allows rapid phenotyping by the response of intracellular molecules to coherent illumination. However, its ability to distinguish numerous types of human leukocytes remains to be investigated. Here, we show that accurate classification of three lymphocyte subtypes can be achieved with features extracted from cross-polarized diffraction image (p-DI) pairs. A deep neural network (DNN) of DINet-PS has been developed for feature extraction from and filtering of, in the angular frequency domain, p-DI pairs acquired from live lymphocytes isolated from human spleen tissues. We built the network in a dual-channel structure and incorporated two adaptive spectral filter blocks to actively suppress extracted features related to the noise component of light in p-DI pairs. The DINet-PS was trained with p-DI pairs acquired from 5311 CD4+ T, 3819 CD8+ T, and 4054 CD19+ B cells after preprocessing and rebelling of manually derived secondary labels and classification accuracy of 96.6 ± 0.40% has been achieved in hold-out test data sets among the three subtypes. Our results show the power of DNN to extract cell-related features from p-DI pairs and the potential of polarization diffraction imaging flow cytometry for accurate and label-free classification of lymphocyte subtypes in particular and leukocytes in general.

Development and Validation of a Routine Electronic Health Record-Based Delirium Prediction Model for Surgical Patients Without Dementia: Retrospective Case-Control Study.

Postoperative delirium (POD) is a common complication after major surgery and is associated with poor outcomes in older adults. Early identification of patients at high risk of POD can enable targeted prevention efforts. However, existing POD prediction models require inpatient data collected during the hospital stay, which delays predictions and limits scalability. This study aimed to develop and externally validate a machine learning-based prediction model for POD using routine electronic health record (EHR) data. We identified all surgical encounters from 2014 to 2021 for patients aged 50 years and older who underwent an operation requiring general anesthesia, with a length of stay of at least 1 day at 3 Indiana hospitals. Patients with preexisting dementia or mild cognitive impairment were excluded. POD was identified using Confusion Assessment Method records and delirium International Classification of Diseases (ICD) codes. Controls without delirium or nurse-documented confusion were matched to cases by age, sex, race, and year of admission. We trained logistic regression, random forest, extreme gradient boosting (XGB), and neural network models to predict POD using 143 features derived from routine EHR data available at the time of hospital admission. Separate models were developed for each hospital using surveillance periods of 3 months, 6 months, and 1 year before admission. Model performance was evaluated using the area under the receiver operating characteristic curve (AUROC). Each model was internally validated using holdout data and externally validated using data from the other 2 hospitals. Calibration was assessed using calibration curves. The study cohort included 7167 delirium cases and 7167 matched controls. XGB outperformed all other classifiers. AUROCs were highest for XGB models trained on 12 months of preadmission data. The best-performing XGB model achieved a mean AUROC of 0.79 (SD 0.01) on the holdout set, which decreased to 0.69-0.74 (SD 0.02) when externally validated on data from other hospitals. Our routine EHR-based POD prediction models demonstrated good predictive ability using a limited set of preadmission and surgical variables, though their generalizability was limited. The proposed models could be used as a scalable, automated screening tool to identify patients at high risk of POD at the time of hospital admission.

Engineering of Generative Artificial Intelligence and Natural Language Processing Models to Accurately Identify Arrhythmia Recurrence.

Large language models (LLMs), such as ChatGPT, excel at interpreting unstructured data from public sources, yet are limited when responding to queries on private repositories, such as electronic health records (EHRs). We hypothesized that prompt engineering could enhance the accuracy of LLMs for interpreting EHR data without requiring domain knowledge, thus expanding their utility for patients and personalized diagnostics. We designed and systematically tested prompt engineering techniques to improve the ability of LLMs to interpret EHRs for nuanced diagnostic questions, referenced to a panel of medical experts. In 490 full-text EHR notes from 125 patients with prior life-threatening heart rhythm disorders, we asked GPT-4-turbo to identify recurrent arrhythmias distinct from prior events and tested 220 563 queries. To provide context, results were compared with rule-based natural language processing and BERT-based language models. Experiments were repeated for 2 additional LLMs. In an independent hold-out set of 389 notes, GPT-4-turbo had a balanced accuracy of 64.3%±4.7% out-of-the-box at baseline. This increased when asking GPT-4-turbo to provide a rationale for its answers, requiring a structured data output, and providing in-context exemplars, rose to a balanced accuracy of 91.4%±3.8% (P<0.05). This surpassed the traditional logic-based natural language processing and BERT-based models (P<0.05). Results were consistent for GPT-3.5-turbo and Jurassic-2 LLMs. The use of prompt engineering strategies enables LLMs to identify clinical end points from EHRs with an accuracy that surpassed natural language processing and approximated experts, yet without the need for expert knowledge. These approaches could be applied to LLM queries for other domains, to facilitate automated analysis of nuanced data sets with high accuracy by nonexperts.

Abstract 4125281: Detection of Acute Myocardial Infarction and Ischemia From Lead-I ECG Using Deep Learning

Introduction: While smartwatches that acquire Lead-I electrocardiogram (ECG) have shown promise in detecting arrhythmias, the detection of ischemic events using Lead-I alone remains understudied. We aim to build and evaluate a deep-learning model to determine whether Lead-I ECG can identify ischemia/infarction in a 12-lead ECG. Methods: We developed a deep-learning model that can identify MI/ischemia using only Lead-I ECG signal as input. For this task, we utilized PTB-XL, a large publicly available dataset of 12-Lead clinical ECG with associated diagnostic labels. Our experiment utilized 8781 ECGs whose labels were validated by at-least one cardiologist with 100% confidence scores. This dataset includes 1391 ECGs that are labeled as either ischemic ST/T changes or MI, and 7390 ECGs with either “normal” labels or non-ischemic ST/T abnormalities. We split the dataset in training, validation, and holdout sets (80-10-10) with equal stratification of the diagnosis classes. The holdout test set contains 909 non-ischemic ECGs and 233 ECGs with MI/ischemia diagnosis, 106 of which are labeled as acute MI or ischemic ST/T changes and 127 labeled as other MI chronicity. The discriminatory ability of the model is evaluated with the Area Under the receiver-operating Curve (AUC), sensitivity, specificity, and the predictive values. We further analyze the model’s capability in discriminating the cohort of acute MI and ischemia diagnoses from other MI as well as from the non-ischemic cohort. Results: The model has an AUC of 0.88 for discriminating an MI/ischemia vs normal. Using a threshold corresponding to 80% sensitivity, the corresponding specificity is 79% and the negative and positive predictive values are 94% and 50% (Table-1). Although the model was trained to identify the combined class of MI/ischemia, it can differentiate acute MI and ischemia cohorts from other MI as well as from non-ischemic diagnoses with p-values&lt;10 -4 (Table-2). Lastly, with the same threshold, the sensitivity for detecting acute inferior myocardial infarctions or inferior ischemia – a class that is difficult to detect with Lead-I alone – is 82%. Conclusions: Our deep learning model, which was trained on Lead-I ECG alone, can effectively detect acute MI and ischemia that would have been identified using a 12-lead ECG. This method presents a proof-of-concept for applications in home settings. Further prospective validation using wearable ECG monitors that record lead-I is warranted.

Abstract 4138373: Machine learning identifies clinically distinct phenotypes in patients with aortic regurgitation.

Background: Aortic regurgitation (AR) is a prevalent valve disease with a long latent period to symptoms. Recent data has suggested the role of novel markers of myocardial overload in assessing onset of decompensation. Method: We sought to evaluate the role of unsupervised cluster analyses in identifying different clinical clusters, including clinical status, and a large number of echocardiographic variables including left ventricular (LV) volumes, and their association with mortality. Patients with ≥moderate-severe chronic AR identified using echocardiography at Mayo Clinic, Rochester were retrospectively analyzed. Primary outcome was all-cause mortality censored at aortic valve surgery/last follow-up. Uniform Manifold Approximation and Projection (UMAP) with K -means algorithm was used to cluster patients using clinical and, echocardiographic variables at the time of presentation. Missing data were imputed with the Multiple Imputation by Chained Equations (MICE) method. A supervised approach trained on the training set was used to find cluster membership in a hold-out validation set. Log-rank tests were used to assess differences in mortality rates between the clusters, both in the training and validation sets. Results: Three distinct clusters were identified among 1100 patients (log-rank P for survival &lt;0.001). Cluster 1 (n=337), which included younger males with severe AR but fewer symptoms, showed the best survival, 75.6% (69.5, 82.3). Cluster 2 (n=235), older and more females with elevated filling pressures, showed intermediate survival of 64.2 % (56.8, 72.5). Cluster 3 (n=253), characterized by severe symptomatic AR, demonstrated the lowest survival of 45.3 % (34.4, 59.8) at 5 years. Similar clusters were formed in the internal validation cohort. Conclusion: Distinct clusters with variable echocardiographic features and mortality differences exist within patients with chronic ≥moderate-severe AR. Recognizing these clusters can refine individual risk stratification and clinical decision-making after verification in future prospective studies.

NIMG-66. DEFINING PROGRESSIVE DISEASE: LOCALIZING VOXEL-AT-RISK ON MAGNETIC RESONANCE IMAGING USING DEEP LEARNING

Abstract Follow-up magnetic resonance imaging (MRI) of post-treatment glioblastoma (GBM) can be diagnostically challenging in differentiating progressive disease (PD) from coexisting treatment-related changes. The study aim was to train a machine learning model using multiple MRI timepoints from treated GBM patients with subsequent PD or at least six months of tumor-like, persistently stable disease (SD) to estimate PD risk. A retrospective, post-treatment GBM cohort (n=189) with 453 timepoints was identified with available MRI sequences (T1-weighted with and without contrast, T2-weighted, and fluid-attenuated inversion recovery). PD cases (n=154) showed new, enhancing PD followed by death within 3-6 months; timepoints included the MRI on the PD diagnosis date and the one directly prior. SD cases (n=35) exhibited stable, tumor-like enhancement over timepoints. Cases underwent voxel-wise annotation of PD versus SD. A multiclass, three-dimensional nnU-Net was trained to label voxels as PD, SD, or neither using timepoint sequences with five-fold cross-validation. Model performance was assessed using the ratio of predicted PD/SD voxels compared to each case’s actual PD or SD status. Classification accuracy of PD versus SD status, PD case-wise sensitivity and specificity, and a Receiver Operating Characteristic (ROC) analysis using the PD/SD ratio were presented. The five-fold model ensemble was evaluated on a holdout test set. Validation set performance showed 74.3% accuracy, 0.73 PD case-wise sensitivity, and 0.80 specificity. The holdout set exhibited 79.8% accuracy, 0.81 sensitivity, and 0.78 specificity for PD cases. The ROC analysis showed an area under the curve of 0.80 and 0.83 on the validation and holdout sets, respectively. The model showed high performance and classification accuracy in predicting which voxels were at risk of PD in post-treatment GBM cases. The model also differentiated between PD and SD regions. The model’s predicted PD/SD ratio may be clinically used to stratify patients’ PD risk.

NIMG-75. SIMILARITY DISTANCES-BASED DELTA RADIOMICS (SDDR) ENABLES LONGITUDINAL MRI COMPARISONS FOR SURVIVAL PREDICTION IN GLIOMAS: A MULTI-INSTITUTIONAL VALIDATION

Abstract Radiomics (computerized feature analysis) on treatment-naive MRI scans has demonstrated great value in outcome prediction for Glioblastoma (GBM). However, delta radiomics (analysis of radiomic feature variation between different events, e.g., before and after treatment) has not been explored on account of challenges with precise spatial correspondences between pre- and post-operative scans. This work presents a survival prediction model for GBM, Similarity Distances-based delta radiomics (SDDR), based on the hypothesis that similarity distance-measures between radiomic features on pre-, and post-treatment images can accurately capture temporal changes within-and-around the tumor, and provide improved prediction of overall survival, compared to models based on a single timepoint. 150 patients from 2 publicly-available Glioma repositories (University of California San Francisco “UCSF” (n=48), LUMIERE (n=67)) and 2 two proprietary datasets (from xCures (n=13), and Cleveland Clinic (CCF) (n=74)) were obtained. UCSF and LUMIERE cohorts were used as separate hold-out sets, in each of which the remaining collections were used as discovery set. Following pre-processing, tumor segmentation into Edema (ED) and Enhancing tumor (ET) were performed. SDDR was then computed, which is the distance between probability densities functions from 156 gray co-occurrence (GLCM) and 52 gradient-based co-occurrences statistics (COLLAGE) features extracted from pre and-post-treatment MRI. Six statistical distances (e.g., Earth mover distance) normalized by the scanning interval (in weeks) yielded n=2496 SDDR measurements per tumor region (1248/region, 936 delta-GLCM, and 312 delta-COLLAGE). The same set of features were computed from pretreatment MRI (Pre-R) for comparison. Features were fed to LASSO-Cox regression for survival analysis. SDDR for GLCM features extracted from ET outperformed Pre-R, and demonstrated significant differences between high and low risk groups for both hold out tests: UCSF (p=0.0069, HR =2.8, C-index=0.69) and LUMIERE (p=0.0061, HR = 1.9, C-index 0.65). SDDR-based multi-institutional survival analysis demonstrated promise in reliable longitudinal risk-stratification in GBM.

Authenticity markers of cultivated blueberries (Vaccinium sp.) and European wild blueberry (V. myrtillus) assessed by hyperspectral imaging and machine learning algorithms

The present study was undertaken to tackle food fraud in the food chain by developing a novel panel of analytical fingerprints to distinguish the fruit of bilberry (Vaccinium myrtillus), a wild European blueberry grown in Finland, from their commercial cultivated counterparts based on V. corymbosum or other originally American Vaccinium species. The analytical fingerprints were derived using hyperspectral imaging (HSI). The cultivated domestic blueberry samples from Finland (N = 188), those imported (N = 338), and domestic Finnish wild bilberry fruit (N = 247) were imaged using Specim IQ and FX17 HSI cameras. The fruit HSI data were then classified using seven machine learning model types. The supervised evolutionary feature selection algorithm was used to recognize the most separating subset of wavelengths (nm predictors, colors) between the fruit class labels, which were 1) wild bilberry fruit vs. cultivated blueberry fruit and 2) domestic cultivated blueberry fruit vs. imported cultivated blueberry fruit. Moreover, we ran 190,822 individual models and 54,137 predictor combinations to determine whether subsets or all wavelengths better distinguished fruit class labels. Overall, the best model type, the rbf kernel support vector machine (rbf-SVM), showed excellent prediction performance when classifying wild bilberries vs. cultivated blueberries with HSI data obtained using both camera types (4.1% and 3.2% holdout set errors). Also the rbf-SVM performed well when classifying domestic cultivated blueberries vs. those imported with HSI data obtained using an IQ camera (2.3% holdout set error). Our results showed that hyperspectral imaging could predict food fraud attempts concerning the Vaccinium fruit of different origin and species.

Machine Learning Identifies Clinically Distinct Phenotypes in Patients With Aortic Regurgitation.

Aortic regurgitation (AR) is a prevalent valve disease with a long latent period before symptoms appear. Recent data has suggested the role of novel markers of myocardial overload in assessing onset of decompensation. The aim of this study was to evaluate the role of unsupervised cluster analyses in identifying different clinical clusters, including clinical status, and a large number of echocardiographic variables including left ventricular volumes, and their associations with mortality. Patients with moderate to severe or greater chronic AR identified using echocardiography at the Mayo Clinic in Rochester, Minnesota, were retrospectively analyzed. The primary outcome was all-cause mortality censored at aortic valve surgery. Uniform manifold approximation and projection with the k-means algorithm was used to cluster patients using clinical and echocardiographic variables at the time of presentation. Missing data were imputed using the multiple imputation by chained equations method. A supervised approach trained on the training set was used to find cluster membership in a hold-out validation set. Log-rank tests were used to assess differences in mortality rates among the clusters in both the training and validation sets. Three distinct clusters were identified among 1,100 patients (log-rank P for survival<.001). Cluster 1 (n=337), which included younger males with severe AR but fewer symptoms, showed the best survival at 75.6% (95% CI, 69.5%-82.3%). Cluster 2 (n=235), including older patients and more females with elevated filling pressures, showed intermediate survival of 64.2% (95% CI, 56.8%-72.5%). Cluster 3 (n=253), characterized by severe symptomatic AR, demonstrated the lowest survival of 45.3% (95% CI, 34.4%-59.8%) at 5years. Similar clusters were identified in the internal validation cohort. Distinct clusters with variable echocardiographic features and mortality differences exist within patients with chronicmoderate to severe or greater AR. Recognizing these clusters can refine individual risk stratification and clinical decision-making after verification in future prospective studies.

Using intrahost single nucleotide variant data to predict SARS-CoV-2 detection cycle threshold values.

Over the last four years, each successive wave of the COVID-19 pandemic has been caused by variants with mutations that improve the transmissibility of the virus. Despite this, we still lack tools for predicting clinically important features of the virus. In this study, we show that it is possible to predict the PCR cycle threshold (Ct) values from clinical detection assays using sequence data. Ct values often correspond with patient viral load and the epidemiological trajectory of the pandemic. Using a collection of 36,335 high quality genomes, we built models from SARS-CoV-2 intrahost single nucleotide variant (iSNV) data, computing XGBoost models from the frequencies of A, T, G, C, insertions, and deletions at each position relative to the Wuhan-Hu-1 reference genome. Our best model had an R2 of 0.604 [0.593-0.616, 95% confidence interval] and a Root Mean Square Error (RMSE) of 5.247 [5.156-5.337], demonstrating modest predictive power. Overall, we show that the results are stable relative to an external holdout set of genomes selected from SRA and are robust to patient status and the detection instruments that were used. This study highlights the importance of developing modeling strategies that can be applied to publicly available genome sequence data for use in disease prevention and control.

Comprehensive machine learning models for prediction of heart failure in 476,393 women and men from the UK Biobank reveal sex differences and underutilized risk factors

Abstract Background Machine learning (ML) models provide potential advantage over ‘traditional’ regression models in heart failure (HF) prediction. Objective To compare performances of Cox PH models and ML survival models for incident HF in men and women without prevalent ischemic heart disease (IHD). We also aimed to identify potential high-risk precursors otherwise ignored by conventional survival models, and to investigate differences between sex-specific models. Methods We included 476,393 participants (55.6% women) from the UK Biobank, after excluding participants with a history of HF or IHD, and defined sex-specific datasets. We predicted incident HF events using over 400 baseline characteristics. We constructed multivariable Cox PH models, which included all predictor variables and subsequently only those remaining after LASSO stability selection. We also developed two supervised ML models (Random Survival Forest (RSF), eXtreme Gradient Survival Boosting (XGBoost)). We identified the 15 most important sex-specific predictors in each model and performances were compared using the C-index. Models were validated using hold-out sets. Results During 12.3 ± 1.9 years of follow-up, 4680 (1.76%) women and 6631 (3.14%) men developed incident HF. XGBoost showed the best performance during model training (C-index, training: 0.89 in men, 0.97 in women; validation 0.77 in men, 0.80 in women). The multivariable Cox model performed second-best (C-index, training: 0.78 in men, 0.82 in women; validation: 0.76 in men, 0.78 in women). RSF performed slightly worse (C-index, training: 0.75 in men, 0.79 in women; validation: 0.75 in men, 0.79 in women) but did not show performance drop during validation. LASSO stability selection performed similar to RSF. Age, self-reported lifetime treatments and medications, cystatin-C, waist circumference and FEV1-scores were identified as strong risk factors in all models for both sexes. Reduced albumin levels and elevated HbA1c were more strongly associated with high risk in men, while elevated systolic BP showed higher importance in women. Traditional Cox models observed CRP as important only in men, while the ML models identified CRP as important for both sexes. Neutrophil count was considered a strong risk factor in both sexes in the traditional Cox models, yet it was not among the most important predictors in both ML models. Presence of other heart disease (which included a.o. pericardial disease, valve disorders and arrhythmias) was an important predictor variable only in the ML models. Conclusion ML models showed similar performance to Cox PH models for HF prediction. Despite this, differences in predictor importance were identified between models. Sex-specific risk predictors were found, and FEV1 score, which is not commonly included in existing models, was identified as an important risk factor. These results suggest that ML models may reveal additional insights that would otherwise remain unnoticed.

Towards the prediction of drug solubility in binary solvent mixtures at various temperatures using machine learning

Drug solubility is an important parameter in the drug development process, yet it is often tedious and challenging to measure, especially for expensive drugs or those available in small quantities. To alleviate these challenges, machine learning (ML) has been applied to predict drug solubility as an alternative approach. However, the majority of existing ML research has focused on the predictions of aqueous solubility and/or solubility at specific temperatures, which restricts the model applicability in pharmaceutical development. To bridge this gap, we compiled a dataset of 27,000 solubility datapoints, including solubility of small molecules measured in a range of binary solvent mixtures under various temperatures. Next, a panel of ML models were trained on this dataset with their hyperparameters tuned using Bayesian optimization. The resulting top-performing models, both gradient boosted decision trees (light gradient boosting machine and extreme gradient boosting), achieved mean absolute errors (MAE) of 0.33 for LogS (S in g/100 g) on the holdout set. These models were further validated through a prospective study, wherein the solubility of four drug molecules were predicted by the models and then validated with in-house solubility experiments. This prospective study demonstrated that the models accurately predicted the solubility of solutes in specific binary solvent mixtures under different temperatures, especially for drugs whose features closely align within the solutes in the dataset (MAE < 0.5 for LogS). To support future research and facilitate advancements in the field, we have made the dataset and code openly available.Scientific contributionOur research advances the state-of-the-art in predicting solubility for small molecules by leveraging ML and a uniquely comprehensive dataset. Unlike existing ML studies that predominantly focus on solubility in aqueous solvents at fixed temperatures, our work enables prediction of drug solubility in a variety of binary solvent mixtures over a broad temperature range, providing practical insights on the modeling of solubility for realistic pharmaceutical applications. These advancements along with the open access dataset and code support significant steps in the drug development process including new molecule discovery, drug analysis and formulation.Graphical

An artificial intelligence-based landmark model for early detection of aortic stenosis in Cardiac MRI

Abstract Background The utilisation of ratiometric intensity measures from cardiac magnetic resonance imaging (CMR) offers a promising biomarker for the diagnosis of aortic stenosis (AS)[1]. The Ao:LV ratio, an approximation of AS severity, is determined by comparing the relative blood signal intensity in the aorta and left ventricle (LV) within three-chamber (3Ch) cine sequences[1]. In patients with AS, the blood signal intensity of aortic blood above the valve is diminished compared to that within the LV in steady-state free precession (SSFP) 3Ch CMR cine. Although the Ao:LV ratio serves as a reliable indicator of AS severity, manual computation of this ratio may encounter difficulties due to visual obstructions caused by small LV cavities or prominent papillary muscles. Purpose This study aims to leverage artificial intelligence (AI) to automate the Ao:LV ratio analysis from 3Ch SSFP cine images and introduce the Ao:LA (aorta to left atrium) ratio, thus enhancing the diagnostic accuracy for AS severity. Methods Our methodology extends the ratiometric analysis to include both Ao:LV and Ao:LA ratios in three main steps: first, developing an AI-based algorithm for automated detection and tracking of cardiac landmarks; second, using these landmarks to automatically compute regions of interest (ROIs) in the cardiac chambers; and third, training and validating classification models with 5-fold cross-validation to select the most effective for AS severity classification. The final model's performance is assessed on a holdout test set. Data for model training and evaluation were sourced from CMR scans across three hospitals and two scanner types. Results We analysed 220 patients of which 78 patients had no AS and 122 had AS (mild AS, n = 29; moderate AS; n = 35, severe AS; n = 78). Our model yielded high efficacy in classifying AS, particularly in differentiating any grade of AS from a normal classification, which is clinically relevant. The Ao:LA ratio demonstrated a stronger correlation with AS severity class than Ao:LV, suggesting a more reliable biomarker. The proposed model demonstrated robust performance in AS classification, achieving an Area Under the Curve (AUC) of 0.845 in the automated binary classification of AS at any grade. The model exhibited a precision of 0.889, recall of 0.865, and an F1 score of 0.877 on a holdout set. Conclusion By the integration of AI in automating the analysis of CMR images through a novel ratiometric approach that includes Ao:LV and Ao:LA ratios, our model significantly improves the accuracy of early AS detection. This advancement enhances imaging strategies for AS evaluation, promoting more efficient and effective use of cardiac MRI sequences as a diagnostic tool for aortic stenosis. It has the potential to be implemented in real-time scanning protocols as a clinical decision support system.AI-inferred cardiac landmarks.ROC curve delineating AS prediction.

Serum metabolite and metal ions profiles for breast cancer screening

Enhancing early-stage breast cancer detection requires integrating additional screening methods with current diagnostic imaging. Omics screening, using easily collectible serum samples, could serve as an initial step. Alongside biomarker identification capabilities, omics analysis allows for a comprehensive analysis of prevalent histological types—DCIS and IDC. Employing metabolomics, metallomics, and machine learning, could yield accurate screening models with valuable insights into organism responses. Serum samples of confirmed breast cancer patients were utilized to analyze metabolite and metal ion profiles, using two distinct analysis methods, proton NMR for metabolomics and ICP-OES for metallomics. The resulting responses were then subjected to discriminant analysis, progression biomarker exploration, examination of correlations between patients’ metabolites and metal ions, and the impact of age and menopause status. Measured NMR spectra and metabolite relative integrals were used to achieve statistically significant discrimination through MVA between breast cancer and control groups. The analysis identified 24 metabolites and 4 metal ions crucial for discrimination. Furthermore, four metabolites were associated with disease progression. Additionally, there were important correlations and relationships between metabolite relative integrals, metal ion concentrations, and age/menopausal status subgroups. Quantified relative integrals allowed for discrimination between studied subgroups, validated with a holdout set. Feature importance and statistical analysis for metabolomics and metallomics extracted a set of common entities which in combination provides valuable insights into ongoing molecular disturbances and disease progression.

PRECISE-GBM: A RADIOGENOMIC MACHINE LEARNING BASED STUDY TO IDENTIFY PREDICTIVE RADIOMICS FOR EVALUATION OF CANCER IMMUNE SIGNATURE IN IDHW GLIOBLASTOMA

Abstract AIMS The aim of the study is to identify machine learning model based on radiological biomarkers that can predict immune status within tumor microenvironment of IDH wildtype glioblastoma. METHOD This is a retrospective multicenter machine learning based study utilizing open access anonymized matched radiogenomic data of TCGA-GBM, CPTAC, Ivy GAP and REMBRANDT datasets. REMBRANDT data acted as hold out dataset. Imaging data consisted of MRI based radiomics features extracted from deep learning based segmented tumors and transcriptomic (RNA seq/microarray) data consisted of immune scores extracted using CIBERSORTx. Outliers were identified and filtered using principal component analysis. Combat and neuroCombat were utilized for data harmonization for each data type respectively. Radiomic feature selection was performed using LASSO technique with cross validation. Support vector machine (SVM) was trained and tested with 80:20 radiomics data split and a median based binary label of immune score (low and high). Validation of the trained model was performed with hold out dataset. Data query and analysis was performed using python and R statistical software. RESULTS One hundred one patients were included in the study with 15 being on the hold out set. Trained SVM model predicted the immune score label with AUC of 0.79 and balanced accuracy of 0.76 (recall 0.75, F1 score 0.75, precision 0.75) in the test data. In the holdout set, the model predicted the immune score with AUC of 0.70 with specificity of 75% in predicting the cold immune cases. The radiomic signature that acted as the label for this model were a group of first order and second order radiomic features. CONCLUSION Radiogenomic SVM model non-invasively predicts immune status in wildtype glioblastoma microenvironment with good accuracy. This model has potential to be utilized in a prospective glioblastoma clinical trial for immunotherapy.

Ethical considerations of use of hold-out sets in clinical prediction model management

AbstractClinical prediction models are statistical or machine learning models used to quantify the risk of a certain health outcome using patient data. These can then inform potential interventions on patients, causing an effect called performative prediction: predictions inform interventions which influence the outcome they were trying to predict, leading to a potential underestimation of risk in some patients if a model is updated on this data. One suggested resolution to this is the use of hold-out sets, in which a set of patients do not receive model derived risk scores, such that a model can be safely retrained. We present an overview of clinical and research ethics regarding potential implementation of hold-out sets for clinical prediction models in health settings. We focus on the ethical principles of beneficence, non-maleficence, autonomy and justice. We also discuss informed consent, clinical equipoise, and truth-telling. We present illustrative cases of potential hold-out set implementations and discuss statistical issues arising from different hold-out set sampling methods. We also discuss differences between hold-out sets and randomised control trials, in terms of ethics and statistical issues. Finally, we give practical recommendations for researchers interested in the use hold-out sets for clinical prediction models.

Development and testing of a deep learning algorithm to detect lung consolidation among children with pneumonia using hand-held ultrasound.

Severe pneumonia is the leading cause of death among young children worldwide, disproportionately impacting children who lack access to advanced diagnostic imaging. Here our objectives were to develop and test the accuracy of an artificial intelligence algorithm for detecting features of pulmonary consolidation on point-of-care lung ultrasounds among hospitalized children. This was a prospective, multicenter center study conducted at academic Emergency Department and Pediatric inpatient or intensive care units between 2018-2020. Pediatric participants from 18 months to 17 years old with suspicion of lower respiratory tract infection were enrolled. Bedside lung ultrasounds were performed using a Philips handheld Lumify C5-2 transducer and standardized protocol to collect video loops from twelve lung zones, and lung features at both the video and frame levels annotated. Data from both affected and unaffected lung fields were split at the participant level into training, tuning, and holdout sets used to train, tune hyperparameters, and test an algorithm for detection of consolidation features. Data collected from adults with lower respiratory tract disease were added to enrich the training set. Algorithm performance at the video level to detect consolidation on lung ultrasound was determined using reference standard diagnosis of positive or negative pneumonia derived from clinical data. Data from 107 pediatric participants yielded 117 unique exams and contributed 604 positive and 589 negative videos for consolidation that were utilized for the algorithm development process. Overall accuracy for the model for identification and localization of consolidation was 88.5%, with sensitivity 88%, specificity 89%, positive predictive value 89%, and negative predictive value 87%. Our algorithm demonstrated high accuracy for identification of consolidation features on pediatric chest ultrasound in children with pneumonia. Automated diagnostic support on an ultraportable point-of-care device has important implications for global health, particularly in austere settings.

A radiomics nomogram based on multiparametric MRI for diagnosing focal cortical dysplasia and initially identifying laterality

BackgroundFocal cortical dysplasia (FCD) is the most common epileptogenic developmental malformation. The diagnosis of FCD is challenging. We generated a radiomics nomogram based on multiparametric magnetic resonance imaging (MRI) to diagnose FCD and identify laterality early.MethodsForty-three patients treated between July 2017 and May 2022 with histopathologically confirmed FCD were retrospectively enrolled. The contralateral unaffected hemispheres were included as the control group. Therefore, 86 ROIs were finally included. Using January 2021 as the time cutoff, those admitted after January 2021 were included in the hold-out set (n = 20). The remaining patients were separated randomly (8:2 ratio) into training (n = 55) and validation (n = 11) sets. All preoperative and postoperative MR images, including T1-weighted (T1w), T2-weighted (T2w), fluid-attenuated inversion recovery (FLAIR), and combined (T1w + T2w + FLAIR) images, were included. The least absolute shrinkage and selection operator (LASSO) was used to select features. Multivariable logistic regression analysis was used to develop the diagnosis model. The performance of the radiomic nomogram was evaluated with an area under the curve (AUC), net reclassification improvement (NRI), integrated discrimination improvement (IDI), calibration and clinical utility.ResultsThe model-based radiomics features that were selected from combined sequences (T1w + T2w + FLAIR) had the highest performances in all models and showed better diagnostic performance than inexperienced radiologists in the training (AUCs: 0.847 VS. 0.664, p = 0.008), validation (AUC: 0.857 VS. 0.521, p = 0.155), and hold-out sets (AUCs: 0.828 VS. 0.571, p = 0.080). The positive values of NRI (0.402, 0.607, 0.424) and IDI (0.158, 0.264, 0.264) in the three sets indicated that the diagnostic performance of Model-Combined improved significantly. The radiomics nomogram fit well in calibration curves (p > 0.05), and decision curve analysis further confirmed the clinical usefulness of the nomogram. Additionally, the contrast (the radiomics feature) of the FCD lesions not only played a crucial role in the classifier but also had a significant correlation (r = -0.319, p < 0.05) with the duration of FCD.ConclusionThe radiomics nomogram generated by logistic regression model-based multiparametric MRI represents an important advancement in FCD diagnosis and treatment.

Predicting Long COVID in the National COVID Cohort Collaborative Using Super Learner: Cohort Study.

Postacute sequelae of COVID-19 (PASC), also known as long COVID, is a broad grouping of a range of long-term symptoms following acute COVID-19. These symptoms can occur across a range of biological systems, leading to challenges in determining risk factors for PASC and the causal etiology of this disorder. An understanding of characteristics that are predictive of future PASC is valuable, as this can inform the identification of high-risk individuals and future preventative efforts. However, current knowledge regarding PASC risk factors is limited. Using a sample of 55,257 patients (at a ratio of 1 patient with PASC to 4 matched controls) from the National COVID Cohort Collaborative, as part of the National Institutes of Health Long COVID Computational Challenge, we sought to predict individual risk of PASC diagnosis from a curated set of clinically informed covariates. The National COVID Cohort Collaborative includes electronic health records for more than 22 million patients from 84 sites across the United States. We predicted individual PASC status, given covariate information, using Super Learner (an ensemble machine learning algorithm also known as stacking) to learn the optimal combination of gradient boosting and random forest algorithms to maximize the area under the receiver operator curve. We evaluated variable importance (Shapley values) based on 3 levels: individual features, temporal windows, and clinical domains. We externally validated these findings using a holdout set of randomly selected study sites. We were able to predict individual PASC diagnoses accurately (area under the curve 0.874). The individual features of the length of observation period, number of health care interactions during acute COVID-19, and viral lower respiratory infection were the most predictive of subsequent PASC diagnosis. Temporally, we found that baseline characteristics were the most predictive of future PASC diagnosis, compared with characteristics immediately before, during, or after acute COVID-19. We found that the clinical domains of health care use, demographics or anthropometry, and respiratory factors were the most predictive of PASC diagnosis. The methods outlined here provide an open-source, applied example of using Super Learner to predict PASC status using electronic health record data, which can be replicated across a variety of settings. Across individual predictors and clinical domains, we consistently found that factors related to health care use were the strongest predictors of PASC diagnosis. This indicates that any observational studies using PASC diagnosis as a primary outcome must rigorously account for heterogeneous health care use. Our temporal findings support the hypothesis that clinicians may be able to accurately assess the risk of PASC in patients before acute COVID-19 diagnosis, which could improve early interventions and preventive care. Our findings also highlight the importance of respiratory characteristics in PASC risk assessment. RR2-10.1101/2023.07.27.23293272.