Hold-out Set Research Articles

DeepEOR: automated perioperative volumetric assessment of variable grade gliomas using deep learning

PurposeVolumetric assessments, such as extent of resection (EOR) or residual tumor volume, are essential criterions in glioma resection surgery. Our goal is to develop and validate segmentation machine learning models for pre- and postoperative magnetic resonance imaging scans, allowing us to assess the percentagewise tumor reduction after intracranial surgery for gliomas.MethodsFor the development of the preoperative segmentation model (U-Net), MRI scans of 1053 patients from the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2021 as well as from patients who underwent surgery at the University Hospital in Zurich were used. Subsequently, the model was evaluated on a holdout set containing 285 images from the same sources. The postoperative model was developed using 72 scans and validated on 45 scans obtained from the BraTS 2015 and Zurich dataset. Performance is evaluated using Dice Similarity score, Jaccard coefficient and Hausdorff 95%.ResultsWe were able to achieve an overall mean Dice Similarity Score of 0.59 and 0.29 on the pre- and postoperative holdout sets, respectively. Our algorithm managed to determine correct EOR in 44.1%.ConclusionAlthough our models are not suitable for clinical use at this point, the possible applications are vast, going from automated lesion detection to disease progression evaluation. Precise determination of EOR is a challenging task, but we managed to show that deep learning can provide fast and objective estimates.

From 2D projections to the 3D rotation matrix: an attempt for finding a machine learning approach for the efficient evaluation of mechanical joining elements in X-ray computed tomography volume data

Destructive and predominantly manual procedures are commonly used in the automotive industry for the testing of mechanical joints, such as rivets or screws. Combining X-ray computed tomography (CT) and machine learning (ML) bears the potential of a non-destructive and largely automated methodology. Assuming the desired result is a comprehensible and documentable evaluation, three basic steps need to be automatized: First, a joint must be detected and identified as such in a CT scan of the joined parts. Second, the detected region containing the joint is rotated to a predefined orientation. Third, key measures in cross-sections from the newly oriented joint are dimensioned and documented. This work deals only with the second step, the rotation. On the one hand, we present a methodology for creating a well-curated data set for the contextual machine learning application. On the other, we evaluate its performance on the well-known ResNet50. More concretely, we investigate if it is possible for a deep convolutional neural network (CNN) to learn the respective rotation matrix from three volume projections that are perpendicular to each other. Two scenarios are investigated: In one scenario we assume that future data that is presented to the network has similar rivet demographics to historic data. We therefore do not employ hold-out sets for the network evaluation. In the other scenario we assume the opposite and therefore evaluating the networks performance with hold-out sets. We show that from a machine learning point of view, a CNN like ResNet50 is well able to learn this relationship with acceptable accuracy. In most cases the validation loss dropped below 0.1 after only a couple of epochs. In one particular case, we even reached both mean and median errors lower than 0.2 for approximately 80% of the entire test set of 1600 examples using our methodology. From an application point of view, however, these low test set errors should be treated with caution since small deviations from the intended rotation matrix can cause volume warping and translation. In another case, in which we used a hold-out set, only a fraction of the median errors were below 0.2.

Deep learning-based optical coherence tomography image analysis of human brain cancer.

Real-time intraoperative delineation of cancer and non-cancer brain tissues, especially in the eloquent cortex, is critical for thorough cancer resection, lengthening survival, and improving quality of life. Prior studies have established that thresholding optical attenuation values reveals cancer regions with high sensitivity and specificity. However, threshold of a single value disregards local information important to making more robust predictions. Hence, we propose deep convolutional neural networks (CNNs) trained on labeled OCT images and co-occurrence matrix features extracted from these images to synergize attenuation characteristics and texture features. Specifically, we adapt a deep ensemble model trained on 5,831 examples in a training dataset of 7 patients. We obtain 93.31% sensitivity and 97.04% specificity on a holdout set of 4 patients without the need for beam profile normalization using a reference phantom. The segmentation maps produced by parsing the OCT volume and tiling the outputs of our model are in excellent agreement with attenuation mapping-based methods. Our new approach for this important application has considerable implications for clinical translation.

Chest X-ray-based opportunistic screening of sarcopenia using deep learning.

Early detection and management of sarcopenia is of clinical importance. We aimed to develop a chest X-ray-based deep learning model to predict presence of sarcopenia. Data of participants who visited osteoporosis clinic at Severance Hospital, Seoul, South Korea, between January 2020 and June 2021 were used as derivation cohort as split to train, validation and test set (65:15:20). A community-based older adults cohort (KURE) was used as external test set. Sarcopenia was defined based on Asian Working Group 2019 guideline. A deep learning model was trained to predict appendicular lean mass (ALM), handgrip strength (HGS) and chair rise test performance from chest X-ray images; then the machine learning model (SARC-CXR score) was built using the age, sex, body mass index and chest X-ray predicted muscle parameters along with estimation uncertainty values. Mean age of the derivation cohort (n=926; women n=700, 76%; sarcopenia n=141, 15%) and the external test (n=149; women n=95, 64%; sarcopenia n=18, 12%) cohort was 61.4 and 71.6years, respectively. In the internal test set (a hold-out set, n=189, from the derivation cohort) and the external test set (n=149), the concordance correlation coefficient for ALM prediction was 0.80 and 0.76, with an average difference of 0.18±2.71 and 0.21±2.28, respectively. Gradient-weight class activation mapping for deep neural network models to predict ALM and HGS commonly showed highly weight pixel values at bilateral lung fields and part of the cardiac contour. SARC-CXR score showed good discriminatory performance for sarcopenia in both internal test set [area under the receiver-operating characteristics curve (AUROC) 0.813, area under the precision-recall curve (AUPRC) 0.380, sensitivity 0.844, specificity 0.739, F1-score 0.540] and external test set (AUROC 0.780, AUPRC 0.440, sensitivity 0.611, specificity 0.855, F1-score 0.458). Among SARC-CXR model features, predicted low ALM from chest X-ray was the most important predictor of sarcopenia based on SHapley Additive exPlanations values. Higher estimation uncertainty of HGS contributed to elevate the predicted risk of sarcopenia. In internal test set, SARC-CXR score showed better discriminatory performance than SARC-F score (AUROC 0.813 vs. 0.691, P=0.029). Chest X-ray-based deep leaning model improved detection of sarcopenia, which merits further investigation.

On a quest for solutions

AbstractA plethora of classification models, and especially convolutional neural networks (CNN), have been proposed for the detection of glaucoma from fundus images. Often trained with data from a single glaucoma clinic, they report impressive performance on internal test sets, but (I) lack in objective insights into the decision making process, and (II) rarely generalize well to external sets.Opening the black‐box: need for explainable AIData‐driven, feature‐agnostic CNN models have shown to outperform classical machine learning approaches that rely on manually curated image features. But this comes at the expense of explainability, a concept introduced to delineate the insights into the decision process of the algorithm. Explainable AI is particularly relevant in the field of medicine, as regularity bodies require understandable decisions, in order to gain trust from both clinicians and patients. The largest area of research in explainable AI proposes the framework of saliency or heat maps, i.e. visualizations of image regions that are deemed the most important to the trained model. However, these techniques are facing increased scrutiny over the years, as some studies indicated that most saliency mapping methods perform no better than random baselines.In our recent work, we used various fundus image cropping policies to investigate the salient regions for glaucoma detection. Our findings provide the first irrefutable evidence that deep learning can detect glaucoma from colour fundus image regions outside the ONH.Generalizability and domain shiftsThe ability to generalize, generalizability, to unseen data is of crucial importance in building clinically useful deep learning models. This can be defined on two levels. First, a model is deemed useful when it performs well on unseen data from the same domain as the training data. More concretely, this would mean that a model trained on glaucoma data from one glaucoma center works well on a holdout set of images from the same clinic. Next to internal testing, it is critical to have models that scale well to imaging data from external sources, such as a glaucoma clinic from a different city. This transition introduces a domain shift, an event where the data distribution changes. This shift can be as small as an undetectable different image colour histogram due to other lighting conditions at the external center.Typically, domain shifts are more marked between ophthalmic centers. The usual suspects include the following non‐exhaustive list: change in camera manufacturer, disease prevalence, and demographic characteristics such as ethnicity and age. A tool that would be suited for glaucoma screening should be able to adapt to a much lower glaucoma prevalence, when initially trained on a data set obtained from a referral center. We demonstrate that a deep learning approach backed by regression loss seems more robust to data shifts than a traditional binary classification system for glaucoma screening from fundus images. Excellent results were obtained on 12 different data sets, with close to 150 000 images evaluated.

Machine Learning Techniques Outperform Conventional Statistical Methods in the Prediction of High Risk QTc Prolongation Related to a Drug-Drug Interaction.

In clinical practice, many drug therapies are associated with prolongation of the QT interval. In literature, estimation of the risk of prescribing drug-induced QT prolongation is mainly executed by means of logistic regression; only one paper reported the use of machine learning techniques. In this paper, we compare the performance of both techniques on the same dataset. High risk for QT prolongation was defined as having a corrected QT interval (QTc) ≥ 450ms or ≥ 470ms for respectively male and female patients. Both conventional statistical methods (CSM) and machine learning techniques (MLT) were used. All algorithms were validated internally and with a hold-out dataset of respectively 512 and 102 drug-drug interactions with possible drug-induced QTc prolongation. MLT outperformed the best CSM in both internal and hold-out validation. Random forest and Adaboost classification performed best in the hold-out set with an equal harmonic mean of sensitivity and specificity (HMSS) of 81.2% and an equal accuracy of 82.4% in a hold-out dataset. Sensitivity and specificity were both high (respectively 75.6% and 87.7%). The most important features were baseline QTc value, C-reactive protein level, heart rate at baseline, age, calcium level, renal function, serum potassium level and the atrial fibrillation status. All CSM performed similarly with HMSS varying between 60.3% and 66.3%. The overall performance of logistic regression was 62.0%. MLT (bagging and boosting) outperform CSM in predicting drug-induced QTc prolongation. Additionally, 19.2% was gained in terms of performance by random forest and Adaboost classification compared to logistic regression (the most used technique in literature in estimating the risk for QTc prolongation). Future research should focus on testing the classification on fully external data, further exploring potential of other (new) machine and deep learning models and on generating data pipelines to automatically feed the data to the classifier used.

Matrix factorization for the reconstruction of cervical cancer screening histories and prediction of future screening results

BackgroundMass screening programs for cervical cancer prevention in the Nordic countries have strongly reduced cancer incidence and mortality at the population level. An alternative to the current mass screening is a more personalised screening strategy adapting the recommendations to each individual. However, this necessitates reliable risk prediction models accounting for disease dynamics and individual data. Herein we propose a novel matrix factorisation framework to classify females by the time-varying risk of being diagnosed with cervical cancer. We cast the problem as a time-series prediction model where the data from females in the Norwegian screening population are represented as sparse vectors in time and then combined into a single matrix. Using novel temporal regularisation and discrepancy terms for the cervical cancer screening context, we reconstruct complete screening profiles from this scarce matrix and use these to predict the next exam results indicating the risk of cervical cancer. The algorithm is validated on both synthetic and registry screening data by measuring the probability of agreement (PoA) between Kaplan-Meier estimates.ResultsIn numerical experiments on synthetic data, we demonstrate that the novel regularisation and discrepancy term can improve the data reconstruction ability as well as prediction performance over varying data scarcity. Using a hold-out set of screening data, we compare several numerical models and find that the proposed framework attains the strongest PoA. We observe strong correlations between the empirical survival curves from our method and the hold-out data, and evaluate the ability of our framework to predict the females’ next results for up to five years ahead in time using only their current screening histories as input.ConclusionsWe have proposed a matrix factorization model for predicting future screening results and evaluated its performance in a female cohort to demonstrate the potential for developing prediction models for more personalized cervical cancer screening.

FLIPI24: An Improved International Prognostic Model Developed on Early Events in Follicular Lymphoma

Background: Early events are associated with inferior outcomes in follicular lymphoma (FL) and increased risk of death due to refractory FL. Risk prediction of early events prior to the start of therapy may enhance clinical management and research strategies. We report the FL International Prognostic Index model developed specifically to predict the risk of event within 24 months (EFS24) of starting first-line immunochemotherapy (IC) (FLIPI24). Methods: Modeling was performed using individual patient data from 10 observational cohorts from Europe, North America, and Australia. Eligible patients were diagnosed with grades 1-3A FL and initiated frontline R-CHOP, R-CVP, R-bendamustine (B-R), or like therapies. Event-free survival (EFS) was defined as time from start of IC to progression, retreatment, transformation, or death due to any cause. Model development utilized a truncated Cox model stratified on IC type and cohort with a time-dependent adjustment for maintenance therapy. Multiple imputation was implemented to address missing data. Functional forms of variables were assessed using splines. Model selection was based on model performance, clinical relevance, and parsimony. Gradient boosting machines were evaluated as a complementary approach and used to inform model selection. Model building was performed on an 80% split sample with 20% held for internal testing. External validation occurred in an independent cohort. FLIPI24 risk scores assumed B-R or R-CHOP based chemotherapy and 50% likelihood of maintenance. Results: 4485 patients initiating frontline IC between 2002 and 2018 were utilized in the analysis; 3577 patients were utilized for model building and 908 for internal model testing. Median age at diagnosis in the model build dataset was 61 years (IQR 53-69) and 51% were male. FLIPI was 21%, 32%, and 48% for low, intermediate, and high risk, respectively. IC type was 1874 R-CHOP or like (52%), 730 B-R or like (20%), and 973 R-CVP (27%); 2164 received CD20 antibody maintenance (61%). At median follow-up of 77 months (IQR 48-115), EFS24 estimate was 81% (95% CI: 79-82) and 835 patients (23%) died. 5-year survival after an early (non-death) event was 46% (95% CI: 42-50). The FLIPI24 model utilizes 5 continuous variables: age (linear 60-90 years with inflection at age 75), hemoglobin (linear 8-17 g/dL), white blood cell count (linear 4-11 109/L), normalized lactate dehydrogenase (linear 0.5-5), and beta-2-microglobulin (linear 1-10 mg/L). Linear variables and valid ranges were utilized based on close examination of functional forms across potential models. Continuous FLIPI24 risk scores were grouped for visualization in K-M curves as follows: very low risk (0-0.10), low risk (0.10-0.15), average risk (0.15-0.20), high risk (0.20-0.40), very high risk (>0.40), Figure 1. Internal validation was performed on the 20% holdout set (N=908). The FLIPI24 had superior concordance (c-stat) for EFS24 (c-stat=0.666) compared to FLIPI (c-stat=0.630) and PRIMA-PI (c-stat=0.590) models, Table 1. Importantly, FLIPI24 also improved prognostic ability for 10-year OS following IC (c-stat = 0.672) vs FLIPI (c-stat=0.629) or PRIMA-PI (c-stat=0.595). External validation was performed in a prospective cohort of 1438 newly diagnosed patients with FL (558 treated with IC) enrolled in the multicenter US LEO Cohort between 2015 and 2020. The FLIPI24 had superior c-stats compared to the FLIPI and PRIMA-PI for EFS24 (0.671 vs 0.600 vs 0.602) and OS (0.732 vs. 0.652 vs 0.635) in IC treated patients and for OS (0.785 vs 0.703 vs 0.662) when assessed in all patients. Conclusion: The FLIPI24 model provides an individual risk score at diagnosis for the likelihood of experiencing an event within 24 months from starting IC. The FLIPI24 was rigorously developed, tested, and externally validated using large harmonized and pooled international observational cohorts from the IC era. The FLIPI24 utilizes objective variables easily and reliably measured at diagnosis. A novel feature of the model is the significance of established blood-based measurements in the model, which appears to capture underlying tumor and host biology. Model performance for FLIPI24 was superior to standard clinical models for prediction of both EFS24 and OS in internal and external validation sets. Elevated FLIPI24 risk represents a patient population for further biologic studies and who should be considered for novel frontline therapies. Figure 1View largeDownload PPTFigure 1View largeDownload PPT Close modal

Abstract 11934: Natural Language Processing Models Can Be Trained to Accurately Recognize the Presence of Disease Within Clinical Notes

Introduction: Automated methods to “label” the presence of disease such as atrial fibrillation (AF) within electronic health record (EHR) data are important for secondary uses such as population health management. Labels generated from billing code patterns (e.g. “at least 2 relevant ICD codes used within 1 year”) can be accurate within one health system, but may fail to generalize across systems due to variations in coding practices. Hypothesis: We hypothesized that a natural language processing (NLP) model could be trained to detect the presence of atrial fibrillation using unstructured clinical notes by learning at scale from labels generated from a validated structured EHR and billing code definition. If successful, this could facilitate scaling disease label methods across large amounts of clinical data without suffering from inaccuracies due to variations in coding practices. Methods: We collected clinical notes from a regional health system into a training set of roughly 29 million code-labeled episodes and a separate hold-out set of roughly 1.8 million code-labeled episodes. We trained our NLP model on the training set and computed model performance compared to the code-based labels on the hold-out set. We also performed targeted blinded chart reviews. This work was conducted under IRB approval. Results: The model achieved an AUPRC of 0.91 with 87% recall and 89% precision and learned to distinguish challenging confounders despite not being explicitly trained for these tasks (Fig). Blinded review of select episodes showed that the NLP model was correct 90% of the time when it labeled the episode as positive for atrial fibrillation despite the code-based label (incorrectly) labeling it negative. Conclusions: NLP models can learn to automatically label the presence or absence of atrial fibrillation within clinical notes and may lead to greater accuracy and generalizability relative to code-based labeling methods.

Abstract 11978: Composite Deep Learning ECG Algorithm Trained to Identify Structural Heart Diseases Can Identify Clinically Ascertained Hypertrophic Cardiomyopathy

Introduction: A deep learning ECG algorithm, rECHOmmend, can accurately identify patients with any of seven structural heart diseases: five valvular diseases, low ejection fraction and interventricular septal (IVS) thickening. Components of the rECHOmmend composite label (IVS>15mm, mitral regurgitation) are also associated with hypertrophic cardiomyopathy (HCM). We hypothesized that despite being trained without HCM-specific labels, rECHOmmend can reliably identify HCM patients and achieve comparable performance to an HCM-specific classifier. Methods: Algorithms were developed from 2,898,979 ECGs acquired from 661,366 patients between 1984-2021. rECHOmmend was trained on a composite label derived from echocardiography and electronic health record (EHR) data. This ensemble model consists of 7 disease specific models with an aggregate model to predict a composite structural heart disease endpoint with shared clinical actionability. Separately, an HCM-specific model was trained on a binary label derived from EHR. To enable comparison, both classifiers were tested on a shared ECG holdout set (ECG prevalence 1.24%, patient prevalence 0.52%). Results: Despite being trained without HCM specific labels, the rECHOmmend ensemble showed comparable performance to a HCM-specific classifier (C-statistic: 0.92 [0.90-0.93] vs 0.90 [0.89-0.91]). At an operating point optimized for the F1-score, the sensitivity to HCM was higher for rECHOmmend at 0.42 [0.33-0.50] compared to 0.18 [0.15-0.21] for the HCM-specific classifier. rECHOmmend sustained performance across a range of IVS thicknesses, suggesting it was not solely reliant on IVS thickening for HCM identification and other ensemble components contributed to performance. Conclusions: A composite deep learning algorithm trained to identify structural heart diseases can identify clinically ascertained HCM with good performance, despite being trained without HCM-specific labels.

Genetically-informed prediction of short-term Parkinson’s disease progression

Parkinson’s disease (PD) treatments modify disease symptoms but have not been shown to slow progression, characterized by gradual and varied motor and non-motor changes overtime. Variation in PD progression hampers clinical research, resulting in long and expensive clinical trials prone to failure. Development of models for short-term PD progression prediction could be useful for shortening the time required to detect disease-modifying drug effects in clinical studies. PD progressors were defined by an increase in MDS-UPDRS scores at 12-, 24-, and 36-months post-baseline. Using only baseline features, PD progression was separately predicted across all timepoints and MDS-UPDRS subparts in independent, optimized, XGBoost models. These predictions plus baseline features were combined into a meta-predictor for 12-month MDS UPDRS Total progression. Data from the Parkinson’s Progression Markers Initiative (PPMI) were used for training with independent testing on the Parkinson’s Disease Biomarkers Program (PDBP) cohort. 12-month PD total progression was predicted with an F-measure 0.77, ROC AUC of 0.77, and PR AUC of 0.76 when tested on a hold-out PPMI set. When tested on PDBP we achieve a F-measure 0.75, ROC AUC of 0.74, and PR AUC of 0.73. Exclusion of genetic predictors led to the greatest loss in predictive accuracy; ROC AUC of 0.66, PR AUC of 0.66–0.68 for both PPMI and PDBP testing. Short-term PD progression can be predicted with a combination of survey-based, neuroimaging, physician examination, and genetic predictors. Dissection of the interplay between genetic risk, motor symptoms, non-motor symptoms, and longer-term expected rates of progression enable generalizable predictions.

MACHINE LEARNING PREDICTION OF IN-HOSPITAL DISEASE PROGRESSION IN COMMUNITY-ACQUIRED PNEUMONIA: DERIVATION AND VALIDATION OF CLINICAL DETERIORATION IN PNEUMONIA (CDIP)

MACHINE LEARNING PREDICTION OF IN-HOSPITAL DISEASE PROGRESSION IN COMMUNITY-ACQUIRED PNEUMONIA: DERIVATION AND VALIDATION OF CLINICAL DETERIORATION IN PNEUMONIA (CDIP)

Using Machine Learning to Predict Likelihood and Cause of Readmission After Hospitalization for Chronic Obstructive Pulmonary Disease Exacerbation.

BackgroundChronic obstructive pulmonary disease (COPD) is a leading cause of hospital readmissions. Few existing tools use electronic health record (EHR) data to forecast patients’ readmission risk during index hospitalizations.ObjectiveWe used machine learning and in-hospital data to model 90-day risk for and cause of readmission among inpatients with acute exacerbations of COPD (AE-COPD).DesignRetrospective cohort study.ParticipantsAdult patients admitted for AE-COPD at the University of Chicago Medicine between November 7, 2008 and December 31, 2018 meeting International Classification of Diseases (ICD)-9 or −10 criteria consistent with AE-COPD were included.MethodsRandom forest models were fit to predict readmission risk and respiratory-related readmission cause. Predictor variables included demographics, comorbidities, and EHR data from patients’ index hospital stays. Models were derived on 70% of observations and validated on a 30% holdout set. Performance of the readmission risk model was compared to that of the HOSPITAL score.ResultsAmong 3238 patients admitted for AE-COPD, 1103 patients were readmitted within 90 days. Of the readmission causes, 61% (n = 672) were respiratory-related and COPD (n = 452) was the most common. Our readmission risk model had a significantly higher area under the receiver operating characteristic curve (AUROC) (0.69 [0.66, 0.73]) compared to the HOSPITAL score (0.63 [0.59, 0.67]; p = 0.002). The respiratory-related readmission cause model had an AUROC of 0.73 [0.68, 0.79].ConclusionOur models improve on current tools by predicting 90-day readmission risk and cause at the time of discharge from index admissions for AE-COPD. These models could be used to identify patients at higher risk of readmission and direct tailored post-discharge transition of care interventions that lower readmission risk.

Deterministic small-scale undulations of image-based risk predictions from the deep learning of lung tumors in motion.

Deep learning (DL) models that use medical images to predict clinical outcomes are poised for clinical translation. For tumors that reside in organs that move, however, the impact of motion (i.e., degenerated object appearance or blur) on DL model accuracy remains unclear. We examine the impact of tumor motion on an image-based DL framework that predicts local failure risk after lung stereotactic body radiotherapy (SBRT). We input pre-therapy free breathing (FB) computed tomography (CT) images from 849 patients treated with lung SBRT into a multitask deep neural network to generate an image fingerprint signature (or DL score) that predicts time-to-event local failure outcomes. The network includes a convolutional neural network encoder for extracting imaging features and building a task-specific fingerprint, a decoder for estimating handcrafted radiomic features, and a task-specific network for generating image signature for radiotherapy outcome prediction. The impact of tumor motion on the DL scores was then examined for a holdout set of 468 images from 39 patients comprising: (1) FB CT, (2) four-dimensional (4D) CT, and (3) maximum-intensity projection (MIP) images. Tumor motion was estimated using a 3D vector of the maximum distance traveled, and its association with DL score variance was assessed by linear regression. The variance and amplitude in 4D CT image-derived DL scores were associated with tumor motion (R2 =0.48 and 0.46, respectively). Specifically, DL score variance was deterministic and represented by sinusoidal undulations in phase with the respiratory cycle. DL scores, but not tumor volumes, peaked near end-exhalation. The mean of the scores derived from 4D CT images and the score obtained from FB CT images were highly associated (Pearson r=0.99). MIP-derived DL scores were significantly higher than 4D- or FB-derived risk scores (p<0.0001). An image-based DL risk score derived from a series of 4D CT images varies in a deterministic, sinusoidal trajectory in a phase with the respiratory cycle. These results indicate that DL models of tumors in motion can be robust to fluctuations in object appearance due to movement and can guide standardization processes in the clinical translation of DL models for patients with lung cancer.

External validation: a simulation study to compare cross-validation versus holdout or external testing to assess the performance of clinical prediction models using PET data from DLBCL patients

AimClinical prediction models need to be validated. In this study, we used simulation data to compare various internal and external validation approaches to validate models.MethodsData of 500 patients were simulated using distributions of metabolic tumor volume, standardized uptake value, the maximal distance between the largest lesion and another lesion, WHO performance status and age of 296 diffuse large B cell lymphoma patients. These data were used to predict progression after 2 years based on an existing logistic regression model. Using the simulated data, we applied cross-validation, bootstrapping and holdout (n = 100). We simulated new external datasets (n = 100, n = 200, n = 500) and simulated stage-specific external datasets (1), varied the cut-off for high-risk patients (2) and the false positive and false negative rates (3) and simulated a dataset with EARL2 characteristics (4). All internal and external simulations were repeated 100 times. Model performance was expressed as the cross-validated area under the curve (CV-AUC ± SD) and calibration slope.ResultsThe cross-validation (0.71 ± 0.06) and holdout (0.70 ± 0.07) resulted in comparable model performances, but the model had a higher uncertainty using a holdout set. Bootstrapping resulted in a CV-AUC of 0.67 ± 0.02. The calibration slope was comparable for these internal validation approaches. Increasing the size of the test set resulted in more precise CV-AUC estimates and smaller SD for the calibration slope. For test datasets with different stages, the CV-AUC increased as Ann Arbor stages increased. As expected, changing the cut-off for high risk and false positive- and negative rates influenced the model performance, which is clearly shown by the low calibration slope. The EARL2 dataset resulted in similar model performance and precision, but calibration slope indicated overfitting.ConclusionIn case of small datasets, it is not advisable to use a holdout or a very small external dataset with similar characteristics. A single small testing dataset suffers from a large uncertainty. Therefore, repeated CV using the full training dataset is preferred instead. Our simulations also demonstrated that it is important to consider the impact of differences in patient population between training and test data, which may ask for adjustment or stratification of relevant variables.

PL01.6.A Radiomics outperforms semantic features for prediction of response to stereotactic radiosurgery in brain metastases

Abstract Background Brain metastases show different patterns of contrast enhancement, potentially reflecting hypoxic and necrotic tumor regions with reduced radiosensitivity. An objective evaluation of these patterns might allow a prediction of response to radiotherapy. We therefore investigated the potential of MRI radiomics in comparison with the visual assessment of semantic features to predict early response to stereotactic radiosurgery in patients with brain metastases. Material and Methods In this retrospective study, 150 patients with 308 brain metastases from solid tumors (NSCLC in 53% of patients) treated by stereotactic radiosurgery (single dose of 17-20 Gy) were evaluated. The response of each metastasis (partial or complete remission vs. stabilization or progression) was assessed within 180 days after radiosurgery. Patterns of contrast enhancement in the pre-treatment T1-weighted MR images were either visually classified (homogenous, heterogeneous, necrotic ring-like) or subjected to a radiomics analysis. Random forest models were optimized by cross-validation and evaluated in a hold-out test data set (30% of metastases). Results In total, 221/308 metastases (72%) responded to radiosurgery. The optimal radiomics model comprised 10 features and outperformed the model solely based on semantic features in the test data set (AUC, 0.71 vs. 0.56; accuracy, 69% vs. 54%). The diagnostic performance could be further improved by combining semantic and radiomics features resulting in an AUC of 0.74 and an accuracy of 75% in the test data set. Conclusion The developed radiomics model allowed prediction of early response to radiosurgery in patients with brain metastases and outperformed the visual assessment of patterns of contrast enhancement.

Predicting remission after internet-delivered psychotherapy in patients with depression using machine learning and multi-modal data

This study applied supervised machine learning with multi-modal data to predict remission of major depressive disorder (MDD) after psychotherapy. Genotyped adult patients (n = 894, 65.5% women, age 18–75 years) diagnosed with mild-to-moderate MDD and treated with guided Internet-based Cognitive Behaviour Therapy (ICBT) at the Internet Psychiatry Clinic in Stockholm were included (2008–2016). Predictor types were demographic, clinical, process (e.g., time to complete online questionnaires), and genetic (polygenic risk scores). Outcome was remission status post ICBT (cut-off ≤10 on MADRS-S). Data were split into train (60%) and validation (40%) given ICBT start date. Predictor selection employed human expertise followed by recursive feature elimination. Model derivation was internally validated through cross-validation. The final random forest model was externally validated against a (i) null, (ii) logit, (iii) XGBoost, and (iv) blended meta-ensemble model on the hold-out validation set. Feature selection retained 45 predictors representing all four predictor types. With unseen validation data, the final random forest model proved reasonably accurate at classifying post ICBT remission (Accuracy 0.656 [0.604, 0.705], P vs null model = 0.004; AUC 0.687 [0.631, 0.743]), slightly better vs logit (bootstrap D = 1.730, P = 0.084) but not vs XGBoost (D = 0.463, P = 0.643). Transparency analysis showed model usage of all predictor types at both the group and individual patient level. A new, multi-modal classifier for predicting MDD remission status after ICBT treatment in routine psychiatric care was derived and empirically validated. The multi-modal approach to predicting remission may inform tailored treatment, and deserves further investigation to attain clinical usefulness.

Deep Learning to Predict Geographic Atrophy Area and Growth Rate from Multimodal Imaging

To develop deep learning models for annualized geographic atrophy (GA) growth rate prediction using fundus autofluorescence (FAF) images and spectral-domain OCT volumes from baseline visits, which can be used for prognostic covariate adjustment to increase power of clinical trials. This retrospective analysis estimated GA growth rate as the slope of a linear fit on all available measurements of lesion area over a 2-year period. Three multitask deep learning models-FAF-only, OCT-only, and multimodal (FAF and OCT)-were developed to predict concurrent GA area and annualized growth rate. Patients were from prospective and observational lampalizumab clinical trials. The 3 models were trained on the development data set, tested on the holdout set, and further evaluated on the independent test sets. Baseline FAF images and OCT volumes from study eyes of patients with bilateral GA (NCT02247479; NCT02247531; and NCT02479386) were split into development (1279 patients/eyes) and holdout (443 patients/eyes) sets. Baseline FAF images from study eyes of NCT01229215 (106 patients/eyes) and NCT02399072 (169 patients/eyes) were used as independent test sets. Model performance was evaluated using squared Pearson correlation coefficient (r2) between observed and predicted lesion areas/growth rates. Confidence intervals were calculated by bootstrap resampling (B= 10 000). On the holdout data set, r2 (95% confidence interval) of the FAF-only, OCT-only, and multimodal models for GA lesion area prediction was 0.96 (0.95-0.97), 0.91 (0.87-0.95), and 0.94 (0.92-0.96), respectively, and for GA growth rate prediction was 0.48 (0.41-0.55), 0.36 (0.29-0.43), and 0.47 (0.40-0.54), respectively. On the 2 independent test sets, r2 of the FAF-only model for GA lesion area was 0.98 (0.97-0.99) and 0.95 (0.93-0.96), and for GA growth rate was 0.65 (0.52-0.75) and 0.47 (0.34-0.60). We show the feasibility of using baseline FAF images and OCT volumes to predict individual GA area and growth rates using a multitask deep learning approach. The deep learning-based growth rate predictions could be used for covariate adjustment to increase power of clinical trials. Proprietary or commercial disclosure may be found after the references.

Predicting Spray Dried Dispersion Particle Size Via Machine Learning Regression Methods

Spray dried dispersion particle size is a critical quality attribute that impacts bioavailability and manufacturability of the spray drying process and final dosage form. Substantial experimentation has been required to relate formulation and process parameters to particle size with the results limited to a single active pharmaceutical ingredient (API). This is the first study that demonstrates prediction of particle size independent of API for a wide range of formulation and process parameters at pilot and commercial scale. Additionally we developed a strategy with formulation and target particle size as inputs to define a set of “first to try” process parameters. An ensemble machine learning model was created to predict dried particle size across pilot and production scale spray dryers, with prediction errors between −7.7% and 18.6% (25th/75th percentiles) for a hold-out evaluation set. Shapley additive explanations identified how changes in formulation and process parameters drove variations in model predictions of dried particle size and were found to be consistent with mechanistic understanding of the particle formation process. Additionally, an optimization strategy used the predictive model to determine initial estimates for process parameter values that best achieve a target particle size for a provided formulation. The optimization strategy was employed to estimate process parameters in the hold-out evaluation set and to illustrate selection of process parameters during scale-up. The results of this study illustrate how trained regression models can reduce the experimental effort required to create an in-silico design space for new molecules during early-stage process development and subsequent scale-up.

Identification of women with high grade histopathology results after conisation by artificial neural networks.

BackgroundThe aim of the study was to evaluate if artificial neural networks can predict high-grade histopathology results after conisation from risk factors and their combinations in patients undergoing conisation because of pathological changes on uterine cervix.Patients and methodsWe analysed 1475 patients who had conisation surgery at the University Clinic for Gynaecology and Obstetrics of University Clinical Centre Maribor from 1993–2005. The database in different datasets was arranged to deal with unbalance data and enhance classification performance. Weka open-source software was used for analysis with artificial neural networks. Last Papanicolaou smear (PAP) and risk factors for development of cervical dysplasia and carcinoma were used as input and high-grade dysplasia Yes/No as output result. 10-fold cross validation was used for defining training and holdout set for analysis.ResultsBas eline classification and multiple runs of artificial neural network on various risk factors settings were performed. We achieved 84.19% correct classifications, area under the curve 0.87, kappa 0.64, F-measure 0.884 and Matthews correlation coefficient (MCC) 0.640 in model, where baseline prediction was 69.79%.ConclusionsWith artificial neural networks we were able to identify more patients who developed high-grade squamous intraepithelial lesion on final histopathology result of conisation as with baseline prediction. But, characteristics of 1475 patients who had conisation in years 1993–2005 at the University Clinical Centre Maribor did not allow reliable prediction with artificial neural networks for every-day clinical practice.