Random Splits Research Articles

A machine learning model to predict treatment initiation among new patients in a community oncology network.

e13539 Background: In The US Oncology Network (The Network), about one-third of new patients with a cancer diagnosis started intravenous (IV) treatment after their first visit. The rest of the patients either came in for a consult only or might have received other treatments such as radiation, surgery, or oral therapy. We developed a machine learning model to predict IV treatment initiation among new patients and discovered features associated with the patient’s decision. This model could suggest interventions to improve patient’s access to care. Methods: A retrospective cohort was formed by identifying new patients with cancer from 27 practices in The Network between July 1, 2021 and June 30, 2022. Structured data were extracted and processed from the electronic health records, claims, physician referrals, and the American Community Survey. Patient characteristics included demographics, clinical information, payor types, and socioeconomic status. The referral pattern and the geographic region of practices, and the provider workload were considered as well. Gradient-boosted decision trees, random forest, neural network, and logistic regression models were developed to predict the probability of starting IV treatment within 90 days of the first visit. Model performance was evaluated based on the area under the receiver operating characteristic (AUROC) curve using cross-valuation and 4:1 training/validation random split. Shapley Additive Explanations (SHAP) values were applied to the model to explain feature importance. Results: A total of 117,340 new patients with a cancer diagnosis were included in the study, of whom 35% initiated IV treatment within 90 days of the first visit. A gradient-boosted decision tree algorithm with control of the imbalanced label was chosen as the final model because of the performance and the ability to handle missing values. The model achieved an AUROC of 0.80 on the validation dataset with both cross-valuation and 4:1 training/validation random split. Based on the SHAP values (log odds), we found that clinical information including diagnosis and stage is the most important feature to predict the initiation of IV treatment (mean absolute SHAP = 0.31 and 1.03, respectively). Medicaid contributes least to treatment initiation among all insurance types (mean absolute SHAP = 0.01). In addition, younger age and male patients have a higher chance to start IV treatment (Pearson correlation = -0.41, p-value < 0.01 for age versus SHAP values; p-value < 0.01, two-sided T-test for SHAP values by gender). Conclusions: This study reports a machine learning model to predict IV treatment initiation among new patients with cancer. Clinical features impact the treatment decision more than others. This model could guide patient service and direct personalized care navigation. Further, the model sheds light on future interventions that could enhance patient access to treatment promptly.

A radiomic-based predictive model of lung adenocarcinoma brain metastases and molecular subtypes.

2014 Background: Non-small cell lung cancer (NSCLC) comprises the largest portion of brain metastases (BM) from solid cancer with 40% of patients developing BM during the course of their disease. There are currently no reliable prediction tools for identifying patients at risk for BM, especially in the early-stage setting where MRI screening is not performed. Furthermore, in the later stage setting, brain MRI are only performed annually. Therefore, there is a critical need to identify high-risk patients for BM that could benefit from MRI surveillance. Methods: We identified 162 lung adenocarcinoma (LUAD) patients with (N = 66) or without (N = 96) BM that had treatment-naïve CT scans with a segmentable lesion. The tumor, surrounding ground glass opacity and necrosis were segmented via 3D slicer to create a volume of interest for radiomic texture analysis and 400 features were extracted. The Least Absolute Shrinkage and Selection Operator (LASSO) logistic regression feature selection method was used to select the most relevant features and models were built using the machine learning method XGBoost classifier. Training and testing sets with random splitting was used for cross validation. We report the accuracy, sensitivity, specificity, and area under the curve (AUC) for each model. Results: Among the extracted features that LASSO deemed as most discriminative for development of BM, we identified the most relevant features using XGBoost that predicted BM with 79% accuracy, 83% sensitivity, 72% specificity, and 79% AUC ( p= 0.01) in the overall population. The addition of ground glass opacity and necrosis to the model did not significantly improve performance. Furthermore, the model distinguished those with metachronous vs synchronous BM with 84% accuracy, 83% sensitivity, 86% specificity, and 83% AUC ( p = 0.04). Our model held up across molecular subtypes ( EGFR and KRAS mutant). Importantly, the model was predictive in early-stage patients with 92% accuracy, 96% sensitivity, 83% specificity, and 95% AUC ( p= 0.0005). Moreover, our model predicted for high vs. low overall survival, and was BM-specific as it was not predictive of other sites of metastases. We further developed a model from the CT features that correctly classified KRAS mutant vs. KRAS wild type LUAD with 77% accuracy, 73% sensitivity, 80% specificity, and 80% AUC ( p= 0.002). Conclusions: Utilizing a radiomics approach, we were able to predict BM from primary lung CT features including in stage I and II disease, predict synchronous vs metachronous BM, and distinguish distinct molecular LUAD subtypes. We are currently validating our BM prediction model in a large independent cohort and developing models to classify targetable LUAD-BM molecular alterations utilizing brain MRI scans. These studies will identify patients that require MRI surveillance in the early-stage setting and more intensive surveillance in the late-stage setting for BM.

A Bayesian approach to predictive uncertainty in chemotherapy patients at risk of acute care utilization.

Machine learning (ML) predictions are becoming increasingly integrated into medical practice. One commonly used method, ℓ1-penalised logistic regression (LASSO), can estimate patient risk for disease outcomes but is limited by only providing point estimates. Instead, Bayesian logistic LASSO regression (BLLR) models provide distributions for risk predictions, giving clinicians a better understanding of predictive uncertainty, but they are not commonly implemented. This study evaluates the predictive performance of different BLLRs compared to standard logistic LASSO regression, using real-world, high-dimensional, structured electronic health record (EHR) data from cancer patients initiating chemotherapy at a comprehensive cancer centre. Multiple BLLR models were compared against a LASSO model using an 80-20 random split using 10-fold cross-validation to predict the risk of acute care utilization (ACU) after starting chemotherapy. This study included 8439 patients. The LASSO model predicted ACU with an area under the receiver operating characteristic curve (AUROC) of 0.806 (95% CI: 0.775-0.834). BLLR with a Horseshoe+prior and a posterior approximated by Metropolis-Hastings sampling showed similar performance: 0.807 (95% CI: 0.780-0.834) and offers the advantage of uncertainty estimation for each prediction. In addition, BLLR could identify predictions too uncertain to be automatically classified. BLLR uncertainties were stratified by different patient subgroups, demonstrating that predictive uncertainties significantly differ across race, cancer type, and stage. BLLRs are a promising yet underutilised tool that increases explainability by providing risk estimates while offering a similar level of performance to standard LASSO-based models. Additionally, these models can identify patient subgroups with higher uncertainty, which can augment clinical decision-making. This work was supported in part by the National Library Of Medicine of the National Institutes of Health under Award Number R01LM013362. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Probabilistic Forecasting of Ground Magnetic Perturbation Spikes at Mid‐Latitude Stations

AbstractThe prediction of large fluctuations in the ground magnetic field (dB/dt) is essential for preventing damage from Geomagnetically Induced Currents. Directly forecasting these fluctuations has proven difficult, but accurately determining the risk of extreme events can allow for the worst of the damage to be prevented. Here we trained Convolutional Neural Network models for eight mid‐latitude magnetometers to predict the probability that dB/dt will exceed the 99th percentile threshold 30–60 min in the future. Two model frameworks were compared, a model trained using solar wind data from the Advanced Composition Explorer (ACE) satellite, and another model trained on both ACE and SuperMAG ground magnetometer data. The models were compared to examine if the addition of current ground magnetometer data significantly improved the forecasts of dB/dt in the future prediction window. A bootstrapping method was employed using a random split of the training and validation data to provide a measure of uncertainty in model predictions. The models were evaluated on the ground truth data during eight geomagnetic storms and a suite of evaluation metrics are presented. The models were also compared to a persistence model to ensure that the model using both datasets did not over‐rely on dB/dt values in making its predictions. Overall, we find that the models using both the solar wind and ground magnetometer data had better metric scores than the solar wind only and persistence models, and was able to capture more spatially localized variations in the dB/dt threshold crossings.

Abstract 1030: Development of a genome-wide methylome enrichment platform for multi-cancer early detection (MCED)

Abstract Background: Plasma cell-free DNA (cfDNA) tests represent a promising approach for cancer screening. cfDNA methylome approaches are well-suited for MCED; however, different methodologies vary in performance and many tests show decreased performance for early-stage or low-shedding tumors. Here we present a retrospective case-control study evaluating the performance of a novel genome-wide methylome enrichment platform for MCED. Methods: The full cohort (N=4,322) includes cancer cases (individuals with newly diagnosed treatment-naïve cancer) and age- and gender-matched non-cancer controls. Samples were analyzed with a bisulfite-free, non-degradative genome-wide DNA methylation enrichment platform using cfDNA isolated from plasma. Samples were split into distinct sets to train and test a machine learning classifier consisting of differentially methylated regions to distinguish cases from controls. Initial training in 1,536 samples across 8 cancer types is reported here (100 iterations of random splits (80:20) for cross-validation). The area under the receiver operating characteristic curves (AUC) and 95% confidence intervals for the median probabilities are reported in the cross-validation. Results: Cancer cases were distinguished from controls with an AUC of 0.94 (0.93, 0.96), with AUCs for individual cancer types ranging from 0.91 to 0.97. The AUC was 0.94 (0.92, 0.95) for stage I/II cancers and 0.95 (0.94, 0.96) for stage III/IV cancers. The AUC was 0.92 (0.91, 0.94) in the subset of cancers typically considered low shedding, with similar performance for stage I/II (0.91; 0.89, 0.93) and stage III/IV (0.93; 0.91, 0.95) in that subset. Conclusions: Initial analysis of case-control data demonstrates feasibility of a genome-wide methylome enrichment platform for MCED. The high detection of low-shedding and early-stage cancers is promising for MCED applications, as this will be critical for screening to identify cancers for which treatment may be more effective. Overall AUC (95% CI) for the full cohort and by cancer type. Cancer Type All Stages* Stage I/II Stage III/IV All Cancers N 931 461 437 AUC (95% CI) 0.94 (0.93, 0.96) 0.94 (0.92, 0.95) 0.95 (0.94, 0.96) Bladder Cancer** N 75 52 16 AUC (95% CI) 0.93 (0.91, 0.96) 0.93 (0.90, 0.96) 0.97 (0.95, 0.99) Breast Cancer** N 131 76 37 AUC (95% CI) 0.94 (0.91, 0.96) 0.92 (0.88, 0.95) 0.96 (0.94, 0.98) Colorectal Cancer N 182 94 88 AUC (95% CI) 0.97 (0.96, 0.98) 0.97 (0.96, 0.98) 0.97 (0.96, 0.99) Head & Neck Cancer N 75 20 55 AUC (95% CI) 0.96 (0.94, 0.98) 0.93 (0.86, 1) 0.97 (0.96, 0.99) Lung Cancer N 147 72 75 AUC (95% CI) 0.96 (0.95, 0.98) 0.96 (0.95, 0.98) 0.96 (0.95, 0.98) Ovarian Cancer N 46 12 34 AUC (95% CI) 0.96 (0.93, 0.98) 0.98 (0.96, 0.99) 0.95 (0.92, 0.98) Prostate Cancer** N 145 84 59 AUC (95% CI) 0.91 (0.88, 0.93) 0.92 (0.89, 0.95) 0.90 (0.87, 0.93) Renal Cancer** N 130 51 73 AUC (95% CI) 0.91 (0.89, 0.94) 0.89 (0.84, 0.94) 0.93 (0.89, 0.96) *Samples without stage information are included in this category **Typically considered low-shedding tumor Citation Format: Ben H. Park, Shu Yi Shen, Jun Min, Neil Fleshner, Jennifer Knox, Taymaa May, Laurie Ailles, Yulia Newton, Junjun Zhang, Rajat Singhania, Morgan Weichert, Justin Burgener, Iulia Cirlan, Jing Zhang, Yarong Wang, Eduardo Sosa, Angelica Polio, Owen Hall, Sarah Goldberg, Peter Mazzone, Brian Rini, Scott Bratman, Brian Allen, Krystal Brown, Abel Licon, Anne-Renee Hartman, Daniel D. De Carvalho, Geoffrey Liu. Development of a genome-wide methylome enrichment platform for multi-cancer early detection (MCED) [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 1030.

Data Modeling Using Vital Sign Dynamics for In-hospital Mortality Classification in Patients with Acute Coronary Syndrome.

This study compared feature selection by machine learning or expert recommendation in the performance of classification models for in-hospital mortality among patients with acute coronary syndrome (ACS) who underwent percutaneous coronary intervention (PCI). A dataset of 1,123 patients with ACS who underwent PCI was analyzed. After assigning 80% of instances to the training set through random splitting, we performed feature scaling and resampling with the synthetic minority over-sampling technique and Tomek link method. We compared two feature selection. recursive feature elimination with cross-validation (RFECV) and selection by interventional cardiologists. We used five simple models: support vector machine (SVM), random forest, decision tree, logistic regression, and artificial neural network. The performance metrics were accuracy, recall, and the false-negative rate, measured with 10-fold cross-validation in the training set and validated in the test set. Patients' mean age was 66.22 ± 12.88 years, and 33.63% had ST-elevation ACS. Fifteen of 34 features were selected as important with the RFECV method, while the experts chose 11 features. All models with feature selection by RFECV had higher accuracy than the models with expert-chosen features. In the training set, the random forest model had the highest accuracy (0.96 ± 0.01) and recall (0.97 ± 0.02). After validation in the test set, the SVM model displayed the highest accuracy (0.81) and a recall of 0.61. Models with feature selection by RFECV had higher accuracy than those with feature selection by experts in identifying patients with ACS at high risk for in-hospital mortality.

Machine Learning Identifies New Predictors on Restenosis Risk after Coronary Artery Stenting in 10,004 Patients with Surveillance Angiography.

Machine learning (ML) approaches have the potential to uncover regular patterns in multi-layered data. Here we applied self-organizing maps (SOMs) to detect such patterns with the aim to better predict in-stent restenosis (ISR) at surveillance angiography 6 to 8 months after percutaneous coronary intervention with stenting. In prospectively collected data from 10,004 patients receiving percutaneous coronary intervention (PCI) for 15,004 lesions, we applied SOMs to predict ISR angiographically 6-8 months after index procedure. SOM findings were compared with results of conventional uni- and multivariate analyses. The predictive value of both approaches was assessed after random splitting of patients into training and test sets (50:50). Conventional multivariate analyses revealed 10, mostly known, predictors for restenosis after coronary stenting: balloon-to-vessel ratio, complex lesion morphology, diabetes mellitus, left main stenting, stent type (bare metal vs. first vs. second generation drug eluting stent), stent length, stenosis severity, vessel size reduction, and prior bypass surgery. The SOM approach identified all these and nine further predictors, including chronic vessel occlusion, lesion length, and prior PCI. Moreover, the SOM-based model performed well in predicting ISR (AUC under ROC: 0.728); however, there was no meaningful advantage in predicting ISR at surveillance angiography in comparison with the conventional multivariable model (0.726, p = 0.3). The agnostic SOM-based approach identified-without clinical knowledge-even more contributors to restenosis risk. In fact, SOMs applied to a large prospectively sampled cohort identified several novel predictors of restenosis after PCI. However, as compared with established covariates, ML technologies did not improve identification of patients at high risk for restenosis after PCI in a clinically relevant fashion.

Quasi-SMILES as a basis for the development of QSPR models to predict the CO2 capture capacity of deep eutectic solvents using correlation intensity index and consensus modelling

The interpretation of the CO2 capture capacity of the deep eutectic solvents (DESs) as a mathematical function of (i) hydrogen bond acceptor(HBA); (ii) hydrogen bond donor (HBD); (iii) Molar ratio (HBA: HBD) (iv) temperature in kelvin and (v) p (MPa) gives quantitative models with good statistical quality. The quasi-SMILES are employed to represent the DESs (deep eutectic solvents under different conditions) for building up QSPR models. Quasi-SMILES differ from regular SMILES (simplified molecular input-line entry system) by including extra characters that indicate experimental parameters. SMILES descriptors may be used to construct quantitative structure–property/activity relationships (QSPRs/QSARs), whilst quasi-SMILES descriptors can be employed to build quantitative models of laboratory findings under diverse situations. Four random splits are prepared from the dataset of 72 DESs and each split is further divided into four sets namely active training, passive training, calibration and validation set. The percentage of the common quasi-SMILES in a set of two splits is zero. Three target functions with and without the index of ideality of correlation (IIC) and correlation intensity index (CII) are applied to build 12 QSPR models. The statistical quality of present QSPR models (mCO2) developed by the third target function is good to excellent, with determination coefficients ranging from 0.8303 to 0.9206 for the external validation set. In the last, consensus modelling using the intelligent consensus tool of DTC lab (https://dtclab.webs.com/software-tools) is also performed to predict the robustness of the developed models. The distribution of the dataset of all splits is applied to generate different consensus models and in all cases, the consensus model is found winner. The determination coefficient of all winner consensus models was in the range of 0.9751 to 0.9910 (CM0Split1R2=0.9909;CM0Split2R2=0.9883;CM2Split3R2=0.9910;andCM3Split4R2=0.9751). The statistical results of the determination coefficient and MAE (95%) of the second consensus model of split 3 (CM2Split3R2=0.9910;CM2Split3MAE95%=0.17776) was found best than the R2 and MAE (95%) of other consensus models of all splits. Therefore, it was considered as a winner consensus model from all consensus models.

PD30-03 IMPORTANCE OF COGNITIVE LOAD FOR AUTOMATED ERROR DETECTION IN ROBOT-ASSISTED SURGERY EDUCATION

You have accessJournal of UrologyCME1 Apr 2023PD30-03 IMPORTANCE OF COGNITIVE LOAD FOR AUTOMATED ERROR DETECTION IN ROBOT-ASSISTED SURGERY EDUCATION Rafal Kocielnik, Jasper A. Laca, Elyssa Y. Wong, Timothy N. Chu, Sidney I. Roberts, Sandra Marshall, Anima Anandkumar, and Andrew J. Hung Rafal KocielnikRafal Kocielnik More articles by this author , Jasper A. LacaJasper A. Laca More articles by this author , Elyssa Y. WongElyssa Y. Wong More articles by this author , Timothy N. ChuTimothy N. Chu More articles by this author , Sidney I. RobertsSidney I. Roberts More articles by this author , Sandra MarshallSandra Marshall More articles by this author , Anima AnandkumarAnima Anandkumar More articles by this author , and Andrew J. HungAndrew J. Hung More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000003316.03AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Cognitive Load (CL) can affect surgeon performance and the ability of training surgeons to receive feedback. We investigate the value of CL for detecting surgical errors in robot-assisted surgery (RAS) with a machine learning (ML)-based model. METHODS: We use 2 datasets (DS1, DS2) where 24 and 44 surgeons, respectively, performed a simulated dry-lab tissue dissection task on a live daVinci Xi surgical robot. The data contains video and per-second Index of Cognitive Activity (ICA), derived from pupillary change tracked via Tobii Eyetrackers. DS1 includes surgeons with low, medium, and high experience (by hours of RAS) as well as annotated tissue Tear (n=70) and Puncture (n=62) errors. DS2 contains data from low experience surgeons only and annotations for Tears (n=445) and Punctures (n=527). We extract 5 sec of video and varying history of ICA (from 5 to 60 sec) before and around human-labeled errors (positive label). We randomly sample the remaining data (negative label). We train variations of ML architecture consisting of video processing (pretrained Resnet18) and/or ICA processing (LSTM). We train the model on 3 different 80/20 train-test between-participant random splits. RESULTS: When including surgeons of mixed experience levels (DS1), ICA alone (AUC=64.28) is competitive with video alone (AUC=62.59) in detecting surgical errors (Figure 1A). However, the use of ICA in combination with video improves AUC in both datasets with gains of 5.55 (DS1) and 3.24 (DS2). History of ICA up to 20-30 seconds seems beneficial for error detection, after which point there is no further gain (Figure 1B). This is likely attributed to gradual build-up in CL leading to an error, and low-frequency of ICA measurements (1/sec), which may hinder extracting useful information from short time windows. The challenge of extracting meaningful information seems supported by Figure 1C, where providing processed ICA is beneficial (AUC=62.95) on top of raw ICA measurements (AUC=58.88). CONCLUSIONS: A surgeon’s CL in the context of surgical errors in RAS education is important. We show how CL is particularly valuable when a dataset includes mixed-experience surgeons. We also show the importance of ICA history length and pre-processing. Further work will explore better methods of extracting valuable features from CL data and its importance in surgery. Source of Funding: This material is based upon work supported by the National Science Foundation under Grant # 2030859 to the Computing Research Association for the CIFellows Project. Additionally, research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA273031. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health © 2023 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 209Issue Supplement 4April 2023Page: e831 Advertisement Copyright & Permissions© 2023 by American Urological Association Education and Research, Inc.MetricsAuthor Information Rafal Kocielnik More articles by this author Jasper A. Laca More articles by this author Elyssa Y. Wong More articles by this author Timothy N. Chu More articles by this author Sidney I. Roberts More articles by this author Sandra Marshall More articles by this author Anima Anandkumar More articles by this author Andrew J. Hung More articles by this author Expand All Advertisement PDF downloadLoading ...

Automated major depressive disorder diagnosis using a dual-input deep learning model and image generation from EEG signals

Major depressive disorder (MDD) is conventionally diagnosed through a questionnaire. Since approaches to diagnose MDD may lead to inaccurate diagnoses, many studies have presented electroencephalogram (EEG)-based machine learning techniques. The present paper introduces a deep learning approach based on image construction from EEGs. Two images are constructed from EEGs based on spectral and functional connectivity features. Afterward, the constructed images are applied to a two-stream convolutional neural network, and the outputs are concatenated. Finally, the concatenating result is applied to a sequential model of long–short-term memory, fully connected, and softmax layers to classify each sample into the MDD and healthy control (HC) classes. To validate the proposed approach, a public EEG dataset was used consisting of EEG data recorded from 34 MDD patients and 30 HC-matched participants. This framework obtained an AC of 98.03%, SE of 98.85%, SP of 97.19%, F1 of 98.07%, and FDR of 2.69% for the random splitting assessment method and achieved an average AC of 99.11%, SE of 98.97%, SP of 99.25%, F1 of 99.13%, and FDR of 0.71% using a 10-fold cross-validation process. Considering the accurate performance of the proposed method, it can be developed as a computer-aided diagnosis tool to diagnose MDD automatically.

Intraoperative Hypotension Is Associated with Postoperative Nausea and Vomiting in the PACU: A Retrospective Database Analysis

Multiple risk factors for postoperative nausea and vomiting (PONV)-a very distressing and outcome-related complication-have been identified, including female sex, absence of a history of smoking, history of PONV, and postoperative opioid use. Evidence of association of intraoperative hypotension with PONV is contradictory. A retrospective analysis of the perioperative documentation of 38,577 surgeries was conducted. The associations between different characterizations of intraoperative hypotension and PONV in the postoperative care unit (PACU) were investigated. First, the relationship between different characterizations of intraoperative hypotension with regard to PONV in the PACU was investigated. Secondly, the performance of the optimal characterization was assessed in an independent dataset derived via random split. The vast majority of characterizations showed an association of hypotension with the incidence of PONV in the PACU. In a multivariable regression, time with a MAP under 50 mmHg showed the strongest association with PONV in terms of the cross-validated Brier score. The adjusted odds for PONV in the PACU were estimated to be 1.34 times higher (95% CI: 1.33-1.35) when a MAP was under 50 mmHg for at least 1.8 min than when a MAP remained above 50 mmHg. The finding indicates that intraoperative hypotension may yet be another risk factor for PONV and therefore emphasizes the importance of intraoperative blood pressure management not only in patients at risk for cardiovascular complications but also in young and healthy patients at risk of PONV.

Smaller is better? Unduly nice accuracy assessments in roof detection using remote sensing data with machine learning and k-fold cross-validation

Deriving the thematic accuracy of models is a fundamental part of image classification analyses. K-fold cross-validation (KCV), as an accuracy assessment technique, can be biased because existing built-in algorithms of software solutions do not handle the high autocorrelation of remotely sensed images, leading to overestimation of accuracies. We aimed to quantify the magnitude of the overestimation of KCV-based accuracies and propose a method to overcome this problem with the example of rooftops using a WorldView-2 (WV2) satellite image, and two orthophotos. Random split to training/testing subsets, independent testing and different types of repeated KCV sampling strategies were used to generate input datasets for classification. Results revealed that applying the random splitting of reference data to training/testing subsets and KCV methods had significantly biased the accuracies by up to 17%; overall accuracies (OAs) can incorrectly reach >99%. We found that repeated KCV can provide similar results to independent testing when spatial sampling is applied with a sufficiently large distance threshold (in our case 10 m). Coarser resolution of WV2 ensured more reliable results (up to a 5–9% increase in OA) than orthophotos. Object-based pixel purity of buildings showed that when using a majority filter for at least of 50% of objects the final accuracy approached 100% with each sampling method. The final conclusion is that KCV-based modelling ensures better accuracy than single models (with better pixel purity on the object level), but the accuracy metrics without spatially filtered sampling are not reliable.

Toward improved prediction of recycled brick aggregate concrete compressive strength by designing ensemble machine learning models

The utilization of recycled brick aggregate concrete (RBC) is an area of active research, in which further investigation is needed to develop accurate models for predicting the behavior of RBC to ensure its safe and sustainable use in construction. This study presents an effective way to determine RBC's compressive strength (CS) based on an appropriate design of ensemble machine learning (ML) models, namely Gradient Boosting, Light Gradient boosting, AdaBoost, Extreme Gradient Boosting, Stacking, and Voting. A database covering 393 test results is compiled from the relevant literature for training and testing the models. In addition, 10-fold cross-validation and random splitting of data are used to ensure the reliability of predictions, as well as prevent overfitting. The findings reveal that the Stacking model has the greatest predictive ability, with a coefficient of determination of 0.95, root mean square error of 2.74 MPa, mean absolute error of 2.09 MPa, mean absolute percentage error of 0.10 on the testing dataset. In addition, Feature importance and Partial dependence plots analysis are utilized to investigate the impact of each RBC component on its CS. In RBC’s mixed-component design, the outcomes of this research, along with a developed Graphical User Interface (GUI) for the CS, might be of great use to material engineers.

How to approach machine learning-based prediction of drug/compound–target interactions

The identification of drug/compound–target interactions (DTIs) constitutes the basis of drug discovery, for which computational predictive approaches have been developed. As a relatively new data-driven paradigm, proteochemometric (PCM) modeling utilizes both protein and compound properties as a pair at the input level and processes them via statistical/machine learning. The representation of input samples (i.e., proteins and their ligands) in the form of quantitative feature vectors is crucial for the extraction of interaction-related properties during the artificial learning and subsequent prediction of DTIs. Lately, the representation learning approach, in which input samples are automatically featurized via training and applying a machine/deep learning model, has been utilized in biomedical sciences. In this study, we performed a comprehensive investigation of different computational approaches/techniques for protein featurization (including both conventional approaches and the novel learned embeddings), data preparation and exploration, machine learning-based modeling, and performance evaluation with the aim of achieving better data representations and more successful learning in DTI prediction. For this, we first constructed realistic and challenging benchmark datasets on small, medium, and large scales to be used as reliable gold standards for specific DTI modeling tasks. We developed and applied a network analysis-based splitting strategy to divide datasets into structurally different training and test folds. Using these datasets together with various featurization methods, we trained and tested DTI prediction models and evaluated their performance from different angles. Our main findings can be summarized under 3 items: (i) random splitting of datasets into train and test folds leads to near-complete data memorization and produce highly over-optimistic results, as a result, should be avoided, (ii) learned protein sequence embeddings work well in DTI prediction and offer high potential, despite interaction-related properties (e.g., structures) of proteins are unused during their self-supervised model training, and (iii) during the learning process, PCM models tend to rely heavily on compound features while partially ignoring protein features, primarily due to the inherent bias in DTI data, indicating the requirement for new and unbiased datasets. We hope this study will aid researchers in designing robust and high-performing data-driven DTI prediction systems that have real-world translational value in drug discovery.

OViTAD: Optimized Vision Transformer to Predict Various Stages of Alzheimer's Disease Using Resting-State fMRI and Structural MRI Data.

Advances in applied machine learning techniques for neuroimaging have encouraged scientists to implement models to diagnose brain disorders such as Alzheimer's disease at early stages. Predicting the exact stage of Alzheimer's disease is challenging; however, complex deep learning techniques can precisely manage this. While successful, these complex architectures are difficult to interrogate and computationally expensive. Therefore, using novel, simpler architectures with more efficient pattern extraction capabilities, such as transformers, is of interest to neuroscientists. This study introduced an optimized vision transformer architecture to predict the group membership by separating healthy adults, mild cognitive impairment, and Alzheimer's brains within the same age group (>75 years) using resting-state functional (rs-fMRI) and structural magnetic resonance imaging (sMRI) data aggressively preprocessed by our pipeline. Our optimized architecture, known as OViTAD is currently the sole vision transformer-based end-to-end pipeline and outperformed the existing transformer models and most state-of-the-art solutions. Our model achieved F1-scores of 97%±0.0 and 99.55%±0.39 from the testing sets for the rs-fMRI and sMRI modalities in the triple-class prediction experiments. Furthermore, our model reached these performances using 30% fewer parameters than a vanilla transformer. Furthermore, the model was robust and repeatable, producing similar estimates across three runs with random data splits (we reported the averaged evaluation metrics). Finally, to challenge the model, we observed how it handled increasing noise levels by inserting varying numbers of healthy brains into the two dementia groups. Our findings suggest that optimized vision transformers are a promising and exciting new approach for neuroimaging applications, especially for Alzheimer's disease prediction.

Implementing deep learning models for the classification of Echinococcus multilocularis infection in human liver tissue

BackgroundThe histological diagnosis of alveolar echinococcosis can be challenging. Decision support models based on deep learning (DL) are increasingly used to aid pathologists, but data on the histology of tissue-invasive parasitic infections are missing. The aim of this study was to implement DL methods to classify Echinococcus multilocularis liver lesions and normal liver tissue and assess which regions and structures play the most important role in classification decisions.MethodsWe extracted 15,756 echinococcus tiles from 28 patients using 59 whole slide images (WSI); 11,602 tiles of normal liver parenchyma from 18 patients using 33 WSI served as a control group. Different pretrained model architectures were used with a 60–20–20% random splitting. We visualized the predictions using probability-thresholded heat maps of WSI. The area-under-the-curve (AUC) value and other performance metrics were calculated. The GradCAM method was used to calculate and visualize important spatial features.ResultsThe models achieved a high validation and test set accuracy. The calculated AUC values were 1.0 in all models. Pericystic fibrosis and necrotic areas, as well as germinative and laminated layers of the metacestodes played an important role in decision tasks according to the superimposed GradCAM heatmaps.ConclusionDeep learning models achieved a high predictive performance in classifying E. multilocularis liver lesions. A possible next step could be to validate the model using other datasets and test it against other pathologic entities as well, such as, for example, Echinococcus granulosus infection.Graphical

In Silico Simulation of Impacts of Metal Nano-Oxides on Cell Viability in THP-1 Cells Based on the Correlation Weights of the Fragments of Molecular Structures and Codes of Experimental Conditions Represented by Means of Quasi-SMILES.

A simulation of the effect of metal nano-oxides at various concentrations (25, 50, 100, and 200 milligrams per millilitre) on cell viability in THP-1 cells (%) based on data on the molecular structure of the oxide and its concentration is proposed. We used a simplified molecular input-line entry system (SMILES) to represent the molecular structure. So-called quasi-SMILES extends usual SMILES with special codes for experimental conditions (concentration). The approach based on building up models using quasi-SMILES is self-consistent, i.e., the predictive potential of the model group obtained by random splits into training and validation sets is stable. The Monte Carlo method was used as a basis for building up the above groups of models. The CORAL software was applied to building the Monte Carlo calculations. The average determination coefficient for the five different validation sets was R2 = 0.806 ± 0.061.

Over-optimism in unsupervised microbiome analysis: Insights from network learning and clustering.

In recent years, unsupervised analysis of microbiome data, such as microbial network analysis and clustering, has increased in popularity. Many new statistical and computational methods have been proposed for these tasks. This multiplicity of analysis strategies poses a challenge for researchers, who are often unsure which method(s) to use and might be tempted to try different methods on their dataset to look for the "best" ones. However, if only the best results are selectively reported, this may cause over-optimism: the "best" method is overly fitted to the specific dataset, and the results might be non-replicable on validation data. Such effects will ultimately hinder research progress. Yet so far, these topics have been given little attention in the context of unsupervised microbiome analysis. In our illustrative study, we aim to quantify over-optimism effects in this context. We model the approach of a hypothetical microbiome researcher who undertakes four unsupervised research tasks: clustering of bacterial genera, hub detection in microbial networks, differential microbial network analysis, and clustering of samples. While these tasks are unsupervised, the researcher might still have certain expectations as to what constitutes interesting results. We translate these expectations into concrete evaluation criteria that the hypothetical researcher might want to optimize. We then randomly split an exemplary dataset from the American Gut Project into discovery and validation sets multiple times. For each research task, multiple method combinations (e.g., methods for data normalization, network generation, and/or clustering) are tried on the discovery data, and the combination that yields the best result according to the evaluation criterion is chosen. While the hypothetical researcher might only report this result, we also apply the "best" method combination to the validation dataset. The results are then compared between discovery and validation data. In all four research tasks, there are notable over-optimism effects; the results on the validation data set are worse compared to the discovery data, averaged over multiple random splits into discovery/validation data. Our study thus highlights the importance of validation and replication in microbiome analysis to obtain reliable results and demonstrates that the issue of over-optimism goes beyond the context of statistical testing and fishing for significance.

Evaluation of automatic discrimination between benign and malignant prostate tissue in the era of high precision digital pathology

BackgroundProstate cancer is a major health concern in aging men. Paralleling an aging society, prostate cancer prevalence increases emphasizing the need for efficient diagnostic algorithms.MethodsRetrospectively, 106 prostate tissue samples from 48 patients (mean age, 66pm 6.6 years) were included in the study. Patients suffered from prostate cancer (n = 38) or benign prostatic hyperplasia (n = 10) and were treated with radical prostatectomy or Holmium laser enucleation of the prostate, respectively. We constructed tissue microarrays (TMAs) comprising representative malignant (n = 38) and benign (n = 68) tissue cores. TMAs were processed to histological slides, stained, digitized and assessed for the applicability of machine learning strategies and open–source tools in diagnosis of prostate cancer. We applied the software QuPath to extract features for shape, stain intensity, and texture of TMA cores for three stainings, H&E, ERG, and PIN-4. Three machine learning algorithms, neural network (NN), support vector machines (SVM), and random forest (RF), were trained and cross-validated with 100 Monte Carlo random splits into 70% training set and 30% test set. We determined AUC values for single color channels, with and without optimization of hyperparameters by exhaustive grid search. We applied recursive feature elimination to feature sets of multiple color transforms.ResultsMean AUC was above 0.80. PIN-4 stainings yielded higher AUC than H&E and ERG. For PIN-4 with the color transform saturation, NN, RF, and SVM revealed AUC of 0.93pm 0.04, 0.91pm 0.06, and 0.92pm 0.05, respectively. Optimization of hyperparameters improved the AUC only slightly by 0.01. For H&E, feature selection resulted in no increase of AUC but to an increase of 0.02–0.06 for ERG and PIN-4.ConclusionsAutomated pipelines may be able to discriminate with high accuracy between malignant and benign tissue. We found PIN-4 staining best suited for classification. Further bioinformatic analysis of larger data sets would be crucial to evaluate the reliability of automated classification methods for clinical practice and to evaluate potential discrimination of aggressiveness of cancer to pave the way to automatic precision medicine.

Tests for high-dimensional single-index models

In this paper, we aim to test the overall significance of regression coefficients in high-dimensional single-index models. We first reformulate the hypothesis testing problem under elliptical distributions for predictors. Applying distribution-based transformation, we introduce a high-dimensional score-type test statistic. Notably, no moment condition for the error term is required. Our introduced procedures are thus robust with respect to outliers in response. Moreover our procedure is free of variance estimation of the error term. We establish the test statistic’s asymptotic normality under null hypothesis. Power analysis is also investigated. To further improve computational efficiency and enhance empirical powers, we also introduce a two-stage test procedure under ultrahigh-dimensional settings based on random data splitting. To eliminate the additional randomness induced by data splitting, we further develop a powerful ensemble algorithm based on multiple data splitting. We show that the ensemble algorithm can control the type I error rate at a given significance level. Extension to partial significance testing problem is also investigated. Lastly, numerical studies and real data analysis are conducted to compare with existing approaches and to illustrate the robustness and validity of our proposed test procedures.