High-dimensional Machine Learning Research Articles

Hydrogen atom scattering at the Al2O3(0001) surface: a combined experimental and theoretical study.

Investigating atom-surface interactions is the key to an in-depth understanding of chemical processes at interfaces, which are of central importance in many fields - from heterogeneous catalysis to corrosion. In this work, we present a joint experimental and theoretical effort to gain insights into the atomistic details of hydrogen atom scattering at the α-Al2O3(0001) surface. Surprisingly, this system has been hardly studied to date, although hydrogen atoms as well as α-Al2O3 are omnipresent in catalysis as reactive species and support oxide, respectively. We address this system by performing hydrogen atom beam scattering experiments and molecular dynamics (MD) simulations based on a high-dimensional machine learning potential trained to density functional theory data. Using this combination of methods we are able to probe the properties of the multidimensional potential energy surface governing the scattering process. Specifically, we compare the angular distribution and the kinetic energy loss of the scattered atoms obtained in experiment with a large number of MD trajectories, which, moreover, allow to identify the underlying impact sites at the surface.

Employee benefits and company performance: Evidence from a high-dimensional machine learning model

By incorporating novel social media data, we analyze in detail how US companies offer different employee benefits and how they are associated with several company performance measures. Benefits such as 401(k), employee discounts, parking, and vision/dental healthcare are the most commonly provided, while free food -related benefits and family-related benefits are the most scarcely offered. Furthermore, with the aid of efficient machine learning -based models and tools from explainable artificial intelligence, we discover that family-related benefits are often associated with the most satisfied employees and best-performing companies. Our findings indicate that high-growth companies tend to provide a broad array of benefits to their employees. In contrast, highly profitable companies often concentrate on delivering a more limited and specialized set of benefits. We argue that companies offer rare and highly sought benefits to keep and recruit high-performers.

Mesoscale material modeling with memoryless isotropic point particles

There has been a proliferation of particle systems developed to model complex systems. These are attractive because they are mesh-free, avoiding issues associated with solver remeshing and convergence. They have however fragmented into niches, using increasingly complex particles that introduce internal degrees of freedom and external solver coupling. We show that, contrary to prior assumptions in the literature, memoryless isotropic point particles can model material properties including anisotropy, hysteresis, and failure solely through the statistics of their distributions. The resulting models offer compact code that is straightforward to accelerate and port, can span between micro- and macro-structure, require few parameters to set up a simulation, and unlike high-dimensional machine learning models they use a low-dimensional representation that can be efficiently learned. Rather than deriving them as approximations to either molecular dynamics or partial differential equations we investigate how these models can be found directly, and illustrate this with both qualitative comparisons of phenomenology and quantitative comparisons of predictions.

Global Neural Network Potential with Explicit Many-Body Functions for Improved Descriptions of Complex Potential Energy Surface.

The high dimensional machine learning potential (MLP) that has developed rapidly in the past decade represents a giant step forward in large-scale atomic simulation for complex systems. The long-range interaction and the poor description of chemical reactions are typical problems of high dimensional MLP, which are mainly caused by the poor structure discrimination of the atom-centered ML model. Herein, we propose a low-cost neural-network-based MLP architecture for fitting global potential energy surface data, namely, G-MBNN, that can offer improved energy and force resolution on a complex potential energy surface. In G-MBNN, a set of many-body energy terms based on the local atomic environment are explicitly included in computing the total energy─the total energy of the system is written as the sum of atomic energy and many-body energy contributions. These extra many-body energy terms are computationally low-cost and, importantly, can provide easy access to delicate energy terms in complex systems such as very short repulsion, long-range attractions, and sensitive angular-dependent covalent interactions. We implement G-MBNN in the LASP code and demonstrate the improved accuracy of the new framework in representative systems, including ternary-element energy materials LiCoOx, TiO2 with defects, and a series of organic reactions.

A highly efficient algorithm for solving exclusive lasso problems

The exclusive lasso (also known as elitist lasso) regularizer has become popular recently due to its superior performance on intra-group feature selection. Its complex nature poses difficulties for the computation of high-dimensional machine learning models involving such a regularizer. In this paper, we propose a highly efficient dual Newton method based proximal point algorithm (PPDNA) for solving large-scale exclusive lasso models. As important ingredients, we systematically study the proximal mapping of the weighted exclusive lasso regularizer and the corresponding generalized Jacobian. These results also make popular first-order algorithms for solving exclusive lasso models more practical. Extensive numerical results are presented to demonstrate the superior performance of the PPDNA against other popular numerical algorithms for solving the exclusive lasso problems.

Developing high-dimensional machine learning models to improve generalization ability and overcome data insufficiency for mixed sugar fermentation simulation

Biorefinery can be promoted by building accurate machine learning models. This work proposed a strategy to enhance model’s generalization ability and overcome insufficient data conditions for mixed sugar fermentation simulation. Multiple inputs single output models, using initial glucose, initial xylose, and time together as inputs, have higher generalization ability than single input single output models with time as sole input in predicting glucose, xylose, ethanol, or biomass separately. Multiple inputs multiple outputs models, integrating outputs, enhanced model accuracy and resulted in an average R2 at 0.99. To overcome data insufficiency conditions, consensus yeast (CY) model, through consolidating data from 4 yeasts, obtained R2 at 0.90. By adjusting the pretrained CY model, the model can save more than 50% data and get R2 at 0.95 and 0.93 for yeast and bacterial fermentation simulation. The strategy can expand the application range and save costs of data curation for ANN models.

Hadean tectonics: Insights from machine learning

Abstract The tectonic affiliations and magma compositions that formed Earth’s earliest crusts remain hotly debated. Previous efforts toward this goal have relied heavily on determining the provenance of Hadean zircons using low-dimensional discriminant diagrams developed from Phanerozoic samples, which are inadequate for capturing systematic differences without considering secular changes in zircon composition. Here, we developed high-dimensional machine learning (ML) approaches using zircon chemistry data (spanning 19 elements over 4.0 b.y.) to characterize zircons that crystallized in some typical tectonic settings (e.g., arcs, plume-related hotspots, and rifts) and from either igneous (I-type) or sedimentary (S-type) magmas. The proposed ML method, from a nonuniformitarian perspective, identifies the tectonic settings and granitoid types of given zircons (from Archean to Phanerozoic) at a higher prediction accuracy of >89% compared to ∼66%–82% for traditional discriminant diagrams (e.g., U/Yb vs. Y and rare earth elements (REE) + Y vs. P). The ML-based discriminators depend on the systematic differences in zircon chemistry, notably, significant differences in U, Th, and heavy REE for tectonic settings, and P and Hf for I- and S-type magmas. Application of the trained ML models to Hadean zircons from Jack Hills, Australia, suggests that these zircons were mainly crystallized in continental arc–forming magmas (90%) with 45% belonging to S-type melts. This result provides clear evidence of sediment recycling associated with subduction activity in the Hadean.

Prediction of hotel booking cancellations: Integration of machine learning and probability model based on interpretable feature interaction

Reliable hotel cancellation prediction can help establish appropriate operational strategies for hotel management. In this sector, personal name records (PNR) data may be the most representative information source for prediction tasks. Despite the popularity of PNR, its inherent lack of availability has been commonly disregarded in the literature. Existing studies have directly input PNR into high-dimensional machine learning (ML) models to achieve cancellation predictions. Another type of model generates cancellation prediction based on the probability modeling of samples. In this study, we propose an interpretable feature interaction method to enrich the existing PNR information. Thereafter, we empirically assess the prediction performance of the two model classes. This study specifically determines whether or not the two methods can cross-fertilize each other to improve cancellation prediction. To do so, we propose a model integrating Bayesian networks (BNs) and Lasso regression for this prediction task. This study utilizes BNs for the probability model consistent with our correlated variables and dichotomous prediction setting. Moreover, we use a linear ML model (i.e., Lasso regression), given its advantages in reducing ineffective predictors and transparency for ranking feature importance. Empirical results show that the proposed integration model has better prediction performance, and the obtained BN estimators and interactive features are the most important predictors. This study contributes to the booking cancellation literature by proposing an interpretable feature interaction and a prediction method integrating two types of effective models. The obtained accurate and interpretable cancellation prediction further contributes to offering practical implications to hoteliers in managerial decision-making.

A Review of Population-Based Metaheuristics for Large-Scale Black-Box Global Optimization—Part I

Scalability of optimization algorithms is a major challenge in coping with the ever-growing size of optimization problems in a wide range of application areas from high-dimensional machine learning to complex large-scale engineering problems. The field of large-scale global optimization is concerned with improving the scalability of global optimization algorithms, particularly, population-based metaheuristics. Such metaheuristics have been successfully applied to continuous, discrete, or combinatorial problems ranging from several thousand dimensions to billions of decision variables. In this two-part survey, we review recent studies in the field of large-scale black-box global optimization to help researchers and practitioners gain a bird’s-eye view of the field, learn about its major trends, and the state-of-the-art algorithms. Part I of the series covers two major algorithmic approaches to large-scale global optimization: 1) problem decomposition and 2) memetic algorithms. Part II of the series covers a range of other algorithmic approaches to large-scale global optimization, describes a wide range of problem areas, and finally, touches upon the pitfalls and challenges of current research and identifies several potential areas for future research.

Elevated blood pressure is associated with advanced brain aging in mid-life: A 30-year follow-up of The CARDIA Study.

High blood pressure (BP) is a risk factor for late-life brain health; however, the association of elevated BP with brain health in mid-life is unclear. We identified 661 participants from the Coronary Artery Risk Development in Young Adults Study (age 18-30 at baseline) with 30 years of follow-up and brain magnetic resonance imaging at year 30. Cumulative exposure of BP was estimated by time-weighted averages (TWA). Ideal cardiovascular health was defined as systolic BP<120mm Hg, diastolic BP<80mm Hg. Brain age was calculated using previously validated high dimensional machine learning pattern analyses. Every 5mmHg increment in TWA systolic BP was associated with approximately 1-year greater brain age (95% confidence interval [CI]: 0.50-1.36) Participants with TWA systolic or diastolic BP over the recommended guidelines for ideal cardiovascular health, had on average 3-year greater brain age (95% CI: 1.00-4.67; 95% CI: 1.45-5.13, respectively). Elevated BP from early to mid adulthood, even below clinical cut-offs, is associated with advanced brain aging in mid-life.

Evaluating High-Dimensional Machine Learning Models to Predict Hospital Mortality Among Older Patients With Cancer.

Older hospitalized cancer patients face high risks of hospital mortality. Improved risk stratification could help identify high-risk patients who may benefit from future interventions, although we lack validated tools to predict in-hospital mortality for patients with cancer. We evaluated the ability of a high-dimensional machine learning prediction model to predict inpatient mortality and compared the performance of this model to existing prediction indices. We identified patients with cancer older than 75 years from the National Emergency Department Sample between 2016 and 2018. We constructed a high-dimensional predictive model called Cancer Frailty Assessment Tool (cFAST), which used an extreme gradient boosting algorithm to predict in-hospital mortality. cFAST model inputs included patient demographic, hospital variables, and diagnosis codes. Model performance was assessed with an area under the curve (AUC) from receiver operating characteristic curves, with an AUC of 1.0 indicating perfect prediction. We compared model performance to existing indices including the Modified 5-Item Frailty Index, Charlson comorbidity index, and Hospital Frailty Risk Score. We identified 2,723,330 weighted emergency department visits among older patients with cancer, of whom 144,653 (5.3%) died in the hospital. Our cFAST model included 240 features and demonstrated an AUC of 0.92. Comparator models including the Modified 5-Item Frailty Index, Charlson comorbidity index, and Hospital Frailty Risk Score achieved AUCs of 0.58, 0.62, and 0.71, respectively. Predictive features of the cFAST model included acute conditions (respiratory failure and shock), chronic conditions (lipidemia and hypertension), patient demographics (age and sex), and cancer and treatment characteristics (metastasis and palliative care). High-dimensional machine learning models enabled accurate prediction of in-hospital mortality among older patients with cancer, outperforming existing prediction indices. These models show promise in identifying patients at risk of severe adverse outcomes, although additional validation and research studying clinical implementation of these tools is needed.

Exploring the robust extrapolation of high-dimensional machine learning potentials

We show that, contrary to popular assumptions, predictions from machine learning potentials built upon high-dimensional atom-density representations almost exclusively occur in regions of the representation space which lie outside the convex hull defined by the training set points. We then propose a perspective to rationalize the domain of robust extrapolation and accurate prediction of atomistic machine learning potentials in terms of the probability density induced by training points in the representation space

Right hemispheric white matter hyperintensities improve the prediction of spatial neglect severity in acute stroke

White matter hyperintensities (WMH) are frequently observed in brain scans of elderly people. They are associated with an increased risk of stroke, cognitive decline, and dementia. However, it is unknown yet if measures of WMH provide information that improve the understanding of poststroke outcome compared to only state-of-the-art stereotaxic structural lesion data.We implemented high-dimensional machine learning models, based on support vector regression, to predict the severity of spatial neglect in 103 acute right hemispheric stroke patients.We found that (1) the additional information of right hemispheric or bilateral voxel-based topographic WMH extent indeed yielded a significant improvement in predicting acute neglect severity (compared to the voxel-based stroke lesion map alone). (2) Periventricular WMH appeared more relevant for prediction than deep subcortical WMH. (3) Among different measures of WMH, voxel-based maps as measures of topographic extent allowed more accurate predictions compared to the use of traditional ordinally assessed visual rating scales (Fazekas-scale, Cardiovascular Health Study-scale).In summary, topographic WMH appear to be a valuable clinical imaging biomarker for predicting the severity of cognitive deficits and bears great potential for rehabilitation guidance of acute stroke patients.

Blood pressure is associated with advanced brain aging patterns on MRI in midlife: The CARDIA Study

AbstractBackgroundHigh blood pressure has been associated with cognitive outcomes such as dementia. Most studies have focused on blood pressure in mid or late life, thus this association in early adulthood is undefined. We aimed to investigate the relationship between sub‐clinical blood pressure in early adulthood and brain aging in midlife.MethodFrom the prospective study of Coronary Artery Risk Development in Young Adults, 662 Black and White participants were identified with follow‐up of 30 years (age 18–30 years at baseline) and brain MRI at Year 30. Cumulative exposure of systolic and diastolic blood pressure (BP) from baseline through Year 30 were estimated by areas under the curve and analyzed as time‐weighted averages. We used a previously‐validated high dimensional machine learning pattern analysis method to derive brain aging indices that quantify individual differences in spatial patterns of age‐related atrophy: the SPARE‐BA index (Spatial Pattern of Atrophy for Recognition of Brain Aging). Cognitive function was assessed at year 30 with the Rey Auditory Verbal Learning Test (RAVLT), Digit Symbol Substitution Test (DSST), Stroop Test, and Verbal Fluency. We used linear regression models to estimate the associations between cumulative BP and predicted brain age and cognitive function at year 30.ResultCumulative systolic and diastolic BP were positively associated with higher predicted brain age (β: 0.10, 95% CI: 0.04‐0.17, β: 0.10, 95% CI: 0.02‐0.19, respectively), adjusted for chronological age at year 30, sex, education, and race. Participants with cumulative systolic or diastolic BP over the recommended guidelines (systolic BP&gt;120, diastolic BP&gt;80), had on average 3‐years older predicted brain age (95% CI: 1.00‐4.67; 95% CI: 1.45‐5.13, respectively), compared to those with normal levels. Moreover, higher predicted brain age was associated with poorer performance on the RAVLT (p= 0.009), DSST (p&lt;0.001), Stroop Test (p&lt;0.001) and Verbal Fluency (0.009).ConclusionElevated blood pressure from early to middle adulthood, especially above recommended guidelines, is associated with worse cognition in midlife and advanced brain aging. Elevated sub‐clinical blood pressure could be an important target for early intervention in order to prevent cognitive impairment.

Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates

Federated Learning (FL) is an emerging decentralized learning framework through which multiple clients can collaboratively train a learning model. However, a major obstacle that impedes the wide deployment of FL lies in massive communication traffic. To train high dimensional machine learning models (such as CNN models), heavy communication traffic can be incurred by exchanging model updates via the Internet between clients and the parameter server (PS), implying that the network resource can be easily exhausted. Compressing model updates is an effective way to reduce the traffic amount. However, a flexible unbiased compression algorithm applicable for both uplink and downlink compression in FL is still absent from existing works. In this work, we devise the Model Update Compression by Soft Clustering (MUCSC) algorithm to compress model updates transmitted between clients and the PS. In MUCSC, it is only necessary to transmit cluster centroids and the cluster ID of each model update. Moreover, we prove that: 1) The compressed model updates are unbiased estimation of their original values so that the convergence rate by transmitting compressed model updates is unchanged; 2) MUCSC can guarantee that the influence of the compression error on the model accuracy is minimized. Then, we further propose the boosted MUCSC (B-MUCSC) algorithm, a biased compression algorithm that can achieve an extremely high compression rate by grouping insignificant model updates into a super cluster. B-MUCSC is suitable for scenarios with very scarce network resource. Ultimately, we conduct extensive experiments with the CIFAR-10 and FEMNIST datasets to demonstrate that our algorithms can not only substantially reduce the volume of communication traffic in FL, but also improve the training efficiency in practical networks.

Predictors of inpatient admission for pediatric cancer patients visiting the emergency department.

e22019 Background: Pediatric cancer patients represent a vulnerable cohort at risk of adverse outcomes after presenting to the emergency department (ED). Given the severity of cancer-related complications and uniqueness of this population, approaches to better risk stratify this cohort could potentially help define future interventions geared towards improving outcomes. We used a high-dimensional machine learning approach to help define the risk of hospitalization after an ED visit among pediatric patients with cancer. Methods: A cohort of cancer patients under 18 was identified from the Nationwide Emergency Department Sample (NEDS) between 2016-2018. We used a lasso regression model to predict inpatient admission after an ED visit. Model covariates included patient demographics, hospital characteristics, and International Classification of Diseases, version 10 (ICD-10) diagnosis codes. The data were split 75%/25% into training/testing data. The model was constructed with training data, and performance assessed on the test data using the area under the curve (AUC), with an AUC of 1.0 indicating perfect prediction. Results: We identified 129,631 pediatric cancer patients who visited the ED, of which 54.5% were subsequently admitted. The final predictive model included 150 variables, including 9 demographic, 6 hospital, and 135 unique ICD-10 codes. The model demonstrated excellent ability to predict hospitalization with an AUC of 0.96. The top 5 most important variables associated with inpatient admission were diagnoses of paralytic ileus/intestinal obstruction, neutropenia, sepsis, aplastic anemia/bone marrow failure, and bacterial infection. Conclusions: Pediatric cancer patients frequently present to the ED with complications of their cancer or their treatment, and over half of these patients are admitted. This study demonstrates the capacity of high-dimensional prediction models to help identify pediatric patients at risk of hospitalization. Additional research is needed to determine how to implement these predictive models in clinical practice.

Asynchronous Parallel, Sparse Approximated SVRG for High-Dimensional Machine Learning

With the increasing of the data size and the development of multi-core computers, asynchronous parallel stochastic optimization algorithms such as KroMagnon have gained significant attention. In this paper, we propose a new Sparse approximation and asynchronous parallel Stochastic Variance Reduced Gradient (SSVRG) method for sparse and high-dimensional machine learning problems. Unlike standard SVRG and its asynchronous parallel variant, KroMagnon, the snapshot point of SSVRG is set to the average of all the iterates in the previous epoch, which allows it to take much larger learning rates and also makes it more robust to the choice of learning rates. In particular, we use the sparse approximation of the popular SVRG estimator to perform completely sparse updates at all iterations. Therefore, SSVRG has a much lower per-iteration computational cost than its dense counterpart, SVRG++, and is very friendly to asynchronous parallel implementation. Moreover, we provide the convergence guarantees of SSVRG for both strongly convex and non-strongly convex problems, while existing asynchronous algorithms (e.g., KroMagnon and ASAGA) only have convergence guarantees for strongly convex problems. Finally, we extend SSVRG to non-smooth and asynchronous parallel settings. Numerical experimental results demonstrate that SSVRG converges significantly faster than the state-of-the-art asynchronous parallel methods, e.g., KroMagnon, and is usually more than three orders of magnitude faster than SVRG++.

A consensus-based global optimization method for high dimensional machine learning problems

We improve recently introduced consensus-based optimization method, proposed in [R. Pinnau, C. Totzeck, O. Tse, S. Martin, Math. Models Methods Appl. Sci. 27 (2017) 183–204], which is a gradient-free optimization method for general non-convex functions. We first replace the isotropic geometric Brownian motion by the component-wise one, thus removing the dimensionality dependence of the drift rate, making the method more competitive for high dimensional optimization problems. Secondly, we utilize the random mini-batch ideas to reduce the computational cost of calculating the weighted average which the individual particles tend to relax toward. For its mean-field limit – a nonlinear Fokker-Planck equation – we prove, in both time continuous and semi-discrete settings, that the convergence of the method, which is exponential in time, is guaranteed with parameter constraints independent of the dimensionality. We also conduct numerical tests to high dimensional problems to check the success rate of the method.

Determining the Effect of Hot Electron Dissipation on Molecular Scattering Experiments at Metal Surfaces.

Nonadiabatic effects that arise from the concerted motion of electrons and atoms at comparable energy and time scales are omnipresent in thermal and light-driven chemistry at metal surfaces. Excited (hot) electrons can measurably affect molecule–metal reactions by contributing to state-dependent reaction probabilities. Vibrational state-to-state scattering of NO on Au(111) has been one of the most studied examples in this regard, providing a testing ground for developing various nonadiabatic theories. This system is often cited as the prime example for the failure of electronic friction theory, a very efficient model accounting for dissipative forces on metal-adsorbed molecules due to the creation of hot electrons in the metal. However, the exact failings compared to experiment and their origin from theory are not established for any system because dynamic properties are affected by many compounding simulation errors of which the quality of nonadiabatic treatment is just one. We use a high-dimensional machine learning representation of electronic structure theory to minimize errors that arise from quantum chemistry. This allows us to perform a comprehensive quantitative analysis of the performance of nonadiabatic molecular dynamics in describing vibrational state-to-state scattering of NO on Au(111) and compare directly to adiabatic results. We find that electronic friction theory accurately predicts elastic and single-quantum energy loss but underestimates multiquantum energy loss and overestimates molecular trapping at high vibrational excitation. Our analysis reveals that multiquantum energy loss can potentially be remedied within friction theory whereas the overestimation of trapping constitutes a genuine breakdown of electronic friction theory. Addressing this overestimation for dynamic processes in catalysis and surface chemistry will likely require more sophisticated theories

Convergence of a first-order consensus-based global optimization algorithm

Global optimization of a non-convex objective function often appears in large-scale machine learning and artificial intelligence applications. Recently, consensus-based optimization (CBO) methods have been introduced as one of the gradient-free optimization methods. In this paper, we provide a convergence analysis for the first-order CBO method in [J. A. Carrillo, S. Jin, L. Li and Y. Zhu, A consensus-based global optimization method for high dimensional machine learning problems, https://arxiv.org/abs/1909.09249v1 ]. Prior to this work, the convergence study was carried out for CBO methods on corresponding mean-field limit, a Fokker–Planck equation, which does not imply the convergence of the CBO method per se. Based on the consensus estimate directly on the first-order CBO model, we provide a convergence analysis of the first-order CBO method [J. A. Carrillo, S. Jin, L. Li and Y. Zhu, A consensus-based global optimization method for high dimensional machine learning problems, https://arxiv.org/abs/1909.09249v1 ] without resorting to the corresponding mean-field model. Our convergence analysis consists of two steps. In the first step, we show that the CBO model exhibits a global consensus time asymptotically for any initial data, and in the second step, we provide a sufficient condition on system parameters — which is dimension independent — and initial data which guarantee that the converged consensus state lies in a small neighborhood of the global minimum almost surely.