High-dimensional Neural Network Research Articles

Nuclear Quantum Effects in Liquid Water Are Marginal for Its Average Structure but Significant for Dynamics.

Isotopic substitution, which can be realized in both experiment and computer simulations, is a direct approach to assess the role of nuclear quantum effects on the structure and dynamics of matter. However, the impact of nuclear quantum effects on the structure of liquid water as probed in experiment by comparing normal to heavy water has remained controversial. To settle this issue, we employ a highly accurate machine-learned high-dimensional neural network potential to perform converged coupled cluster-quality path integral simulations of liquid H2O versus D2O at ambient conditions. We find substantial H/D quantum effects on the rotational and translational dynamics of water, in close agreement with the experimental benchmarks. However, in stark contrast to the role for dynamics, H/D quantum effects turn out to be small, on the order of 1/1000 Å, on both average intramolecular and H-bonding structures of water. The most probable structure of water remains nearly unaffected by nuclear quantum effects, but effects on fluctuations away from average are appreciable, rendering H2O substantially more "liquid" than D2O.

Deep neural networks for endemic measles dynamics: Comparative analysis and integration with mechanistic models.

Measles is an important infectious disease system both for its burden on public health and as an opportunity for studying nonlinear spatio-temporal disease dynamics. Traditional mechanistic models often struggle to fully capture the complex nonlinear spatio-temporal dynamics inherent in measles outbreaks. In this paper, we first develop a high-dimensional feed-forward neural network model with spatial features (SFNN) to forecast endemic measles outbreaks and systematically compare its predictive power with that of a classical mechanistic model (TSIR). We illustrate the utility of our model using England and Wales measles data from 1944-1965. These data present multiple modeling challenges due to the interplay between metapopulations, seasonal trends, and nonlinear dynamics related to demographic changes. Our results show that while the TSIR model yields similarly performant short-term (1 to 2 biweeks ahead) forecasts for highly populous cities, our neural network model (SFNN) consistently achieves lower root mean squared error (RMSE) across other forecasting windows. Furthermore, we show that our spatial-feature neural network model, without imposing mechanistic assumptions a priori, can uncover gravity-model-like spatial hierarchy of measles spread in which major cities play an important role in driving regional outbreaks. We then turn our attention to integrative approaches that combine mechanistic and machine learning models. Specifically, we investigate how the TSIR can be utilized to improve a state-of-the-art approach known as Physics-Informed-Neural-Networks (PINN) which explicitly combines compartmental models and neural networks. Our results show that the TSIR can facilitate the reconstruction of latent susceptible dynamics, thereby enhancing both forecasts in terms of mean absolute error (MAE) and parameter inference of measles dynamics within the PINN. In summary, our results show that appropriately designed neural network-based models can outperform traditional mechanistic models for short to long-term forecasts, while simultaneously providing mechanistic interpretability. Our work also provides valuable insights into more effectively integrating machine learning models with mechanistic models to enhance public health responses to measles and similar infectious disease systems.

From electrons to phase diagrams with machine learning potentials using pyiron based automated workflows

We present a comprehensive and user-friendly framework built upon the pyiron integrated development environment (IDE), enabling researchers to perform the entire Machine Learning Potential (MLP) development cycle consisting of (i) creating systematic DFT databases, (ii) fitting the Density Functional Theory (DFT) data to empirical potentials or MLPs, and (iii) validating the potentials in a largely automatic approach. The power and performance of this framework are demonstrated for three conceptually very different classes of interatomic potentials: an empirical potential (embedded atom method - EAM), neural networks (high-dimensional neural network potentials - HDNNP) and expansions in basis sets (atomic cluster expansion - ACE). As an advanced example for validation and application, we show the computation of a binary composition-temperature phase diagram for Al-Li, a technologically important lightweight alloy system with applications in the aerospace industry.

Theoretical infrared spectroscopy of protonated methane isotopologues.

The vibrational spectroscopy of protonated methane and its mixed hydrogen/deuterium isotopologues remains a challenge to both experimental and computational spectroscopy due to the iconic floppiness of CH5+. Here, we compute the finite-temperature broadband infrared spectra of CH5+ and all its isotopologues, i.e. CHnD5-n+ up to CD5+, from path integral molecular dynamics in conjunction with interactions and dipoles computed consistently at CCSD(T) coupled cluster accuracy. The potential energy and dipole moment surfaces have been accurately represented in full dimensionality in terms of high-dimensional neural networks. The resulting computational efficiency allows us to establish CCSD(T) accuracy at the level of converged path integral simulations. For all six isotopologues, the computed broadband spectra compare very favorably to the available experimental broadband spectra obtained from laser induced reactions action vibrational spectroscopy. The current approach is found to consistently and significantly improve on previous calculations of these broadband vibrational spectra and defines the new cutting-edge for what has been dubbed the "enfant terrible" of molecular spectroscopy in view of its pronounced large-amplitude motion that involves all intramolecular degrees of freedom.

Hybrid Quantum Mechanical, Molecular Mechanical, and Machine Learning Potential for Computing Aqueous-Phase Adsorption Free Energies on Metal Surfaces.

Performing reliable computer simulations of elementary processes occurring at metal-water interfaces is pivotal for novel catalyst design in sustainable energy applications. Computational catalyst design hinges on the ability to reliably and efficiently compute the potential energy surface (PES) of the system. Due to the large system sizes needed for studying processes at liquid water-metal interfaces, these systems can currently not be described using density functional theory (DFT). In this work, we used a hybrid quantum mechanical, molecular mechanical, and machine learning potential for studying the adsorption behavior of phenol, atomic hydrogen, 2-butanol, and 2-butanone on the (0001) facet of Ru under reducing conditions when Ru is not oxidized. Specifically, we describe the adsorbate and the surrounding metal atoms at the DFT level of theory. Here, we also considered the electrostatic field effect of the water molecules on adsorbate-metal interactions. Next, for the water-water and water-adsorbate interactions, we used established classical force fields. Finally, for the water-Ru surface interaction, for which no reliable force fields have been published, we used Behler-Parrinello high-dimensional neural network potentials (HDNNPs). Employing this setup, we used our explicit solvation for metal surface (eSMS) approach to compute the aqueous-phase effect on the low-coverage adsorption of selected molecules and atoms on the (0001) facet of Ru. In agreement with previous experimental and computational studies of oxygenated molecules over transition metal facets, we found that liquid water destabilizes the tested adsorbates on Ru(0001). Interestingly, our findings indicate that adsorbates on Ru are less affected by the presence of an aqueous phase than on other transition metals (e.g., Pt), highlighting the necessity of experimental investigations of Ru-based catalytic systems in liquid water.

Learning automatic navigation control skills for miniature helical robots from human demonstrations

Magnetic micro-robotic technology holds immense potential for revolutionizing minimally invasive procedures, particularly in the realm of interventional medicine. The ability to effectively and precisely control micro-scale, magnetically actuated robots in real-world scenarios is important. Mastering this capability promises to elevate the precision and efficacy of medical interventions, thereby enhancing patient outcomes. Numerous methods were proposed in previous work and achieved significant progress. However, these efforts primarily focused on model-based strategies and control improvements, with little attention given to the manipulation of the surgeon. This paper presents an exploration of an imitation learning approach, leveraging extensive manual operation experiments, to replicate the task of robotic navigation in simulated vascular environments. The control strategies are directly acquired from experimental observations and encapsulated within a high-dimensional neural network, specifically a tailored variant of the Residual Network (ResNet). The robustness and effectiveness of our proposed methodology are validated through comprehensive experimentation. In automatic navigation trials, the average error spanned from 2.29 mm to 3.32 mm, leading to a mean trajectory deviation of approximately 2.92 mm. The average error rate is 31% lower than that observed in traditional model-based Proportional Integral Derivative (PID) controller (approximately 3.81 mm). In addition, the maximum error (4.87 mm) is 83% of that of the traditional method (5.85 mm). Our findings emphasize the viability and benefits of learning-based techniques in micro-robot control, paving the way for innovative control strategies in interventional surgeries applications.

First-Principles Nonadiabatic Dynamics of Molecules at Metal Surfaces withVibrationally Coupled Electron Transfer.

Accurate description of nonadiabatic dynamics of molecules at metal surfaces involving electron transfer has been a long-standing challenge for theory. Here, we tackle this problem by first constructing high-dimensional neural network diabatic potentials including state crossings determined by constrained density functional theory, then applying mixed quantum-classical surface hopping simulations to evolve coupled electron-nuclear motion. Our approach accurately describes the nonadiabatic effects in CO scattering from Au(111) without empirical parameters and yields results agreeing well with experiments under various conditions for this benchmark system. We find that both adiabatic and nonadiabatic energy loss channels have important contributions to the vibrational relaxation of highly vibrationally excited CO(v_{i}=17), whereas relaxation of low vibrationally excited states of CO(v_{i}=2) is weak and dominated by nonadiabatic energy loss. The presented approach paves the way for accurate first-principles simulations of electron transfer mediated nonadiabatic dynamics at metal surfaces.

Multi-origins of pathological theta oscillation from neuron to network inferred by a combined data and model study with cubature Kalman filter

The brain rhythm is strongly associated with the brain function. Alzheimer's disease (AD) is characterized reflected by the brain rhythm switching from the alpha band (9–12 Hz) to the theta band (4–8 Hz), accompanied with the loss of brain function. However, extracting the implicating intrinsic characteristic variations of the brain network by utilizing the Electroencephalogram (EEG) information is challenging. Kaman observer, serving as an effective Bayesian technique, can provide a visualization service for probing the intrinsic characteristics underlying the pathological theta oscillations. This work first establishes an excitation-inhibitory neural network model and explores the role of the fraction of the inhibitory neurons and inhibitory synapses in the pathological theta oscillation. The results indicate that the reduced inhibitory neuronal proportion and attenuated inhibitory synaptic weight are the main neural bases of the frequency reduction of neural oscillation. Then, we further explore the intrinsic spiking characteristic by considering spike frequency adaptation (SFA) to the inhibitory neurons. The results show that the SFA reduces the firing rate of neurons, which facilitates the theta rhythm. The enhancement of SFA current by increasing the time constant of its gating variable can further decrease the theta frequency from 7 Hz to 4 Hz. Furthermore, for this high-dimensional nonlinear excitation-inhibitory neural network model, cubature Kalman filter (CKF) is employed to estimate the above potential variations from the EEG data. The observation results show that the attenuated trends of the inhibitory neuronal proportion and the decreased inhibitory SFA current result in the descending brain rhythm. Finally, the theoretical simulation is deduced by utilizing the mean field theory for simplifying high-dimensional model to verify the simulation results. The theoretical variations of inhibitory parameters and adaptation gating variable are consistent with the simulation results. In summary, we investigate the multi-origin factors related to inhibitory neuronal intrinsic characteristics from forward model simulation and inverse EEG estimation process. And we further verify the simulation and data-driven results by theoretical derivation. This work enhances the understanding of the systematic function of inhibitory intrinsic characteristics on pathological theta oscillation and provides an effective method to decode the dynamics underlying neural activities.

Loss-Based Variational Bayes Prediction

We propose a new approach to Bayesian prediction that caters for models with a large number of parameters and is robust to model misspecification. Given a class of high-dimensional (but parametric) predictive models, this new approach constructs a posterior predictive using a variational approximation to a generalized posterior that is directly focused on predictive accuracy. The theoretical behavior of the new prediction approach is analyzed and a form of optimality demonstrated. Applications to both simulated and empirical data using high-dimensional Bayesian neural network and autoregressive mixture models demonstrate that the approach provides more accurate results than various alternatives, including misspecified likelihood-based predictions. Supplementary materials for this article are available online.

Feature selection for high-dimensional neural network potentials with the adaptive group lasso

Neural network potentials are a powerful tool for atomistic simulations, allowing to accurately reproduce ab initio potential energy surfaces with computational performance approaching classical force fields. A central component of such potentials is the transformation of atomic positions into a set of atomic features in a most efficient and informative way. In this work, a feature selection method is introduced for high dimensional neural network potentials, based on the adaptive group lasso (AGL) approach. It is shown that the use of an embedded method, taking into account the interplay between features and their action in the estimator, is necessary to optimize the number of features. The method’s efficiency is tested on three different monoatomic systems, including Lennard–Jones as a simple test case, Aluminium as a system characterized by predominantly radial interactions, and Boron as representative of a system with strongly directional components in the interactions. The AGL is compared with unsupervised filter methods and found to perform consistently better in reducing the number of features needed to reproduce the reference simulation data at a similar level of accuracy as the starting feature set. In particular, our results show the importance of taking into account model predictions in feature selection for interatomic potentials.

Active and Transfer Learning of High-Dimensional Neural Network Potentials for Transition Metals.

Classical molecular dynamics (MD) simulations represent a very popular and powerful tool for materials modeling and design. The predictive power of MD hinges on the ability of the interatomic potential to capture the underlying physics and chemistry. There have been decades of seminal work on developing interatomic potentials, albeit with a focus predominantly on capturing the properties of bulk materials. Such physics-based models, while extensively deployed for predicting the dynamics and properties of nanoscale systems over the past two decades, tend to perform poorly in predicting nanoscale potential energy surfaces (PESs) when compared to high-fidelity first-principles calculations. These limitations stem from the lack of flexibility in such models, which rely on a predefined functional form. Machine learning (ML) models and approaches have emerged as a viable alternative to capture the diverse size-dependent cluster geometries, nanoscale dynamics, and the complex nanoscale PESs, without sacrificing the bulk properties. Here, we introduce an ML workflow that combines transfer and active learning strategies to develop high-dimensional neural networks (NNs) for capturing the cluster and bulk properties for several different transition metals with applications in catalysis, microelectronics, and energy storage, to name a few. Our NN first learns the bulk PES from the high-quality physics-based models in literature and subsequently augments this learning via retraining with a higher-fidelity first-principles training data set to concurrently capture both the nanoscale and bulk PES. Our workflow departs from status-quo in its ability to learn from a sparsely sampled data set that nonetheless covers a diverse range of cluster configurations from near-equilibrium to highly nonequilibrium as well as learning strategies that iteratively improve the fingerprinting depending on model fidelity. All the developed models are rigorously tested against an extensive first-principles data set of energies and forces of cluster configurations as well as several properties of bulk configurations for 10 different transition metals. Our approach is material agnostic and provides a methodology to transfer and build upon the learnings from decades of seminal work in molecular simulations on to a new generation of ML-trained potentials to accelerate materials discovery and design.

Accelerated four-dimensional free-breathing whole-liver water-fat magnetic resonance imaging with deep dictionary learning and chemical shift modeling.

Multi-echo chemical-shift-encoded magnetic resonance imaging (MRI) has been widely used for fat quantification and fat suppression in clinical liver examinations. Clinical liver water-fat imaging typically requires breath-hold acquisitions, with the free-breathing acquisition method being more desirable for patient comfort. However, the acquisition for free-breathing imaging could take up to several minutes. The purpose of this study is to accelerate four-dimensional free-breathing whole-liver water-fat MRI by jointly using high-dimensional deep dictionary learning and model-guided (MG) reconstruction. A high-dimensional model-guided deep dictionary learning (HMDDL) algorithm is proposed for the acceleration. The HMDDL combines the powers of the high-dimensional dictionary learning neural network (hdDLNN) and the chemical shift model. The neural network utilizes the prior information of the dynamic multi-echo data in spatial respiratory motion, and echo dimensions to exploit the features of images. The chemical shift model is used to guide the reconstruction of field maps, maps, water images, and fat images. Data acquired from ten healthy subjects and ten subjects with clinically diagnosed nonalcoholic fatty liver disease (NAFLD) were selected for training. Data acquired from one healthy subject and two NAFLD subjects were selected for validation. Data acquired from five healthy subjects and five NAFLD subjects were selected for testing. A three-dimensional (3D) blipped golden-angle stack-of-stars multi-gradient-echo pulse sequence was designed to accelerate the data acquisition. The retrospectively undersampled data were used for training, and the prospectively undersampled data were used for testing. The performance of the HMDDL was evaluated in comparison with the compressed sensing-based water-fat separation (CS-WF) algorithm and a parallel non-Cartesian recurrent neural network (PNCRNN) algorithm. Four-dimensional water-fat images with ten motion states for whole-liver are demonstrated at several R values. In comparison with the CS-WF and PNCRNN, the HMDDL improved the mean peak signal-to-noise ratio (PSNR) of images by 9.93 and 2.20 dB, respectively, and improved the mean structure similarity (SSIM) of images by 0.058 and 0.009, respectively, at R=10. The paired t-test shows that there was no significant difference between HMDDL and ground truth for proton-density fat fraction (PDFF) and values at R up to 10. The proposed HMDDL enables features of water images and fat images from the highly undersampled multi-echo data along spatial, respiratory motion, and echo dimensions, to improve the performance of accelerated four-dimensional (4D) free-breathing water-fat imaging.

Review of High Dimensional Neural Network Potentials

Over the past few years, machine learning potentials (MLPs) have been fully developed and can now be applied to a variety of large-scale atomic simulations. MLPs encompass a number of different machine learning algorithms, of which neural network potentials are the most dominant and representative, and have achieved great success. This review focuses on the second generation of neural network potentials, high-dimensional neural network potentials (HDNNPs). While HDNNP has accomplished noteworthy results in the implementation of ion diffusion, nuclear magnetic resonance parameters, defect formation, and thermal conductivity, it still encounters hurdles in the compilation of training datasets, transferability, accuracy constraints, and the management of multiple chemical species. In spite of this, HDNNP remains a highly promising methodology. Looking forward, we anticipate the development of more specialized software for HDNNP to augment research efficiency and lower the barrier to usage.

Comparing machine learning potentials for water: Kernel-based regression and Behler-Parrinello neural networks.

In this paper, we investigate the performance of different machine learning potentials (MLPs) in predicting key thermodynamic properties of water using RPBE + D3. Specifically, we scrutinize kernel-based regression and high-dimensional neural networks trained on a highly accurate dataset consisting of about 1500 structures, as well as a smaller dataset, about half the size, obtained using only on-the-fly learning. This study reveals that despite minor differences between the MLPs, their agreement on observables such as the diffusion constant and pair-correlation functions is excellent, especially for the large training dataset. Variations in the predicted density isobars, albeit somewhat larger, are also acceptable, particularly given the errors inherent to approximate density functional theory. Overall, this study emphasizes the relevance of the database over the fitting method. Finally, this study underscores the limitations of root mean square errors and the need for comprehensive testing, advocating the use of multiple MLPs for enhanced certainty, particularly when simulating complex thermodynamic properties that may not be fully captured by simpler tests.

Hutchinson Trace Estimation for high-dimensional and high-order Physics-Informed Neural Networks

Physics-Informed Neural Networks (PINNs) have proven effective in solving partial differential equations (PDEs), especially when some data are available by seamlessly blending data and physics. However, extending PINNs to high-dimensional and even high-order PDEs encounters significant challenges due to the computational cost associated with automatic differentiation in the residual loss function calculation. Herein, we address the limitations of PINNs in handling high-dimensional and high-order PDEs by introducing the Hutchinson Trace Estimation (HTE) method. Starting with the second-order high-dimensional PDEs, which are ubiquitous in scientific computing, HTE is applied to transform the calculation of the entire Hessian matrix into a Hessian vector product (HVP). This approach not only alleviates the computational bottleneck via Taylor-mode automatic differentiation but also significantly reduces memory consumption from the Hessian matrix to an HVP’s scalar output. We further showcase HTE’s convergence to the original PINN loss and its unbiased behavior under specific conditions. Comparisons with the Stochastic Dimension Gradient Descent (SDGD) highlight the distinct advantages of HTE, particularly in scenarios with significant variability and variance among dimensions. We further extend the application of HTE to higher-order and higher-dimensional PDEs, specifically addressing the biharmonic equation. By employing tensor-vector products (TVP), HTE efficiently computes the colossal tensor associated with the fourth-order high-dimensional biharmonic equation, saving memory and enabling rapid computation. The effectiveness of HTE is illustrated through experimental setups, demonstrating comparable convergence rates with SDGD under memory and speed constraints. Additionally, HTE proves valuable in accelerating the Gradient-Enhanced PINN (gPINN) version as well as the Biharmonic equation. Overall, HTE opens up a new capability in scientific machine learning for tackling high-order and high-dimensional PDEs.

Solving Euler equations with gradient-weighted multi-input high-dimensional feature neural network

The study found that it is difficult to capture the solutions at the shock wave and discontinuity surfaces when solving Euler equations using physics informed neural network. Thus, this paper proposes an improved neural network based on adaptive weights for multi-input high-dimensional features to solve the Euler equations. First, adaptive weights regarding the velocity are added to the control equation of each residual to train the shock wave region better. Then, more residual training points are used in regions with initial time discontinuities to improve the training efficiency. The problem that the sigmoid activation function is more prone to gradient pathologies than tanh in the training process is also analyzed to show that the Euler equations can be better solved using tanh. Numerical experiments verify that even though the solution process becomes complicated, it outperforms the original physics informed neural network in terms of computational efficiency and computational accuracy and can better portray the physical phenomena of Euler equations.

High-dimensional order parameters and neural network classifiers applied to amorphous ices.

Amorphous ice phases are key constituents of water's complex structural landscape. This study investigates the polyamorphic nature of water, focusing on the complexities within low-density amorphous ice (LDA), high-density amorphous ice, and the recently discovered medium-density amorphous ice (MDA). We use rotationally invariant, high-dimensional order parameters to capture a wide spectrum of local symmetries for the characterization of local oxygen environments. We train a neural network to classify these local environments and investigate the distinctiveness of MDA within the structural landscape of amorphous ice. Our results highlight the difficulty in accurately differentiating MDA from LDA due to structural similarities. Beyond water, our methodology can be applied to investigate the structural properties and phases of disordered materials.

From Oxygen Redox to Sulfur Redox: A Paradigm for Li-Rich Layered Cathodes.

The utilization of anionic redox chemistry provides an opportunity to further improve the energy density of Li-ion batteries, particularly for Li-rich layered oxides. However, oxygen-based hosts still suffer from unfavorable structural rearrangement, including the oxygen release and transition metal (TM)-ion migration, in association with the tenuous framework rooted in the ionicity of the TM-O bonding. An intrinsic solution, by using a sulfur-based host with strong TM-S covalency, is proposed here to buffer the lattice distortion upon the highly activating sulfur redox process, and it achieves howling success in stabilizing the host frameworks. Experimental results demonstrate the prolonged preservation of the layered sulfur lattice, especially the honeycomb superlattice, during the Li+ extraction/insertion process in contrast to the large structural degeneration in Li-rich oxides. Moreover, the Li-rich sulfide cathodes exhibited a negligible overpotential of 0.08 V and a voltage drop of 0.13 mV/cycle, while maintaining a substantial reversible capacity upon cycling. These superior electrochemical performances can be unambiguously ascribed to the much shorter trajectories of sulfur in comparison to those of oxygen revealed by molecular dynamics simulations at a large scale (∼30 nm) and a long time scale (∼300 ps) via high-dimensional neural network potentials during the delithiation process. Our findings highlight the importance of stabilizing host frameworks and establish general guidance for designing Li-rich cathodes with durable anionic redox chemistry.

A novel sp3 carbon allotrope with 4+5+6+7+8 odd-even ring.

In this study, we report a novel monoclinic phase of carbon that contains 4+5+6+7+8 member rings in P21/m symmetry, identified by applying the stochastic surface walking method combined with high dimensional neural network potentials. We demonstrate that this phase possesses lower energy than graphite above 21.5 GPa. The phonon spectra show that this structure is stable under ambient pressure. This phase is a super hard material with a shear hardness as high as 81.9 GPa while it possesses an indirect band gap of 3.16 eV. The energy barrier of graphite to the Y phase is 0.27 eV, slightly higher than that of the hexagonal diamond (0.21 eV) in a similar phase transition mechanism. Two types of thermodynamically stable interfaces can be formed with the hexagonal diamond (HD), namely (001)Y//(100)HD, [100]Y//[010]HD and (001)Y//(001)HD, [010]Y//[001]HD. Although the discrete bulk Y phase is hard to synthesize, a faulted structure between HD is possible because of the well-matched interface between Y and HD. Our work shows that the Y phase may be formed in some special conditions and enhances our understanding of the formation of novel carbon allotropes.

Exploring the stable structures of cerium oxide nanoclusters using high-dimensional neural network potential

In this work, a machine learning model is developed to construct high-dimensional neural network potential to search for the most stable structures of cerium oxide nanoclusters.