Extrapolation Capability Research Articles

A physics-informed deep neural network for surrogate modeling in classical elasto-plasticity

In this work, we present a deep neural network architecture that can efficiently surrogate classical elasto-plastic constitutive relations. The network is enriched with crucial physics aspects of classical elasto-plasticity, including additive decomposition of strains into elastic and plastic parts, and nonlinear incremental elasticity. This leads to a Physics-Informed Neural Network (PINN) surrogate model named here as Elasto-Plastic Neural Network (EPNN). Detailed analyses show that embedding these physics into the architecture of the neural network facilitates a more efficient training of the network with less training data, while also enhancing the extrapolation capability for loading regimes outside the training data. The architecture of EPNN is model and material-independent; it can be adapted to a wide range of elasto-plastic material types, including geomaterials; and experimental data can potentially be directly used in training the network. To demonstrate the robustness of the proposed architecture, we adapt its general framework to the elasto-plastic behavior of sands. We use synthetic data generated from material point simulations based on a relatively advanced dilatancy-based constitutive model for granular materials to train the neural network. The superiority of EPNN over regular neural network architectures is demonstrated through predicting unseen strain-controlled loading paths for sands with different initial densities.

An Adaptive Data-Driven Iterative Feedforward Tuning Approach Based on Fast Recursive Algorithm: With Application to a Linear Motor

The feedforward control can effectively improve the servo performance in applications with high requirements of velocity and acceleration. The iterative feedforward tuning method (IFFT) enables the possibility of both removing the need for prior knowledge of the system plant in model-based feedforward and improving the extrapolation capability for varying tasks of iterative learning control. However, most IFFT methods require to set the number of basis functions in advance, which is inconvenient to the system design. To tackle this problem, an adaptive data-driven IFFT based on a fast recursive algorithm (IFFT-FRA) is developed in this article. Explicitly, based on FRA, the proposed approach can adaptively tune the feedforward structure, which significantly increases the intelligence of the approach. Additionally, a data-based iterative tuning procedure is introduced to achieve the unbiased estimation of parameters optimization in the presence of noise. Comparative experiments on a linear motor confirm the effectiveness of the proposed approach.

Combined space–time reduced-order model with three-dimensional deep convolution for extrapolating fluid dynamics

There is a critical need for efficient and reliable active flow control strategies to reduce drag and noise in aerospace and marine engineering applications. While traditional full-order models based on the Navier–Stokes equations are not feasible, advanced model reduction techniques can be inefficient for active control tasks, especially with strong non-linearity and convection-dominated phenomena. Using convolutional recurrent autoencoder network architectures, deep-learning-based reduced-order models have been recently shown to be effective while performing several orders of magnitude faster than full-order simulations. However, these models encounter significant challenges outside the training data, limiting their effectiveness for active control and optimization tasks. In this study, we aim to improve the extrapolation capability by modifying the network architecture and integrating coupled space–time physics as an implicit bias. Reduced-order models via deep learning generally employ decoupling in spatial and temporal dimensions, which can introduce modeling and approximation errors. To alleviate these errors, we propose a novel technique for learning coupled spatial–temporal correlation using a three-dimensional convolution network. We assess the proposed technique against a standard encoder–propagator–decoder model and demonstrate a superior extrapolation performance. To demonstrate the effectiveness of the three-dimensional convolution network, we consider a benchmark problem of the flow past a circular cylinder at laminar flow conditions and use the spatiotemporal snapshots from the full-order simulations. Our proposed three-dimensional convolution architecture accurately captures the velocity and pressure fields for varying Reynolds numbers. Compared to the standard encoder–propagator–decoder network, the spatiotemporal-based three-dimensional convolution network improves the prediction range of Reynolds numbers outside of the training data.

Critical Comparison of Li-Ion Aging Models for Second Life Battery Applications

Lithium-ion batteries (LIBs) from electrified vehicles (EVs) that have reached the automotive end of life (EoL) may provide a low-cost, highly available energy storage solution for grid-connected systems, such as peak shaving and ancillary services. There are several issues related to the integration of second life batteries (SLBs) in power systems, such as the variability of the pack design and cell chemistry, in-field assessments of the state of health (SoH), and estimations of the expected lifetimes of SLBs in different power system applications. Model-based approaches are commonly used in the automotive industry for estimating/predicting the capacity and power fade trajectories of LIBs during their life. However, a large variety of models are available with different fidelities, complexities, and computational costs. The accuracy of these estimations is critical for the derivation of business models for SLB applications. This paper presents a qualitative and quantitative assessment of the performance of two well-accepted, state-of-the-art aging models, initially developed for automotive applications and here applied to different SLB applications to predict both the capacity and power fade. These models are evaluated with respect to several performance metrics, such as fidelity of estimation and capability of extrapolation outside the calibration data range. The considered models are classified as semi-empirical physics-based and empirical models, respectively. Three different SLB power profiles, bulk energy for DC fast charge stations and two frequency regulation profiles, are considered, corresponding to different ranges of the SoC, C-rates, and battery temperatures, with the aim of exciting different aging mechanisms. The numerical results provide insight for the selection of aging models for SLB applications based on their performances and limitations.

Interpolation and Extrapolation Performance Measurement of Analytical and ANN-Based Flow Laws for Hot Deformation Behavior of Medium Carbon Steel

In the present work, a critical analysis of the most-commonly used analytical models and recently introduced ANN-based models was performed to evaluate their predictive accuracy within and outside the experimental interval used to generate them. The high-temperature deformation behavior of a medium carbon steel was studied over a wide range of strains, strain rates, and temperatures using hot compression tests on a Gleeble-3800. The experimental flow curves were modeled using the Johnson–Cook, Modified-Zerilli–Armstrong, Hansel–Spittel, Arrhenius, and PTM models, as well as an ANN model. The mean absolute relative error and root-mean-squared error values were used to quantify the predictive accuracy of the models analyzed. The results indicated that the Johnson–Cook and Modified-Zerilli–Armstrong models had a significant error, while the Hansel–Spittel, PTM, and Arrhenius models were able to predict the behavior of this alloy. The ANN model showed excellent agreement between the predicted and experimental flow curves, with an error of less than 0.62%. To validate the performance, the ability to interpolate and extrapolate the experimental data was also tested. The Hansel–Spittel, PTM, and Arrhenius models showed good interpolation and extrapolation capabilities. However, the ANN model was the most-powerful of all the models.

On the improvement of the extrapolation capability of an iterative machine-learning based RANS Framework

A bounded normalization method is proposed to improve the extrapolation capability of the recently developed iterative ML based turbulence modeling framework (Liu et al., 2021), whose performance degrades when the Reynolds number is beyond the training range. The bounded normalization method constrains the input feature into a finite range, and the learning target is also re-normalized with the eddy viscosity from a traditional turbulence model. Consequently, a certain level of physical similarity can be maintained when the Reynolds number goes beyond the training range. Tests in channel flows demonstrates an improved performance against the conventional normalization method. The improved model is further tested in a spatially developing boundary layer with a varying local Reynolds number, and the result shows that the model can also predict the skin-friction and velocity profiles with a favorable accuracy, even if the model is trained under a different flow configuration.

Can neural networks extrapolate? Discussion of a theorem by Pedro Domingos

Neural networks trained on large datasets by minimizing a loss have become the state-of-the-art approach for resolving data science problems, particularly in computer vision, image processing and natural language processing. In spite of their striking results, our theoretical understanding about how neural networks operate is limited. In particular, what are the extrapolation capabilities of trained neural networks if any? In this paper we discuss a theorem of Domingos stating that “every machine learned by continuous gradient descent is approximately a kernel machine”. According to Domingos, this fact leads to conclude that all machines trained on data are mere kernel machines. We first extend Domingo’s result in the discrete case and to networks with vector-valued output. We then study its relevance and significance on simple examples. We find that in simple cases, the “neural tangent kernel” arising in Domingos’ theorem does provide understanding of the networks’ predictions. When the task given to the network grows in complexity, the interpolation capability of the network can be effectively explained by Domingos’ theorem, and no extrapolation capability of the network beyond its learning domain is found, even when the network’s structure would allow for it. We illustrate this fact on a classic perception theory problem: recovering a shape from its boundary.

Predicting single-phase solid solutions in as-sputtered high entropy alloys: High-throughput screening with machine-learning model

Searching for single-phase solid solutions (SPSSs) in high-entropy alloys (HEAs) is a prerequisite for the intentional design and manipulation of microstructures of alloys in vast composition space. However, to date, reported SPSS HEAs are still rare due to the lack of reliable guiding principles for the synthesis of new SPSS HEAs. Here, we demonstrate an ensemble machine-learning method capable of discovering SPSS HEAs by directly predicting quinary phase diagrams based only on atomic composition. A total of 2198 experimental structure data are extracted from as-sputtered quinary HEAs in the literature and used to train a random forest classifier (termed AS-RF) utilizing bagging, achieving a prediction accuracy of 94.6% compared with experimental results. The AS-RF model is then utilized to predict 224 quinary phase diagrams including ∼32,000 SPSS HEAs in Cr-Co-Fe-Ni-Mn-Cu-Al composition space. The extrapolation capability of the AS-RF model is then validated by performing first-principle calculations using density functional theory as a benchmark for the predicted phase transition of newly predicted HEAs. Finally, interpretation of the AS-RF model weighting of the input parameters also sheds light on the driving forces behind HEA formation in sputtered systems with the main contributors being: valance electron concentration, work function, atomic radius difference and elementary symmetries.

Multiaxial fatigue life prediction for various metallic materials based on the hybrid CNN‐LSTM neural network

AbstractA new algorithm optimization‐based hybrid neural network model is proposed in the present study for the multiaxial fatigue life prediction of various metallic materials. Firstly, a convolutional neural network (CNN) is applied to extract the in‐depth features from the loading sequence composed of the critical fatigue loading conditions. Meanwhile, the multiaxial historical loading information with time‐series features is retained. Then, a long short‐term memory (LSTM) network is adopted to capture the time‐series features and in‐depth features of the CNN output. Finally, a full connection layer is used to achieve dimensional transformation, which makes the fatigue life predictable. Herein, the hyperparameters of the LSTM network are automatically determined using the slime mold algorithm (SMA). The test results demonstrate that the proposed model has pleasant prediction performance and extrapolation capability, and it is suitable for the life prediction of various metallic materials under uniaxial, proportional multiaxial, nonproportional multiaxial loading conditions.

An unsupervised latent/output physics-informed convolutional-LSTM network for solving partial differential equations using peridynamic differential operator

This study presents a novel unsupervised convolutional Neural Network (NN) architecture with nonlocal interactions for solving Partial Differential Equations (PDEs). The nonlocal Peridynamic Differential Operator (PDDO) is employed as a convolutional filter for evaluating derivatives the field variable. The NN captures the time-dynamics in smaller latent space through encoder–decoder layers with a Convolutional Long–short Term Memory (ConvLSTM) layer between them. The ConvLSTM architecture is modified by employing a novel activation function to improve the predictive capability of the learning architecture for physics with periodic behavior. The physics is invoked in the form of governing equations at the output of the NN and in the latent (reduced) space. By considering a few benchmark PDEs, we demonstrate the training performance and extrapolation capability of this novel NN architecture by comparing against Physics Informed Neural Networks (PINN) type solvers. It is more capable of extrapolating the solution for future timesteps than the other existing architectures.

Semi-empirical cyclic aging model for stationary storages based on graphite anode aging mechanisms

Battery life prediction is steadily becoming more relevant due to the increased use of batteries in stationary and mobile applications. Current semi-empirical aging models are often supplemented by empirical model equations and parametrized on the entire available measurement data set. This practice limits their extrapolation capability and transferability. The model in this work describes the two important anode aging mechanisms, solid electrolyte interface (SEI) cracking and reforming and cracking of the active material, by completely physically based equations. A simple incremental capacity analysis (ICA) method is introduced to allow targeted parameterization of the model equations with measurement data, in which the aging modes associated with the respective aging mechanism are present. The overall model can accurately describe the battery capacity loss under dynamic frequency containment reserve loading. It is transferable to all graphite-based battery cell chemistries and provides a basis for future semi-empirical aging models, describing the capacity loss in a wide variety of applications by considering further aging mechanisms.

Deep hybrid model‐based predictive control with guarantees on domain of applicability

AbstractA hybrid model integrates a first‐principles model with a data‐driven model which predicts certain unknown dynamics of the process, resulting in higher accuracy than first‐principles model. Additionally, a hybrid model has better extrapolation capabilities compared with a data‐based model, which is useful for process control and optimization purposes. Nonetheless, the domain of applicability (DA) of a hybrid model is finite and should be taken into account when developing a hybrid model‐based predictive controller in order to maximize its performance. To this end, a Control Lyapunov–Barrier Function‐based model predictive controller (CLBF‐based MPC) is developed which utilizes a deep hybrid model (DHM), that is, a deep neural network (DNN) combined with a first‐principles model. Additionally, theoretical guarantees are provided on stability as well as on system states to stay within the DA of the DHM. The efficacy of the proposed control framework is demonstrated on a continuous stirred tank reactor.

Ground-Motion Model (GMM) for Crustal Earthquakes in Northern South America (NoSAm Crustal GMM)

ABSTRACT Crustal earthquakes are some of the main contributors to the seismic hazard in northern South America (NoSAm). There is evidence of historical crustal events with epicenters near populated cities, such as the 1999 Mw 6.2 Coffee Region earthquake, whose damages added up to 1.9% of Colombia’s gross domestic product and reported about 1200 deaths. Because the global crustal ground-motion models (GMMs) routinely used in seismic hazard assessments of the region are biased with respect to the available ground-motion records, this article presents a regional GMM developed using local data from earthquakes in Colombia, Ecuador, and Venezuela. The filtered database contains 709 triaxial records from 56 earthquakes, recorded at 92 stations between 1994 and 2020 by the Colombian Geological Survey. The moment magnitudes of the events range between 4.5 and 6.8, with hypocentral depths ≤60 km. The model covers rupture distances ≤350 km. The model site amplification is based on a categorization approach relying on the predominant site period, identified through the horizontal-to-vertical response ratios of 5%-damped response spectra. The proposed GMM is developed as a regionalization of the global Next Generation Attenuation-West2 Project ASK14 model. Our model corrects the misfit of the ASK14 GMM with respect to the observed ground-motion data in NoSAm for moderate magnitudes and intermediate to large distances while keeping the extrapolation capabilities. The proposed GMM considers the added attenuation for ray paths crossing the volcanic arc. Analysis of the variance components allows approximating plausible reductions of the standard deviation in future nonergodic models.

Physics-guided Bayesian neural networks by ABC-SS: Application to reinforced concrete columns

This manuscript proposes a physics-guided Bayesian neural network, which combines Approximate-Bayesian-Computation training with physics-based models. This hybrid algorithm uses the laws of physics to mitigate the lack of data, and the flexibility of neural networks to model the complexities inherent in nature. The state-of-the-art approaches often introduce the physics in the loss function, or through some known boundary conditions, and then use backpropagation to adjust the weights. However, this training method involves some rigidity and drawbacks, mostly related to the adoption of a predefined loss/likelihood function and the evaluation of its gradient during training. The use of approximate Bayesian computation as the learning engine results in a greater prediction accuracy and flexibility to quantify the uncertainty, due to the gradient-free nature of the algorithm, the absence of loss/likelihood function and the non-parametric formulation of the weights. Furthermore, the physics-based model is introduced in the forward pass of the neural network, which significantly increases the extrapolation capabilities of the proposed hybrid model. The proposed algorithm has been applied to lateral-load tests in reinforced concrete columns, providing promising results when making predictions about future loading cycles, surpassing the purely data-driven and physics-based methods as well as the state-of-the-art physics-guided neural networks. In light of the performance shown during the experiments, the proposed algorithm has the potential to become a useful tool for fast evaluation of critical buildings after seismic events.

A critical analysis of the AHRI polynomials for scroll compressor characterization

• Scroll compressor energy and mass flow surfaces with evaporating and condensing conditions. • ARHI polynomials overfit scroll compressor behavior. • Two-degree polynomials are more indicated to scroll compressors. • Less experimental points are required to characterize scroll compressor. • Pressure as polynomial variable is more suitable than temperature. This paper presents the analysis of the energy consumption and mass flow rate of scroll-type compressors. The study has included the data of several AHRI reports (especially AHRI 11 and AHRI 21) and data from other sources. A total of 7 different scroll compressors of different sizes have been considered in the study, some of them tested with various refrigerants (R134a, R32, R410A, R404a…). For all the studied compressors and refrigerants, the compressor energy consumption and mass flow rate values have been analyzed. The main objective is to better understand the dependence of these variables on the operating conditions and the refrigerant used. The analyzed data include tests following different superheat control, i.e., constant superheat or constant return temperature, so the effect of the inlet temperature on these variables is also discussed. As the main novelty of this study, the analysis of the response surfaces has allowed the authors to evaluate the most suitable correlation to use, including an analysis of the necessary experimental tests and where to place them to increase the model's accuracy. It was found that using the condensing and evaporating pressure terms is more universal than the classical temperature domain. In scroll compressors, AHRI polynomials overfits the compressor performance introducing significant deviations in the interpolation and extrapolation capabilities if the experimental data are not properly selected. Finally, it was found that lower degree polynomials are more suitable for this kind of compressor and has also the advantage of requiring fewer experimental point measurements to characterize the compressor with the corresponding cost-savings.

Accurate and Effective Nonlinear Behavioral Modeling of a 10-W GaN HEMT Based on LSTM Neural Networks

This paper presents a novel nonlinear behavioral modeling methodology based on long-short-term memory (LSTM) networks for gallium nitride (GaN) high-electron-mobility transistors (HEMTs). There are both theoretical foundations and practical implementations of the modeling procedure provided in this paper. To determine the most appropriate optimizer algorithm for the model presented in this work, four different optimization algorithms are examined. The results of both simulation and experimental validation are provided based on a 10-W GaN HEMT device. According to the developed investigation, the model is capable of extrapolating and interpolating over multiple input power levels and frequencies, including linear, weakly nonlinear, and strongly nonlinear areas. The analysis of the simulated and measured results shows that the developed model has superior performance also when considering the DC drain current ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ids</i> .). Compared with the existing support vector regression (SVR) based model and the Bayesian based model, the proposed approach shows a significantly improved extrapolation capability.

Data-Driven Models for Predicting Intrinsically Disordered Protein Polymer Physics Directly from Composition or Sequence.

The molecular-level understanding of intrinsically disordered proteins is challenging due to experimental characterization difficulties. Computational understanding of IDPs also requires fundamental advances, as the leading tools for predicting protein folding (e.g., AlphaFold), typically fail to describe the structural ensembles of IDPs. The focus of this paper is to 1) develop new representations for intrinsically disordered proteins and 2) pair these representations with classical machine learning and deep learning models to predict the radius of gyration and derived scaling exponent of IDPs. Here, we build a new physically-motivated feature called the bag of amino acid interactions representation, which encodes pairwise interactions explicitly into the representation. This feature essentially counts and weights all possible non-bonded interactions in a sequence and thus is, in principle, compatible with arbitrary sequence lengths. To see how well this new feature performs, both categorical and physically-motivated featurization techniques are tested on a computational dataset containing 10,000 sequences simulated at the coarse-grained level. The results indicate that this new feature outperforms the other purely categorical and physically-motivated features and possesses solid extrapolation capabilities. For future use, this feature can potentially provide physical insights into amino acid interactions, including their temperature dependence, and be applied to other protein spaces.

Data-driven wall models for Reynolds Averaged Navier–Stokes simulations

This article presents a data-based methodology to build Reynolds-Averaged Navier–Stokes (RANS) wall models for aerodynamic simulations at low Mach numbers. Like classical approaches, the model is based on nondimensional local quantities derived from the wall friction velocity uτ, the wall viscosity μw, and the wall density ρw. A fully-connected neural network approximates the relation u+=f(y+,p+). We consider reference data (obtained with RANS simulations based on fine meshes up to the wall) of attached turbulent flows at various Reynolds numbers over different geometries of bumps, covering a range of wall pressure gradients. After training the neural networks on a subset of the reference data, the paper assesses their ability to accurately recover data for unseen conditions on meshes that have been trimmed from the wall up to an interface height where the learned wall law is applied. The network’s interpolation and extrapolation capabilities are quantified and carefully examined. Overall, when tested within its interpolation and extrapolation capabilities, the neural network model shows good robustness and accuracy. The global error on the skin friction coefficient is a few percent and behaves consistently over all the considered test cases.

Data-driven model discovery for plasma turbulence modelling

An important problem in nuclear fusion plasmas is the prediction and control of turbulence which drives the cross-field transport, thus leading to energy loss from the system and deteriorating confinement. Turbulence, being a highly nonlinear and multiscale process, is challenging to theoretically describe and computationally model. Most advanced computational models fall into one of the two categories: fluid or gyro-kinetic. They both come at a high computational cost and cannot be applied for routine simulation of plasma discharge evolution and control. Development of reduced models based on (physics informed) artificial neural networks could potentially fulfil the need for affordable simulations of plasma turbulence. However, the training requires an extensive data base and the obtained models lack extrapolation capability to scenarios not originally encountered during training. This leads to reduced models of limited validity which may not prove adequate for predicting scenarios in future machines. In contrast, we explore a data-driven model discovery approach based on sparse regression to infer governing nonlinear partial differential equations directly from the data. Our input data are generated by simulations of drift-wave turbulence according to the Hasegawa–Wakatani and modified Hasegawa–Wakatani models. Balancing model accuracy and complexity enables the reconstruction of the systems of partial differential equations accurately describing the dynamics simulated in the input data sets. Sparse regression is not data hungry and can be extrapolated to unexplored parameter ranges. We explore and demonstrate the potential of this approach for fusion plasma turbulence modelling. The findings show that the methodology is promising for the development of reduced and computationally efficient turbulence models as well as for existing model cross-validation.

Learning and extrapolation of robotic skills using task-parameterized equation learner networks

Imitation learning approaches achieve good generalization within the range of the training data, but tend to generate unpredictable motions when querying outside this range. We present a novel approach to imitation learning with enhanced extrapolation capabilities that exploits the so-called Equation Learner Network (EQLN). Unlike conventional approaches, EQLNs use supervised learning to fit a set of analytical expressions that allows them to extrapolate beyond the range of the training data. We augment the task demonstrations with a set of task-dependent parameters representing spatial properties of each motion and use them to train the EQLN. At run time, the features are used to query the Task-Parameterized Equation Learner Network (TP-EQLN) and generate the corresponding robot trajectory. The set of features encodes kinematic constraints of the task such as desired height or a final point to reach. We validate the results of our approach on manipulation tasks where it is important to preserve the shape of the motion in the extrapolation domain. Our approach is also compared with existing state-of-the-art approaches, in simulation and in real setups. The experimental results show that TP-EQLN can respect the constraints of the trajectory encoded in the feature parameters, even in the extrapolation domain, while preserving the overall shape of the trajectory provided in the demonstrations.