Universal Function Approximation Research Articles

A Novel Consolidation Analysis Framework: Universal Function Approximators Regularized by Physical Principles

Analytical and numerical techniques are widely used to analyse and interpret soil consolidation problems. An important limitation is the requirement for significant geotechnical knowledge and expertise to derive ‘true’ solutions. This study proposes an alternative: a universal function approximator regularized with known physical principles. The proposed approach here advances previous work to solve one-dimensional (1D) consolidation considering both self-weight and large strains, as well as two-dimensional (2D) consolidation by vertical drains. Initial and boundary conditions of the studied consolidation problems are first strongly and weakly enforced for comparison. A neural network is adopted here as the function approximator. To boost prediction accuracy, a novel strategy is proposed to adaptively sample data points for training. An estimate of epistemic uncertainty is achieved using the confidence interval of ensembled multiple outputs. The results show that the proposed approach accurately predicts the behaviours of complex consolidation processes. Results also indicate that regularization using weak physical constraints can alleviate the imbalance of back-propagated gradients of different loss terms and, in turn, achieve higher accuracy. The proposed method is generic, mesh-free, more robust and can be applied to a wide range of geotechnical problems.

Generalization of Direct Adaptive Control Using Fractional Calculus Applied to Nonlinear Systems

This paper presents a new direct adaptive control (DAC) technique using Caputo’s definition of the fractional-order derivative. This is the first time a fractional-order adaptive law is introduced to work together with an integer-order stable manifold for approximating the uncertainty of a class of nonlinear systems. The DAC approach uses universal function approximators such as multi-layer perceptrons with one hidden layer or fuzzy systems to approximate the controller. This paper presents a new lemma, which elucidates and clarifies the link between the Caputo and the Riemann–Liouville definitions. The introduced lemma is useful in developing a Lyapunov candidate to prove the stability of using the proposed fractional-order adaptive law. This is further explained by a numerical example, which is provided to elucidate the practicality of using the fractional-order derivative for updating the approximator parameters. The main novelty of the results in this paper is a rigorous stability proof of the fractional DAC approach for a class of nonlinear systems that is subjected to unstructured uncertainty and deals with the adaptation mechanism using a traditional integer-order stable manifold. This makes the control scheme easier to implement in practice. The fractional-order adaptation law provides greater degrees of freedom and a potentially larger functional control structure than the conventional adaptive control. Finally, the paper demonstrates that traditional integer-order DAC is a special case of the more general fractional-order DAC scheme introduced here.

An indirect training approach for implicit constitutive modelling using recurrent neural networks and the virtual fields method

Accurate material description is crucial to achieve high-quality results in computational analysis software. Phenomenological constitutive laws generalize the material behaviour observed in simple mechanical tests. The resulting empirical expressions contain parameters that need to be calibrated through an inverse optimization process. Advancements in Digital Image Correlation (DIC) techniques have enabled the extraction of non-uniform multi-axial displacement fields, facilitating the development of heterogeneous mechanical specimens and unlocking access to richer material behaviour data. Despite these advancements, constitutive models are still limited by mathematical formulations and biases due to simplifying assumptions. Artificial Neural Networks (ANNs), as universal function approximators, could potentially drive a paradigm shift in the field. ANNs do not require explicit pre-formulations, hence avoiding the need for identifying unknown parameters. Moreover, ANNs are able to implicitly learn patterns purely from data, allowing to circumvent the biases induced by the simplifying assumptions of first-order principles. However, the training of ANNs for implicit constitutive modelling is not straightforward, given the impossibility of obtaining direct measurements of stress. This paper explores the integration of Recurrent Neural Networks (RNNs) with the Virtual Fields Method (VFM) for material modelling. This approach uses global force and displacement data to indirectly train the neural network, with the equilibrium being evaluated globally through the VFM loss function. The proposed method is (i) computationally efficient by not requiring Finite Element Analysis (FEA), (ii) compatible with both full-field measurements and numerically generated data, and (iii) able to handle experimental boundary conditions.

Composite adaptive fuzzy bipartite consensus of fractional-order multiagent systems with a switched event-triggered mechanism

This paper focuses on online recorded-data-based composite adaptive fuzzy bipartite consensus control for uncertain fractional-order multiagent systems with interconnected terms and external disturbances by employing a switched-threshold-based event-triggered mechanism (ETM) under the backstepping structure. Fuzzy logic system is used as a universal function approximation to deal with function uncertainties that are not prone to model in the system. A new composite learning adaptive parameter design scheme that synthesizes both prediction error and tracking error is developed to enhance the tracking performance, where the prediction error is raised from the utilization of online recorded data and instantaneous data. A unique switched-threshold-based ETM is introduced, in which the information transmission between the sensor and the controller is imposed on one of the individuals. One merit of this work consists in that it can automatically and rapidly switch and adjust between the fixed threshold and relative threshold ETM according to the amplitude of input signals to balance the network resources and impede the occurrence of pulse phenomenon. In addition, it is theoretically proven that the proposed scheme can ensure that all internal signals of the closed-loop system are bounded and achieve local bipartite consistent errors through the fractional Lyapunov stability criterion. Finally, a numerical example is provided to confirm the feasibility of the proposed approach.

Solving seepage equation using physics-informed residual network without labeled data

Physics-informed neural network (PINN) is an innovative universal function approximator which adds physical constraints to neural network to make the fitting results satisfy the physical laws better. In this paper, a physics-informed residual network (PIResNet) is proposed to solve the single-phase seepage equation without labeled data. The loss function is constructed by summarizing the residuals of the discretized seepage equation based on the finite volume method (FVM), and the boundary conditions are embedded in the PDE residuals in a “hard constraint” way. The PIResNet is simple in network structure, fast in convergence and easy to optimize. Furthermore, the convergence of the residual structure used in this paper is proved and the theoretical analysis of time complexity illustrates the computational efficiency of PIResNet. The correctness of the proposed method is proved and the experiments display that the solution time of PIResNet is reduced by more than six times compared to physics-informed convolutional neural networks (PICNN).

Studying turbulent flows with physics-informed neural networks and sparse data

Physics-informed neural networks (PINNs) have recently become a viable modelling method for the scientific machine-learning community. The appeal of this network architecture lies in the coupling of a deep neural network (DNN) with partial differential equations (PDEs): the DNN can be considered a universal function approximator, and the physical knowledge from the embedded PDEs regularises the network during training. This regularisation improves robustness of the network and means the network only requires sparse training data in the domain. We apply PINNs with embedded Reynolds-averaged Navier–Stokes (RANS) equations to a spatially developing adverse-pressure-gradient (APG) boundary layer 1000<Reθ<3000, and also to the periodic hill problem 5600≤Reb≤37,000. We do not use a turbulence model to close the RANS equations, rather the network infers the Reynolds-stress fields during training as a means to satisfy closure. As PINNs are able to produce robust predictions with sparse training data, this paper demonstrates how PINNs can be trained with data typical of experimental campaigns. For the APG boundary layer, we seek to quantify the amount and location of data required to model a time-averaged turbulent flow with PINNs, and show that PINNs can accurately model wall shear stress and wall pressure. We demonstrate how PINNs perform when trained without information of a critical feature of the flow (such as a separation bubble), and we highlight how the cost of training the network does not increase with Reynolds number. Finally, we undertake a series of benchmarks to demonstrate how the method scales with network size and the number of residual points in the problem. This work seeks to establish the quantity and type of data required to successfully apply PINNs to time-averaged flow without a closure model, and seeks to set expectations on the prediction quality of the network when trained on an available data set.

Supervised Learning in a Multilayer, Nonlinear Chemical Neural Network.

The development of programmable or trainable molecular circuits is an important goal in the field of molecular programming. Multilayer, nonlinear, artificial neural networks are a powerful framework for implementing such functionality in a molecular system, as they are provably universal function approximators. Here, we present a design for multilayer chemical neural networks with a nonlinear hyperbolic tangent transfer function. We use a weight perturbation algorithm to train the neural network which uses a simple construction to directly approximate the loss derivatives required for training. We demonstrate the training of this system to learn all 16 two-input binary functions from a common starting point. This work thus introduces new capabilities in the field of adaptive and trainable chemical reaction network (CRN) design. It also opens the door to potential future experimental implementations, including DNA strand displacement reactions.

Using an Opportunity Matrix to Select Centers for RBF Neural Networks

When designed correctly, radial basis function (RBF) neural networks can approximate mathematical functions to any arbitrary degree of precision. Multilayer perceptron (MLP) neural networks are also universal function approximators, but RBF neural networks can often be trained several orders of magnitude more quickly than an MLP network with an equivalent level of function approximation capability. The primary challenge with designing a high-quality RBF neural network is selecting the best values for the network’s “centers”, which can be thought of as geometric locations within the input space. Traditionally, the locations for the RBF nodes’ centers are chosen either through random sampling of the training data or by using k-means clustering. The current paper proposes a new algorithm for selecting the locations of the centers by relying on a structure known as an “opportunity matrix”. The performance of the proposed algorithm is compared against that of the random sampling and k-means clustering methods using a large set of experiments involving both a real-world dataset from the steel industry and a variety of mathematical and statistical functions. The results indicate that the proposed opportunity matrix algorithm is almost always much better at selecting locations for an RBF network’s centers than either of the two traditional techniques, yielding RBF neural networks with superior function approximation capabilities.

DeepQMC: An open-source software suite for variational optimization of deep-learning molecular wave functions.

Computing accurate yet efficient approximations to the solutions of the electronic Schrödinger equation has been a paramount challenge of computational chemistry for decades. Quantum Monte Carlo methods are a promising avenue of development as their core algorithm exhibits a number of favorable properties: it is highly parallel and scales favorably with the considered system size, with an accuracy that is limited only by the choice of the wave function Ansatz. The recently introduced machine-learned parametrizations of quantum Monte Carlo Ansätze rely on the efficiency of neural networks as universal function approximators to achieve state of the art accuracy on a variety of molecular systems. With interest in the field growing rapidly, there is a clear need for easy to use, modular, and extendable software libraries facilitating the development and adoption of this new class of methods. In this contribution, the DeepQMC program package is introduced, in an attempt to provide a common framework for future investigations by unifying many of the currently available deep-learning quantum Monte Carlo architectures. Furthermore, the manuscript provides a brief introduction to the methodology of variational quantum Monte Carlo in real space, highlights some technical challenges of optimizing neural network wave functions, and presents example black-box applications of the program package. We thereby intend to make this novel field accessible to a broader class of practitioners from both the quantum chemistry and the machine learning communities.

System for dynamic prediction of the technical condition of the equipment of a combined electric power system

This paper analyses the neuro-fuzzy modelling method for predicting the technical state of a system. The model of the system for dynamic forecasting of the technical condition of the equipment of an integrated power system consists of: a simulation model of a solar power plant, a simulation model of a wind turbine, and a simulation model of a synchronous generator acting as a thermal power plant. To perform the modelling, a special system of tools Fuzzy extension logic toolbox was used to assess the quality of the synchronous generator. The obtained estimates of the technical condition of the equipment are processed using an adaptive neuro-fuzzy inference system (ANFIS), which results in a forecast of the technical condition of the equipment at a given time interval. It has been established that fuzzy models have found wide practical application in the construction of fuzzy controllers, data recognition and processing systems, etc. Modelling of real systems with fuzzy models is based on the fact that fuzzy models of the Sugeno and Mamdani types are universal function approximators. The constructed model of the system for dynamic forecasting of the technical condition of equipment can be used in the design of renewable energy facilities, as well as in the development and testing of algorithms for control and monitoring systems. The system for assessing and predicting the state of equipment using fuzzy logic can be used to support decision-making by the power plant operator when determining the need for repair and relocation of equipment. The key advantage of the presented system is its ability to take into account dynamic changes in the state of a solar power plant depending on the transmitted operating conditions, which makes it possible to bring the obtained data even closer to the real ones. Also, a feature of the system for dynamic forecasting of the technical condition of equipment is the use of a simulation model of a solar power plant, which has a model to bring the simulation closer to real operating conditions, taking into account dust, surface heating etc.

Homogeneous Parametric Modeling of Airloads

This work proposes two parametric modeling strategies for steady aerodynamic forces. We point out that airloads are homogeneous and introduce a parametrization based on spherical harmonics and a neural network. The parametrization using spherical harmonics enables an analogue of frequency-based truncation and a variation on the Singular Value Decomposition (SVD), constituting an orthogonal decomposition of the modeled airloads. Since neural networks are universal function approximators, the model based on this allows for more flexible parametrizations, including actuations and model inversions. Both parametrization strategies are showcased for model identification and reduction purposes, highlighting their strengths and weaknesses.

IDEA: Artificial neural network models for 11-species air properties at thermochemical equilibrium

Accurate prediction of high-temperature air properties is essential in many aerodynamic applications under hypersonic flight conditions. Various curve-fit models using piecewise polynomial fittings have been commonly adopted to approximate equilibrium air properties at high temperatures. Several shortcomings including low accuracy, lack of diversity, and discontinuity at curve-fit boundary still remain, causing numerical troubles in computational procedures. To address the issues, IDEA, an open-source C++ library that enables fast and accurate computations of the equilibrium air properties and their first and second derivatives, is newly developed based on the artificial neural network (ANN).IDEA, which stands for the Infinitely Differentiable Equilibrium Air, predicts thermodynamic and transport properties of 11-species (N2, O2, N, O, NO, NO+, N+, O+, N++, O++, and e−) thermochemical equilibrium air at the temperature range up to 25,000 K and density range from 10−7 to 103 amagats. The training data is constructed from the kinetic molecular theory using the equilibrium constant method with the rigid-rotor, harmonic-oscillator model. As the name suggests, IDEA's models are infinitely differentiable in the application range; thus, they have enhanced convergence in computational fluid dynamics (CFD) when using gradient-based methods. Using a newly developed training process based on the Levenberg–Marquardt algorithm with weighted mean squared error loss, IDEA provides more accurate and diverse property models with much fewer parameters than previous piecewise polynomial fitting models. In addition, the proposed training method offers easy extensions to various property models with different species data. IDEA provides C interfaces that can be used for programs in various computer languages, such as C/C++, Fortran, Python, and MATLAB. IDEA's modeling routines are thread-safe, so they can be safely used for parallel programs without performance loss. The accuracy and enhanced convergence of IDEA is demonstrated via several high-speed flow computations Program summaryProgram title: IDEACPC library link to program files:https://doi.org/10.17632/84rhtfz9n2.1Developers' repository link:https://github.com/HojunYouKr/IDEALicensing provisions: BSD-3-ClauseProgramming language: C++11Nature of problem: Accurate prediction of equilibrium air properties is essential in many aerodynamic applications under hypersonic flight conditions. Direct calculation of air properties by the kinetic molecular theory requires time-consuming iterative methods to solve nonlinear equations of species concentrations. Various piecewise polynomial models have been developed to avoid such iterations. However, the regression error of the polynomial models requires improvement, and the lack of diversity of the polynomial models makes iterative computations inevitable, causing computational inefficiency. Furthermore, these polynomial models are not continuously differentiable, which deteriorates the convergence characteristics of computational fluid dynamics (CFD) solvers.Solution method: ANN is chosen to model the equilibrium properties of air owing to its excellent capability as a universal function approximator and better adaptability to a large dataset than kernel methods. The hyperbolic tangent activation function makes the ANN model infinitely differentiable, which enables to use a superlinear training process and thus improves convergence characteristics. The newly proposed superlinear training process based on the Levenberg–Marquardt algorithm with weighted mean squared error loss provides more accurate and diverse property models with much fewer parameters than the existing polynomial models. As a result, the proposed ANN model is completely free from iterative computations.

Interpretable polynomial neural ordinary differential equations.

Neural networks have the ability to serve as universal function approximators, but they are not interpretable and do not generalize well outside of their training region. Both of these issues are problematic when trying to apply standard neural ordinary differential equations (ODEs) to dynamical systems. We introduce the polynomial neural ODE, which is a deep polynomial neural network inside of the neural ODE framework. We demonstrate the capability of polynomial neural ODEs to predict outside of the training region, as well as to perform direct symbolic regression without using additional tools such as SINDy.

Physics informed deep neural network embedded in a chemical transport model for the Amazon rainforest

Secondary organic aerosols (SOA) are fine particles in the atmosphere, which interact with clouds, radiation and affect the Earth’s energy budget. SOA formation involves chemistry in gas phase, aqueous aerosols, and clouds. Simulating these chemical processes involve solving a stiff set of differential equations, which are computationally expensive steps for three-dimensional chemical transport models. Deep neural networks (DNNs) are universal function approximators that could be used to represent the complex nonlinear changes in aerosol physical and chemical processes; however, key challenges such as generalizability to extended time periods, preservation of mass balance, simulating sparse model outputs, and maintaining physical constraints have limited their use in atmospheric chemistry. Here, we develop an approach of using a physics-informed DNN that overcomes previous such challenges and demonstrates its applicability for the chemical formation processes of isoprene epoxydiol SOA (IEPOX-SOA) over the Amazon rainforest. The DNN is trained with data generated by simulating IEPOX-SOA over the entire atmospheric column, using the Weather Research and Forecasting Model coupled with Chemistry (WRF-Chem). The trained DNN is then embedded within WRF-Chem to replace the computationally expensive default solver of IEPOX-SOA formation. The trained DNN predictions generalizes well with the default model simulation of the IEPOX-SOA mass concentrations and its size distribution (20 size bins) over several days of simulations in both dry and wet seasons. The embedded DNN reduces the computational expense of WRF-Chem by a factor of 2. Our approach shows promise in terms of application to other computationally expensive chemistry solvers in climate models.

Sediment load prediction in Johor river: deep learning versus machine learning models

Sediment transport is a normal phenomenon in rivers and streams, contributing significantly to ecosystem production and preservation by replenishing vital nutrients and preserving aquatic life’s natural habitats. Thus, sediment transport prediction through modeling is crucial for predicting flood events, tracking coastal erosion, planning for water supplies, and managing irrigation. The predictability of process-driven models may encounter various restrictions throughout the validation process. Given that data-driven models work on the assumption that the underlying physical process is not requisite, this opens up the avenue for AI-based model as alternative modeling. However, AI-based models, such as ANN and SVM, face problems, such as long-term dependency, which require alternative dynamic procedures. Since their performance as universal function approximation depends on their compatibility with the nature of the problem itself, this study investigated several distinct AI-based models, such as long short-term memory (LSTM), artificial neural network (ANN), and support vector machine (SVM), in predicting sediment transport in the Johor river. The collected historical daily sediment transport data from January 1, 2008, to December 01, 2018, through autocorrelation function, were used as input for the model. The statistical results showed that, despite their ability (deep learning and machine learning) to provide sediment predictions based on historical input datasets, machine learning, such as ANN, might be more prone to overfitting or being trapped in a local optimum than deep learning, evidenced by the worse in all metrics score. With RMSE = 11.395, MAE = 18.094, and R2 = 0.914, LSTM outperformed other models in the comparison.

Using feedforward neural networks to represent ecosystem dynamics for bioeconomic analysis

We applied feedforward neural networks to represent ecosystem dynamics that are vital to bioeconomic analysis, ecosystem-based management, or what-if analysis regarding the underlying natural resources. Neural networks are flexible, universal function approximators, recognized for their ability to recover complex nonlinear relationships. In this paper, we treated outputs from an end-to-end Atlantis model as synthetic data and used them as training data for the neural networks. After learning the seasonal dynamics of a multispecies system, we forecasted system states with different sets of specified harvest policies using the trained networks. We demonstrate that neural networks can capture key dynamics in part of the ecosystem efficiently, and give fast updates of states that are needed for optimization and decision-making. The trained networks are reduced and more flexible systems compared to the large-scale simulator model, which is more costly to run and does not have a format allowing human actions in the form of feedback policies, or harvest control rules, i.e. decisions depending on states as well as time.

PHYSICS-INFORMED NEURAL NETWORKS FOR INFORMED VACCINE DISTRIBUTION INMETA-POPULATIONS

Accurate numerical and physical models play an important role in modeling the spread of infectious disease as well as informing policy decisions. Vaccination programs rely on the estimation of disease parameters from limited, error-prone reported data. Using physics-informed neural networks (PINNs) as universal function approximators of the susceptible-infected-recovered (SIR) compartmentalized differential equation model, we create a data-driven framework that uses reported data to estimate disease spread and approximate corresponding disease parameters. We apply this to data from a London boarding school, demonstrating the framework's ability to produce accurate disease and parameter estimations despite noisy data. However, real-world populations contain sub-populations, each exhibiting different levels of risk and activity. Thus, we expand our framework to model meta-populations of preferentially-mixed subgroups with various contact rates, introducing a new substitution to decrease the number of parameters. Optimal parameters are estimated through PINNs which are then used in a negative gradient approach to calculate an optimal vaccine distribution plan for informed policy decisions. We also manipulate a new hyperparameter in the loss function of the PINNs network to expedite training. Together, our work creates a data-driven tool for future infectious disease vaccination efforts in heterogeneously mixed populations.

Eikonal Tomography With Physics‐Informed Neural Networks: Rayleigh Wave Phase Velocity in the Northeastern Margin of the Tibetan Plateau

AbstractWe present a novel eikonal tomography approach using physics‐informed neural networks (PINNs) for Rayleigh wave phase velocities based on the eikonal equation. The PINN eikonal tomography (pinnET) neural network utilizes deep neural networks as universal function approximators and extracts traveltimes and velocities of the medium during the optimization process. Whereas classical eikonal tomography uses a generic non‐physics based interpolation and regularization step to reconstruct traveltime surfaces, optimizing the network parameters in pinnET means solving a physics constrained traveltime surface reconstruction inversion tackling measurement noise and satisfying physics. We demonstrate this approach by applying it to 25 s surface wave data from ChinArray II sampling the northeastern Tibetan plateau. We validate our results by comparing them to results from conventional eikonal tomography in the same area and find good agreement.

On the Sufficient Condition for Solving the Gap-Filling Problem Using Deep Convolutional Neural Networks.

Deep convolutional neural networks (DCNNs) are routinely used for image segmentation of biomedical data sets to obtain quantitative measurements of cellular structures like tissues. These cellular structures often contain gaps in their boundaries, leading to poor segmentation performance when using DCNNs like the U-Net. The gaps can usually be corrected by post-hoc computer vision (CV) steps, which are specific to the data set and require a disproportionate amount of work. As DCNNs are Universal Function Approximators, it is conceivable that the corrections should be obsolete by selecting the appropriate architecture for the DCNN. In this article, we present a novel theoretical framework for the gap-filling problem in DCNNs that allows the selection of architecture to circumvent the CV steps. Combining information-theoretic measures of the data set with a fundamental property of DCNNs, the size of their receptive field, allows us to formulate statements about the solvability of the gap-filling problem independent of the specifics of model training. In particular, we obtain mathematical proof showing that the maximum proficiency of filling a gap by a DCNN is achieved if its receptive field is larger than the gap length. We then demonstrate the consequence of this result using numerical experiments on a synthetic and real data set and compare the gap-filling ability of the ubiquitous U-Net architecture with variable depths. Our code is available at https://github.com/ai-biology/dcnn-gap-filling.

Image-based deep learning of finite element simulations for fast surrogate biomechanical organ deformations

Abstract Introduction: We apply deep learning to emulate Finite Element simulations by exploiting neural networks as universal function approximators. A novel processing pipeline automates the incorporation of three-dimensional biomechanical simulation data into an image-based deep learning task where spatially resolved deformations are predicted using pixelwise regression. We show that a model trained in this way can quickly reproduce simulation results with reasonable preliminary performance. Methods: We trained a customized U-Net (32 input features) to predict displacement models based on the input force vectors. The pipeline was applied to a basic indentation process for an idealized reference geometry (puck) and a 3D liver model from the LiTS dataset. The generated 3D simulation data is normalized and projected onto 2D images using the midpoint circle algorithm such that x, y, and z coordinates are mapped to pixel values of RGB image channels. This way, 1500 simulations of the reference geometry and 500 simulations of the liver deformation with force-controlled, randomly initialized load distributions were performed. Simulations took about 2-5 minutes of computation time with FE models of 6000 and 10000 elements. The neural network was trained on the simulation database using pixelwise regression. Results: We successfully trained a model on the pixelwise regression of simulation results. The model optimization reached an f1- score of 0.69 and 0.61 for the puck and the liver. The displacement images were reprojected and compared to the 3D ground truth. The trained models reached an average surface distance of 3.1 ± 1.2 and 4.2 ± 2.2, as well as 4.3 ± 0.4 and 6.4 ± 0.2 for the Hausdorff distances. Conclusion: The model performance is still impractical for clinical decisions. Still, with mean computation times of under 250 ms, the approach's potential with a more extensive database and a more sophisticated deep learning architecture, e.g., GNN, is recognizable.