Function Approximation Problems Research Articles

Power-enhanced residual network for function approximation and physics-informed inverse problems

In this study, we investigate how the updating of weights during forward operation and the computation of gradients during backpropagation impact the optimization process, training procedure, and overall performance of the neural network, particularly the multi-layer perceptrons (MLPs). This paper introduces a novel neural network structure called the Power-Enhancing residual network, inspired by highway network and residual network, designed to improve the network's capabilities for both smooth and non-smooth functions approximation in 2D and 3D settings. By incorporating power terms into residual elements, the architecture enhances the stability of weight updating, thereby facilitating better convergence and accuracy. The study explores network depth, width, and optimization methods, showing the architecture's adaptability and performance advantages. Consistently, the results emphasize the exceptional accuracy of the proposed Power-Enhancing residual network, particularly for non-smooth functions. Real-world examples also confirm its superiority over plain neural network in terms of accuracy, convergence, and efficiency. Moreover, the proposed architecture is also applied to solving the inverse Burgers' equation, demonstrating superior performance. In conclusion, the Power-Enhancing residual network offers a versatile solution that significantly enhances neural network capabilities by emphasizing the importance of stable weight updates for effective training in deep neural networks. The codes implemented are available at: https://github.com/CMMAi/ResNet_for_PINN.

Intelligent vectorial surrogate modeling framework for multi-objective reliability estimation of aerospace engineering structural systems

To improve the computational efficiency and accuracy of multi-objective reliability estimation for aerospace engineering structural systems, the Intelligent Vectorial Surrogate Modeling (IVSM) concept is presented by fusing the compact support region, surrogate modeling methods, matrix theory, and Bayesian optimization strategy. In this concept, the compact support region is employed to select effective modeling samples; the surrogate modeling methods are employed to establish a functional relationship between input variables and output responses; the matrix theory is adopted to establish the vector and cell arrays of modeling parameters and synchronously determine multi-objective limit state functions; the Bayesian optimization strategy is utilized to search for the optimal hyperparameters for modeling. Under this concept, the Intelligent Vectorial Neural Network (IVNN) method is proposed based on deep neural network to realize the reliability analysis of multi-objective aerospace engineering structural systems synchronously. The multi-output response function approximation problem and two engineering application cases (i.e., landing gear brake system temperature and aeroengine turbine blisk multi-failures) are used to verify the applicability of IVNN method. The results indicate that the proposed approach holds advantages in modeling properties and simulation performances. The efforts of this paper can offer a valuable reference for the improvement of multi-objective reliability assessment theory.

Optimal with Respect to Accuracy Recovery of Some Classes Functions by Fourier Series

Introduction. Function approximation (approximation or restoration) is widely used in data analysis, model building, and forecasting. The goal of function approximation is to find the function that best approximates the original function. This can be useful when the original function is too complex to analyze or when a model needs to be simplified for more efficient computation or interpretation. Function approximation is an important tool in science, engineering, economics, and other fields where data analysis and modeling are required. It allows you to simplify complex functions, identify patterns in the behavior of the object of study, and predict the value of a function beyond the available data. The purpose of the paper is consider the problems of approximation of a function, which on some interval is given by its values in some set of nodal points and belongs to some class of functions by trigonometric Fourier series with a given accuracy and at fulfillment of given constraints on its execution time. The main attention is paid to obtaining estimates of computational complexity (implementation time) and solving the problem of function approximation by Fourier series with a given or maximum possible accuracy using efficient algorithms for solving optimization problems. Results. The general formulation of the problem of approximation of functions by Fourier series in accordance with the technology of solving problems of computational and applied mathematics with specified values of quality characteristics is presented. Estimates of the error of the proposed approximation algorithms using for the computation of Fourier coefficients the optimal in accuracy and close to them quadrature formulas for the computation of integrals from rapidly oscillating functions of the classes of Helder and Lipschitz with given fixed values in the nodes of a fixed grid are given. The corresponding quadrature formulas and constructive estimates of the error of the method of approximation of functions of the specified classes are given. Estimates of computational complexity of the given algorithms are obtained, which allow us to set real constraints on the time of algorithm implementation with a given or maximum possible accuracy. Conclusions. A comprehensive analysis of the quality of the considered algorithms for the approximation of functions by Fourier series using the accuracy-optimal (or close to them) quadrature formulas for the computation of Fourier coefficients for the computation of integrals from rapidly oscillating functions is presented. The estimates of their main characteristics – accuracy and computational complexity – are obtained. Keywords: function approximation, Fourier series, Fourier series coefficients, approximation error, computational complexity.

Solving Maxmin Optimization Problems via Population Games

Population games are games with a finite set of available strategies and an infinite number of players, in which the reward for choosing a given strategy is a function of the distribution of players over strategies. The paper shows that, in a certain class of maxmin optimization problems, it is possible to associate a population game to a given maxmin problem in such a way that solutions to the optimization problem are found from Nash equilibria of the associated game. Iterative solution methods for maxmin optimization problems can then be derived from systems of differential equations whose trajectories are known to converge to Nash equilibria. In particular, we use a discrete-time version of the celebrated replicator equation of evolutionary game theory, also known in machine learning as the exponential multiplicative weights algorithm. The resulting algorithm can be viewed as a generalization of the Iteratively Reweighted Least Squares (IRLS) method, which is well known in numerical analysis as a useful technique for solving Chebyshev function approximation problems on a finite grid. Examples are provided to show the use of the generalized IRLS method in collective investment and in decision making under model uncertainty.

Moduli of smoothness, K-functionals and Jackson-type inequalities associated with kernel function approximation in learning theory

We give investigations on a kernel function approximation problem arising from learning theory and show the convergence rate from the view of classical Fourier analysis. First, we provide the general definition for a modulus of smoothness and a [Formula: see text]-functional, and show that they are equivalent. In particular, we give explicit representation for some moduli of smoothness. Second, we establish some Jackson-type inequalities for the approximation error associated with some non-radial kernels. Also we apply these results to some concrete classical kernel function spaces and give Jackson-type inequalities for some concrete RKHS approximation problems. Finally, we apply these discussions to learning theory and describe the learning rates with the moduli of smoothness. The tools we used are Fourier analysis and the semigroup operator. The results show that the Jackson-type inequalities of approximation by some radial kernel functions on compact set with nonempty interiors cannot be expressed with the classical moduli of smoothness.

Efficient and stable SAV-based methods for gradient flows arising from deep learning

The optimization algorithm plays an important role in deep learning and significantly affects the stability and efficiency of the training process, and consequently the accuracy of the neural network approximation. A suitable (initial) learning rate is crucial for the optimization algorithm in deep learning. However, a small learning rate is usually needed to guarantee the convergence of the optimization algorithm, resulting in a slow training process. We develop in this work efficient and energy stable optimization methods for function approximation problems as well as solving partial differential equations using deep learning. In particular, we consider the gradient flows arising from deep learning from the continuous point of view, and employ the particle method (PM) and the smoothed particle method (SPM) for the space discretization while we adopt SAV-based schemes, combined with the adaptive strategy used in Adam algorithm, for the time discretization. To illustrate the effectiveness of the proposed methods, we present a number of numerical tests to demonstrate that the SAV-based schemes significantly improve the efficiency and stability of the training as well as the accuracy of the neural network approximation. We also show that the SPM approach gives a slightly better accuracy than the PM approach. We further demonstrate the advantage of using adaptive learning rate in dealing with more complex problems.

A data-driven implicit deep adaptive neuro-fuzzy inference system capable of manifold learning for function approximation

Fuzzy Neural Networks (FNN) have the ability of decision-making based on constructing semi-ellipsoidal clusters in the input space as the antecedent parts of their fuzzy rules. To determine the output value for each input instance, FNNs consider its membership degree to different sub-regions of the input space. However, forming such meaningful sub-regions is not possible in all applications due to the nonlinear interactions among input variables and their low information gain. Indeed, the samples could be distributed on a manifold in the input space. Therefore, to cover the input space, we need lots of rules, each representing a small region of input space. This issue decreases the generalization ability of the model along with its explainability. Consequently, to efficiently form fuzzy rules, first, it is necessary to unfold the manifold by mapping the samples to an appropriate embedding space. Next, the fuzzy rules in the form of semi-ellipsoidal regions should be constructed in this extracted feature space. Deep Fuzzy Neural Networks address this problem by representation learning through stacking multiple cascade mapping layers. In this paper, we propose a novel approach for nonlinear function approximation and time-series prediction problems, based on using the kernel trick to implicitly learn the mapping function to the new feature space. Moreover, to initialize the fuzzy rules, a KNN-based method using the kernel trick is proposed. A hierarchical Levenberg–Marquardt approach is applied to learn the model’s parameters. The performance and structure of the proposed method are studied and compared with some other relevant methods in synthetic and real-world benchmarks. Based on these experiments, the proposed method has the best performance with the most parsimonious architecture. According to these experiments, the test RMSE of the proposed method is 0.002 for Mc-Glass chaotic time-series prediction, 0.015 for a Nonlinear dynamic system identification, 0.0345 for Box–Jenkins nonlinear system identification, 0.0609 for Fuel consumption prediction of automobiles, 10.24 for Sydney stock price tracking, and 0.595 for California housing.

Mathematical Modeling on a Physics-Informed Radial Basis Function Network

The article is devoted to approximate methods for solving differential equations. An approach based on neural networks with radial basis functions is presented. Neural network training algorithms adapted to radial basis function networks are proposed, in particular adaptations of the Nesterov and Levenberg-Marquardt algorithms. The effectiveness of the proposed algorithms is demonstrated for solving model problems of function approximation, differential equations, direct and inverse boundary value problems, and modeling processes in piecewise homogeneous media.

Comparison of estimating vegetation index for outdoor free-range pig production using convolutional neural networks.

This study aims to predict the change in corn share according to the grazing of 20 gestational sows in a mature corn field by taking images with a camera-equipped unmanned air vehicle (UAV). Deep learning based on convolutional neural networks (CNNs) has been verified for its performance in various areas. It has also demonstrated high recognition accuracy and detection time in agricultural applications such as pest and disease diagnosis and prediction. A large amount of data is required to train CNNs effectively. Still, since UAVs capture only a limited number of images, we propose a data augmentation method that can effectively increase data. And most occupancy prediction predicts occupancy by designing a CNN-based object detector for an image and counting the number of recognized objects or calculating the number of pixels occupied by an object. These methods require complex occupancy rate calculations; the accuracy depends on whether the object features of interest are visible in the image. However, in this study, CNN is not approached as a corn object detection and classification problem but as a function approximation and regression problem so that the occupancy rate of corn objects in an image can be represented as the CNN output. The proposed method effectively estimates occupancy for a limited number of cornfield photos, shows excellent prediction accuracy, and confirms the potential and scalability of deep learning.

A gradient-enhanced physics-informed neural network (gPINN) scheme for the coupled non-fickian/non-fourierian diffusion-thermoelasticity analysis: A novel gPINN structure

This paper proposes a modified artificial intelligence (AI) approach based on the gradient-enhanced physics-informed neural network (gPINN) with a novel structure for the generalized coupled non-Fickian/non-Fourierian diffusion-thermoelasticity analysis. Previous successful application of gPINN in function approximation and single partial differential equation (PDE) problems gave us enough motivation to develop its application with a novel structure for solving a system of coupled PDEs. The governing equations are formulated for a copper-made strip based upon the Lord-Shulman theory of the generalized coupled thermoelasticity. In order to shed light on the capacity of the gPINN-based approach, three examples with different boundary conditions for a one-dimensional half-space are presented, as well as an example for a two-dimensional half-space. The primary goal is to investigate the transient behavior of field variables (i.e., non-dimensional molar concentration, non-dimensional displacement, and non-dimensional temperature). Variations of dimensionless stress are also examined in depth. Performance comparisons between the gPINN-based method and commonly used approaches support the efficacy of the gPINN with a novel structure. Detailed sensitivity analyses are provided to sufficiently explore the impact of neural network hyperparameters. Moreover, the effect of relaxation time on the behavior of mass concentration is addressed in a single scenario. Overall, the proposed gPINN-based method with a novel structure yields consistently encouraging and satisfactory results, and this approach entails the capability to be applied to more complicated problems. Numerical results underline the importance of the weights assigned to the derivatives of the loss terms to achieve more accurate predictions. Nonparametric statistical tests offer further evidence for exceptional agreement between gPINN solutions with certain weights and outcomes derived by commonly used analytical techniques.

Instability and its prediction of an inverted inhomogeneous plate loaded by a subsonic airflow: Theory and experiment

This paper presents a study on the instability of an inverted cantilevered plate with an inhomogeneous cross-section in axial flow from both theory and experiment. The plate is free at the leading edge and clamped at the trailing one. It has a cross-section with discontinuity, bringing difficulty in solving the fluid–structure interaction equation by the traditional methods. We study this problem in state space and transfer its solution into a function approximation problem. We bring no approximation at the first equation level, and the derived instability equation is on the continuum. A numerical scheme is proposed to solve the instability equation. Series expansion and least square method apply to the numerical solutions. The convergence and reliability of the present method are entirely tested. This present modeling method preserves the original information of the problem and facilitates the exploration of the effects of inhomogeneity. Based on the present continuum instability equation, prediction formulas for critical dynamic pressure and instability slope of an inhomogeneous plate are first given and verified. They can apply to the stability prediction of arbitrary cross-sectional plates and have application potential for the plate design. In addition, we report an optimized stiffness distribution of the plate for the maximum critical dynamic pressure. Finally, a wind tunnel test is designed and implemented. A comparative study shows that the present method and prediction formula agree well with the experiment. The present modeling method for a fluid–structure interaction problem in a continuous sense, avoiding approximation at the beginning level, can serve as another new way to address the static instability problem of plates.

Incremental Multilayer Broad Learning System With Stochastic Configuration Algorithm for Regression

Broad learning system (BLS) is a novel randomized learning framework which has a faster modeling efficiency. Although BLS with incremental learning has a better extendibility for updating model rapidly, the incremental mode of BLS lacks self-supervision mechanism which cannot adjust the structure adaptively. Learning from the idea of stochastic configuration network (SCN), a novel incremental multi-layer broad learning system based on stochastic configuration (SC) algorithm is proposed for regression, termed as IMLBLS-SC. Firstly, to improve the feature learning ability, SC algorithm is adopted to configure the parameters of enhancement nodes instead of random weights. Secondly, the multi-layer model with enhancement nodes can be added gradually according to supervision mechanism without human intervention. Thirdly, all the enhancement nodes and feature nodes are fully connected with output nodes. Finally, 2 function approximation problems and 8 classical datasets are selected to verify the regression performance of IMLBLS-SC, experimental results demonstrate that IMLBLS-SC outperforms the RVFLN, SCN, BLS and BSCN.

Nonlinear independent component analysis for discrete-time and continuous-time signals

We study the classical problem of recovering a multidimensional source signal from observations of nonlinear mixtures of this signal. We show that this recovery is possible (up to a permutation and monotone scaling of the source’s original component signals) if the mixture is due to a sufficiently differentiable and invertible but otherwise arbitrarily nonlinear function and the component signals of the source are statistically independent with ‘nondegenerate’ second-order statistics. The latter assumption requires the source signal to meet one of three regularity conditions which essentially ensure that the source is sufficiently far away from the nonrecoverable extremes of being deterministic or constant in time. These assumptions, which cover many popular time series models and stochastic processes, allow us to reformulate the initial problem of nonlinear blind source separation as a simple-to-state problem of optimisation-based function approximation. We propose to solve this approximation problem by minimizing a novel type of objective function that efficiently quantifies the mutual statistical dependence between multiple stochastic processes via cumulant-like statistics. This yields a scalable and direct new method for nonlinear Independent Component Analysis with widely applicable theoretical guarantees and for which our experiments indicate good performance.

AN ENHANCED DIFFERENTIAL EVOLUTION ALGORITHM WITH ADAPTIVE WEIGHT BOUNDS FOR EFFICIENT TRAINING OF NEURAL NETWORKS

Artificial neural networks are essential intelligent tools for various learning tasks. Training them is challenging due to the nature of the data set, many training weights, and their dependency, which gives rise to a complicated high-dimensional error function for minimization. Thus, global optimization methods have become an alternative approach. Many variants of differential evolution (DE) have been applied as training methods to approximate the weights of a neural network. However, empirical studies show that they suffer from generally fixed weight bounds. In this research, we propose an enhanced differential evolution algorithm with adaptive weight bound adjustment (DEAW) for the efficient training of neural networks. The DEAW algorithm uses small initial weight bounds and adaptive adjustment in the mutation process. It gradually extends the bounds when a component of a mutant vector reaches its limits. We also experiment with using several scales of an activation function with the DEAW algorithm. Then, we apply the proposed method with its suitable setting to solve function approximation problems. DEAW can achieve satisfactory results compared to exact solutions.

Fast Computational Approach to the Levenberg-Marquardt Algorithm for Training Feedforward Neural Networks

Abstract This paper presents a parallel approach to the Levenberg-Marquardt algorithm (LM). The use of the Levenberg-Marquardt algorithm to train neural networks is associated with significant computational complexity, and thus computation time. As a result, when the neural network has a big number of weights, the algorithm becomes practically ineffective. This article presents a new parallel approach to the computations in Levenberg-Marquardt neural network learning algorithm. The proposed solution is based on vector instructions to effectively reduce the high computational time of this algorithm. The new approach was tested on several examples involving the problems of classification and function approximation, and next it was compared with a classical computational method. The article presents in detail the idea of parallel neural network computations and shows the obtained acceleration for different problems.

Memory, evolutionary operator, and local search based improved Grey Wolf Optimizer with linear population size reduction technique

Optimization of multi-modal functions is challenging even for evolutionary and swarm-based algorithms as it requires an efficient exploration for finding the promising region of the search space, and effective exploitation to precisely find the global optimum. Grey Wolf Optimizer (GWO) is a recently developed metaheuristic algorithm that is inspired by nature with a relatively small number of parameters for tuning. However, GWO and most of its variants may suffer from the lack of population diversity, premature convergence, and the inability to preserve a good balance between exploratory and exploitative behaviors. To address these limitations, this work proposes a new variant of GWO incorporating memory, evolutionary operators, and a stochastic local search technique. It further integrates Linear Population Size Reduction (LPSR) technique. The proposed algorithm is comprehensively tested on 23 numerical benchmark functions, high dimensional benchmark functions, 13 engineering case studies, four data classifications, and three function approximation problems. The benchmark functions are mostly taken from the CEC 2005 and CEC 2010 special sessions, and they include rotated, shifted functions. The engineering case studies are from the CEC 2020 real-world non-convex constrained optimization problems. The performance of the proposed GWO is compared with popular metaheuristics, namely, particle swarm optimization (PSO), gravitational search algorithm (GSA), slap swarm algorithm (SSA), differential evolution (DE), self-adaptive differential evolution (SADE), basic GWO and its three recently improved variants. Statistical analysis and Friedman tests have been conducted to thoroughly compare their performance. The obtained results demonstrate that the proposed GWO outperforms the algorithms compared for the benchmark functions and engineering case studies tested.

Batch Gradient Learning Algorithm with Smoothing L1 Regularization for Feedforward Neural Networks

Regularization techniques are critical in the development of machine learning models. Complex models, such as neural networks, are particularly prone to overfitting and to performing poorly on the training data. L1 regularization is the most extreme way to enforce sparsity, but, regrettably, it does not result in an NP-hard problem due to the non-differentiability of the 1-norm. However, the L1 regularization term achieved convergence speed and efficiency optimization solution through a proximal method. In this paper, we propose a batch gradient learning algorithm with smoothing L1 regularization (BGSL1) for learning and pruning a feedforward neural network with hidden nodes. To achieve our study purpose, we propose a smoothing (differentiable) function in order to address the non-differentiability of L1 regularization at the origin, make the convergence speed faster, improve the network structure ability, and build stronger mapping. Under this condition, the strong and weak convergence theorems are provided. We used N-dimensional parity problems and function approximation problems in our experiments. Preliminary findings indicate that the BGSL1 has convergence faster and good generalization abilities when compared with BGL1/2, BGL1, BGL2, and BGSL1/2. As a result, we demonstrate that the error function decreases monotonically and that the norm of the gradient of the error function approaches zero, thereby validating the theoretical finding and the supremacy of the suggested technique.

Generic compressive strength prediction model applicable to multiple lithologies based on a broad global database

In this study, we present a global database of ten parameters, which include measurements of rock index properties, strength stiffness and dynamic properties. Hoek–Brown constant mi, is included, and was estimated, using Hoek and Brown proposed guidelines for determining mi values for different rock types that can be used for preliminary design when triaxial tests are not available. This broad database is compiled from 96 studies and is labelled as “ROCK/10/4025”, to describe the type of geomaterial, the number of the parameters, and the number of the data samples included. It consists of 35.4 % igneous, 54.8 % sedimentary, and 9.2% metamorphic rocks. The purpose of this paper is to propose a generic soft computing model applicable to multiple lithologies, that can become more reliable and perhaps more suitable for a specific site study when used in order to densify often limited similar site-specific data. To this end four broad samples of data were selected, and served as training data sets for developed machine learning models, to develop a generic compression strength prediction model applicable to multiple lithologies. The suggested algorithms in this study are Back-Propagation Artificial Neural Networks, Artificial Neuro-Fuzzy Inference Systems, Support Vector Machines, Nearest Neighbour classifiers and Ensemble Bagged Trees. According to the findings of this study, Artificial Neuro-Fuzzy Inference Systems model performance was found to be marginally superior, while Back Propagation Artificial Neural Networks, Support Vector Machines and Ensemble Bagged Trees models were found to have good performance. Constant mi seems to be an important training parameter when training predictive models centred on data from multiple lithologies. As a result, we can suggest that these models are powerful tools that allow for a reliable estimation of compressive strength, based on the performance indicators. The performance was found to be 70%–82% when the problem of compressive strength prediction was approached as a classification problem (that is successful prediction of class from very weak to very strong), and 80%–96% when solved as a function approximation problem.

Static aeroelastic instability analysis of an inverted plate in ground effect and comparison with a wind-tunnel test

This paper visits the static instability of an inverted cantilevered plate in axial flow with the ground effect. This plate model is clamped at the trailing edge, and its leading-edge is allowed to move freely. A theoretical analysis strategy and corresponding convergent numerical solutions are reported and compared with a wind-tunnel experiment. In theory, we try a new way to study such an instability problem. We derive the instability equation by operator theory in state-space formulation and model such an instability problem as a mathematical function approximation problem. Glauert’s expansion and the least square method are applied for the numerical solutions of vorticity in ground effect using the mirror image method. The unknown function of the plate slope is expanded as a series of essential polynomial functions based on the Weierstrass theorem, and the least square method is applied for its numerical solutions. The experimental study develops a multi-point contact method using a control rod to preload the tested plate with a slight static deformation close to its natural modal shape. The contact state change between the initially deformed tested plate and the control rod is reflected by the evolution of the voltage output of a designed circuit. A criterion for determining the critical condition of plate instability is established based on such voltage change, and it has a better anti-interference ability to the flow. Results show that the present theoretical analysis agrees well with the experiment and other archived theories. The critical plate slopes (i.e., the plate instability modes) are not dependent on the ground confinement. This conclusion allows us to establish an analytical approximation between the critical dynamic pressure and the ground confinement. The present study and results can successfully predict the plate instability in the ground effect.

Efficient function approximation on general bounded domains using splines on a Cartesian grid

Functions on a bounded domain in scientific computing are often approximated using piecewise polynomial approximations on meshes that adapt to the shape of the geometry. We study the problem of function approximation using splines on a regular but oversampled grid that is defined on a bounding box. This approach allows the use of high-order and highly structured splines as a basis for piecewise polynomials. The methodology is analogous to that of Fourier extensions, using Fourier series on a bounding box, which leads to spectral accuracy for smooth functions. However, Fourier extension approximations involve solving a highly ill-conditioned linear system, and this is an expensive step. The computational complexity of recent algorithms is $\mathcal {O}\left (N\log ^{2}\left (N\right )\right )$ in 1-D and $\mathcal {O}\left (N^{2}\log ^{2}\left (N\right )\right )$ in 2-D. We show that, compared to Fourier extension, the compact support of B-splines enables improved complexity for multivariate approximations, namely $\mathcal {O}(N)$ in 1-D, $\mathcal {O}\left (N^{3/2}\right )$ in 2-D and more generally $\mathcal {O}\left (N^{3(d-1)/d}\right )$ in d-D with d > 1. By using a direct sparse QR solver for a related linear system, we also observe that the computational complexity can be nearly linear in practice. This comes at the cost of achieving only algebraic rates of convergence. Our statements are corroborated with numerical experiments and Julia code is available.