Universal Approximation Research Articles

Gradient free stochastic training of ANNs, with local approximation in partitions

We present a numerical scheme for computation of Artificial Neural Networks (ANN) weights, which stems from the Universal Approximation Theorem, avoiding costly iterations. The proposed algorithm adheres to the underlying theory, is highly fast, and results in remarkably low errors when applied to regression and classification problems of complex data sets with {textbf{x}} in {mathbb {R}}^{n} (e.g. Griewank, Gomez-Levy, Shekel, and Polynomial functions) with random noise addition (i.e. Uniform, Normal, Generalized Pareto, Log-Normal, and a mixture of Log-Normal, Exponential, and Frechet), as well as the database for handwritten digits recognition MNIST (Modified National Institute of Standards and Technology) with 7times 10^4 images. The same mathematical formulation was found capable of approximating highly nonlinear functions in multiple dimensions, with low errors (e.g. 10^{-10}) for the test set of the unknown functions, their higher-order partial derivatives, as well as numerically solving Partial Differential Equations, such as those appearing in Physics, Engineering, Environmental Sciences, etc. The method is based on the calculation of the weights of each neuron in small neighbourhoods of the data. Accordingly, optimization of hyperparameters is not necessary, as the number of neurons stems directly from the dimensionality of the data, further improving the algorithmic speed. Under this setting, overfitting is inherently avoided, and the results are interpretable and reproducible. The complexity of the proposed algorithm is of class P with {mathcal {O}}(mNni_{cl} + Nmn^2+Nn^3 + mN^2+N^3) computing time, with respect to the observations m, features n, and Neurons N, contrary to the NP-Complete class of standard algorithms for ANN training. The performance of the method is high, irrespective of the size of the data set, and the test set errors are similar or smaller than the training errors, indicating the generalization efficiency of the algorithm. A supplementary computer code in Julia and Python Languages is provided, which can be used to reproduce the validation examples, and/or apply the algorithm to other data sets.

Meshless methods for American option pricing through Physics-Informed Neural Networks

Nowadays, Deep Learning is drastically revolutionizing financial research as well as industry. Many methods have been discussed in the last few years, mainly related to option pricing. In fact, traditional approaches such as Monte Carlo simulation or finite difference methods are seriously harmed by multi-dimensional underlying and path dependency. Thus, dealing with particular contracts such as American multi-asset options is still rough. This paper addresses such a problem by pricing said put options with a novel meshless methodology, named Physics-Informed Neural Networks (PINNs), based on Artificial Intelligence. PINN paradigm has been recently introduced in Deep Learning literature. It exploits the theoretical background of the universal approximation theorem for neural networks to solve Partial Differential Equations numerically. This Deep Learning meshless method incorporates the equation and its initial and boundary conditions thanks to a specially designed loss function. We develop a suitable PINN for the proposed problem by introducing an algorithmic trick for improving the convergence of the free boundary problem. Furthermore, the worthiness of the proposal is assessed by several experiments concerned with single and multi-asset options. Finally, a parametric model is built to benefit further studies of option value behaviour related to particular market conditions.

Cooperative multiple elastic‐joint arms control using the (p, q)‐analogue of Bernstein operators as the uncertainty approximator

AbstractThis paper presents the position‐force tracking control problem of cooperative multiple elastic‐joint robotic arms, encountering model nonlinearities, unknown perturbations, and dynamic uncertainties. To this mean, an adaptive function approximation technique (FAT) is proposed, authorizing the reference trajectory to be tracked by the object. This potential originates from the universal approximation feature of the FAT‐based methodologies. The (p, q)‐analogue of the Bernstein operators is utilized for this purpose. Since the systems' parameters are not precisely known, adaptive laws are employed for adjusting the uncertainty coefficients. The Lyapunov stability theorem assures that all signals in the error space remain uniformly ultimately bounded (UUB). In the end, two elastic‐joint arms carrying a rigid object are utilized for validation of the theoretical outcomes. The suggested strategy is also compared to the Chebyshev neural network (CNN). The results show the benefit of the existing method, managing the system even in the presence of disturbances and uncertainties with lower tuning parameters compared with CNN.

Distributional reinforcement learning with unconstrained monotonic neural networks

The distributional reinforcement learning (RL) approach advocates for representing the complete probability distribution of the random return instead of only modelling its expectation. A distributional RL algorithm may be characterised by two main components, namely the representation of the distribution together with its parameterisation and the probability metric defining the loss. The present research work considers the unconstrained monotonic neural network (UMNN) architecture, a universal approximator of continuous monotonic functions which is particularly well suited for modelling different representations of a distribution. This property enables the efficient decoupling of the effect of the function approximator class from that of the probability metric. The research paper firstly introduces a methodology for learning different representations of the random return distribution (PDF, CDF and QF). Secondly, a novel distributional RL algorithm named unconstrained monotonic deep Q-network (UMDQN) is presented. To the authors’ knowledge, it is the first distributional RL method supporting the learning of three, valid and continuous representations of the random return distribution. Lastly, in light of this new algorithm, an empirical comparison is performed between three probability quasi-metrics, namely the Kullback–Leibler divergence, Cramer distance, and Wasserstein distance. The results highlight the main strengths and weaknesses associated with each probability metric together with an important limitation of the Wasserstein distance.

Refinements of Approximation Results of Conditional Restricted Boltzmann Machines.

Conditional restricted Boltzmann machines (CRBMs) are the conditional variant of restricted Boltzmann machines (RBMs), which are used to simulate conditional probability distributions. While promising for practical applications, there is a lack of theoretical studies on the approximation ability of CRBMs. In this article, by contributing analysis tools, especially designed for the conditional models, we improve the results of the representational power of CRBMs based on existing work on RBMs. We first study the properties of CRBMs to approximate conditional Markov random fields. On this basis, a universal approximation result is obtained by deriving upper bounds on the minimal number of hidden units, which improves the previous result. Furthermore, the result about maximal approximation errors of CRBMs is also improved by reducing the number of hidden units that can guarantee approximations within a given error tolerance. Furthermore, the representational efficiency of CRBMs in computing deterministic conditional distributions is investigated. Specifically, we show that there are exponentially many deterministic conditional distributions that cannot be computed by CRBMs whose size does not exponentially grow with the number of input units. Some specific examples of these hard-to-present deterministic conditional distributions are provided. Finally, some numerical experiments are carried out to verify the theoretical results of the properties of CRBMs to approximate conditional Markov random fields and the universal approximation of CRBMs.

Optical Logic Gate Operations With Single-Pixel Imaging

Optical computing has potential advantages of light-speed parallel processing and low power consumption over electronic computing. Logic gates are fundamental building blocks of modern computing systems. In this work, two types of single-pixel imaging systems are investigated for performing a non-imaging task of optical logic gate operations for the first time. In the first scheme, the spatial light intensity distribution of object image and illumination pattern is encoded based on the input binary values. Then the logic gate output result is indicated by the maximum single-pixel intensity when the encoded object image is incoherently illuminated by a most matched pattern. In the second scheme, the light amplitudes in corresponding sub-regions are directly proportional to the binary input values. A universal linear approximation model in the complex-amplitude domain allows the detected light intensity of a single-pixel holography system representing the logic gate output result. Various logic gate operations such as AND, OR and XOR can be implemented by these two systems in an individual or compound manner.

Which neural networks can be computed by an algorithm? – Generalised hardness of approximation meets Deep Learning

AbstractClassical hardness of approximation (HA) is the phenomenon that, assuming P ≠ NP, one can easily compute an ϵ‐approximation to the solution of a discrete computational problem for ϵ > ϵ0 > 0, but for ϵ < ϵ0 – where ϵ0 is the approximation threshold – it becomes intractable. Recently, a similar yet more general phenomenon has been documented in AI: Generalised hardness of approximation (GHA). This phenomenon includes the following occurrence: For any approximation threshold ϵ1 > 0, there are AI problems for which provably there exist stable neural networks (NNs) that solve the problem, but no algorithm can compute any NN that approximates the AI problem to ϵ1‐accuracy. Moreover, this issue is independent of the P vs NP question and thus is a rather different mathematical phenomenon than HA. GHA implies that the universal approximation theorem for NNs only provides a partial understanding of the power of NNs in AI. Thus, a classification theory describing which NNs can be computed by algorithms to particular accuracies is needed to fill this gap. We initiate such a theory by showing the correspondence between the functions that can be computed to ϵ‐accuracy by an algorithm and those functions that can be approximated by NNs which can be computed to ϵ̂‐accuracy by an algorithm. In particular, the approximation thresholds ϵ and ϵ̂ cannot differ by more than a factor of 12. This means that computing function approximations through NNs will be optimal – in the sense of best approximation accuracy achievable by an algorithm – up to a small constant, compared to any other computational technique.

A Gradient-Based Particle-Bat Algorithm for Stochastic Configuration Network

Stochastic configuration network (SCN) is a mathematical model of incremental generation under a supervision mechanism, which has universal approximation property and advantages in data modeling. However, the efficiency of SCN is affected by some network parameters. An optimized searching algorithm for the input weights and biases is proposed in this paper. An optimization model with constraints is first established based on the convergence theory and inequality supervision mechanism of SCN; Then, a hybrid bat-particle swarm optimization algorithm (G-BAPSO) based on gradient information is proposed under the framework of PSO algorithm, which mainly uses gradient information and local adaptive adjustment mechanism characterized by pulse emission frequency to improve the searching ability. The algorithm optimizes the input weights and biases to improve the convergence rate of the network. Simulation results over some datasets demonstrate the feasibility and validity of the proposed algorithm. The training RMSE of G-BAPSO-SCN increased by 5.57×10−5 and 3.2×10−3 compared with that of SCN in the two regression experiments, and the recognition accuracy of G-BAPSO-SCN increased by 0.07% on average in the classification experiments.

Nonlinear function-on-scalar regression via functional universal approximation.

We consider general nonlinear function-on-scalar (FOS) regression models, where the functional response depends on multiple scalar predictors in a general unknown nonlinear form. Existing methods either assume specific model forms (e.g., additive models) or directly estimate the nonlinear function in a space with dimension equal to the number of scalar predictors, which can only be applied to models with a few scalar predictors. To overcome these shortcomings, motivated by the classic universal approximation theorem used in neural networks, we develop a functional universal approximation theorem which can be used to approximate general nonlinear FOS maps and can be easily adopted into the framework of functional data analysis. With this theorem and utilizing smoothness regularity, we develop a novel method to fit the general nonlinear FOS regression model and make predictions. Our new method does not make any specific assumption on the model forms, and it avoids the direct estimation of nonlinear functions in a space with dimension equal to the number of scalar predictors. By estimating a sequence of bivariate functions, our method can be applied to models with a relatively large number of scalar predictors. The good performance of the proposed method is demonstrated by empirical studies on various simulated and realdatasets.

Sediment load prediction in Johor river: deep learning versus machine learning models

Sediment transport is a normal phenomenon in rivers and streams, contributing significantly to ecosystem production and preservation by replenishing vital nutrients and preserving aquatic life’s natural habitats. Thus, sediment transport prediction through modeling is crucial for predicting flood events, tracking coastal erosion, planning for water supplies, and managing irrigation. The predictability of process-driven models may encounter various restrictions throughout the validation process. Given that data-driven models work on the assumption that the underlying physical process is not requisite, this opens up the avenue for AI-based model as alternative modeling. However, AI-based models, such as ANN and SVM, face problems, such as long-term dependency, which require alternative dynamic procedures. Since their performance as universal function approximation depends on their compatibility with the nature of the problem itself, this study investigated several distinct AI-based models, such as long short-term memory (LSTM), artificial neural network (ANN), and support vector machine (SVM), in predicting sediment transport in the Johor river. The collected historical daily sediment transport data from January 1, 2008, to December 01, 2018, through autocorrelation function, were used as input for the model. The statistical results showed that, despite their ability (deep learning and machine learning) to provide sediment predictions based on historical input datasets, machine learning, such as ANN, might be more prone to overfitting or being trapped in a local optimum than deep learning, evidenced by the worse in all metrics score. With RMSE = 11.395, MAE = 18.094, and R2 = 0.914, LSTM outperformed other models in the comparison.

Intelligent Discrete Deep Learning Based Classification Methodology in Chemometrics.

In recent years, deep learning models have attracted much attention for classification purposes in chemometrics. The popularity of deep learning models in this field comes from their unique features like universal approximation capability with the desired accuracy. Deep learning classifiers use several intelligent processing layers to model mixed, complex, and nonlinear patterns in the underlying data sets, which is why the development of deep learning based models has never been stopped in the chemometrics literature. Despite the variety of deep learning classification models used in this field, they all use a continuous distance-based cost function in their learning processes. Although using a continuous cost function for learning deep classifiers is a common approach, it conflicts with the discrete nature of the classification problem. In fact, applying a continuous cost function for inherently discrete classification problems can reduce the performance of the classification. In this research, a novel discrete learning based classification approach is proposed and implemented on a deep feed-forward neural network as one of the most commonly used deep learning models to develop a different learning process for deep classification models. The basis of the proposed learning approach is maximizing a discrete matching function of the actual and fitted values instead of minimizing a continuous distance-based cost function. The proposed classification approach is evaluated on five benchmark data sets in the chemistry field. The empirical results indicated the superiority of the proposed discrete deep learning approach over its classic continuous form. The results of this study demonstrate the important effect of discrete learning processes on the performances of deep learning classification models. Therefore, the proposed methodology can be a powerful alternative to common classification approaches to analyze chemical data in the chemometrics field.

Application of dense neural networks for manifold-based modeling of flame-wall interactions

Artifical neural networks (ANNs) are universal approximators capable of learning any correlation between arbitrary input data with corresponding outputs, which can also be exploited to represent a low-dimensional chemistry manifold in the field of combustion. In this work, a procedure is developed to simulate a premixed methane-air flame undergoing side-wall quenching utilizing an ANN chemistry manifold. In the investigated case, the flame characteristics are governed by two canonical problems: the adiabatic flame propagation in the core flow and the non-adiabatic flame- wall interaction governed by enthalpy losses to the wall. Similar to the tabulation of a Quenching Flamelet-Generated Manifold (QFM), the neural network is trained on a 1D head-on quenching flame database to learn the intrinsic chemistry manifold. The control parameters (i.e. the inputs) of the ANN are identified from thermo-chemical state variables by a sparse principal component analysis (PCA) without using prior knowledge about the flame physics. These input quantities are then transported in the coupled CFD solver and used for manifold access during simulation runtime. The chemical source terms are corrected at the manifold boundaries to ensure bounded- ness of the thermo-chemical state at all times. Finally, the ANN model is assessed by comparison to simulation results of the 2D side-wall quenching (SWQ) configuration with detailed chemistry and with a flamelet-based manifold (QFM).

MOSFET Physics-Based Compact Model Mass-Produced: An Artificial Neural Network Approach

The continued scaling-down of nanoscale semiconductor devices has made it very challenging to obtain analytic surface potential solutions from complex equations in physics, which is the fundamental purpose of the MOSFET compact model. In this work, we proposed a general framework to automatically derive analytical solutions for surface potential in MOSFET, by leveraging the universal approximation power of deep neural networks. Our framework incorporated a physical-relation-neural-network (PRNN) to learn side-by-side from a general-purpose numerical simulator in handling complex equations of mathematical physics, and then instilled the "knowledge'' from the simulation data into the neural network, so as to generate an accurate closed-form mapping between device parameters and surface potential. Inherently, the surface potential was able to reflect the numerical solution of a two-dimensional (2D) Poisson equation, surpassing the limits of traditional 1D Poisson equation solutions, thus better illustrating the physical characteristics of scaling devices. We obtained promising results in inferring the analytic surface potential of MOSFET, and in applying the derived potential function to the building of 130 nm MOSFET compact models and circuit simulation. Such an efficient framework with accurate prediction of device performances demonstrates its potential in device optimization and circuit design.

Stochastic configuration networks with chaotic maps and hierarchical learning strategy

Stochastic configuration networks (SCNs) have universal approximation capability and fast modeling properties, which have been successfully employed in large-scale data analytics. Based on SCNs, Stochastic configuration networks with block increments (BSC) use the node block increments mechanism to improve training speed but increase the complexity of the model. This paper presents a parallel configuration method (PCM), develops an extension of the original BSC with chaos theory and proposes stochastic configuration networks with chaotic maps (SCNCM), and establishes a hierarchical learning strategy (HLS) to enhance the compactness and construction speed of the model. Firstly, PCM randomly assigns the input weights w and biases b of hidden layer nodes by using uniform and normal distributions. In PCM, an iterative learning algorithm is intended to generate the scope control set and improve configuration efficiency. Secondly, the paper presents two kinds of stochastic configuration networks with chaotic maps, which are SCNCM-I and SCNCM-II. SCNCM-I adjusts block size by using multiple error values and chaotic maps to improve the training speed. Based on SCNCM-I, SCNCM-II utilizes node removal mechanism to enhance the compactness. Finally, HLS integrates with SCNCM-I, SCNCM-II, and the Harris-hawks optimization algorithm (HHO). The purpose of training is to enhance the training speed and compactness for three algorithms. The experiments are conducted on four benchmark data sets and an industrial application shows its effectiveness.

Piecewise integrable neural network: An interpretable chaos identification framework.

Artificial neural networks (ANNs) are an effective data-driven approach to model chaotic dynamics. Although ANNs are universal approximators that easily incorporate mathematical structure, physical information, and constraints, they are scarcely interpretable. Here, we develop a neural network framework in which the chaotic dynamics is reframed into piecewise models. The discontinuous formulation defines switching laws representative of the bifurcations mechanisms, recovering the system of differential equations and its primitive (or integral), which describe the chaotic regime.

Equivariant quantum graph circuits: constructions for universal approximation over graphs

We investigate quantum circuits for graph representation learning, and propose equivariant quantum graph circuits (EQGCs), as a class of parameterized quantum circuits with strong relational inductive bias for learning over graph-structured data. Conceptually, EQGCs serve as a unifying framework for quantum graph representation learning, allowing us to define several interesting subclasses subsuming existing proposals. In terms of the representation power, we prove that the subclasses of interest are universal approximators for functions over the bounded graph domain. This theoretical perspective on quantum graph machine learning methods opens many directions for further work, and could lead to models with capabilities beyond those of classical approaches. We also provide experimental evidence, and observe that the performance of EQGCs scales well with the depth of the model.

Distributed integrated sliding mode control via neural network and disturbance observer for heterogeneous vehicle systems with uncertainties

A distributed sliding mode control based on neural network and disturbance observer is investigated for heterogeneous vehicle systems in this paper. The vehicle systems in this paper consider both parameter uncertainty and disturbance. First, the radial basis function neural network is applied to estimate the parameter uncertainty owing to its universal approximation ability, and the disturbance observer is designed to compensate for the external disturbance. Compared with the existing adaptive methods, the disturbance observer can estimate the external disturbance directly and effectively. Then, constant time headway policy is employed to regulate the inter-vehicle distance and improve the string stability. Unlike most existing sliding mode control strategy for vehicle platoons, the string stability is achieved via employing sufficient conditions on the control parameters rather than employing coupled sliding mode control. Afterward, modified constant time headway policy is designed to reduce the effect of nonzero initial inter-vehicle spacing and improve the traffic density. Finally, the simulation results with constant time headway policy and modified constant time headway policy are provided to demonstrate the effectiveness and advantages of the proposed approaches.

Formula omitted]-adic statistical field theory and deep belief networks

In this work we initiate the study of the correspondence between p-adic statistical field theories (SFTs) and neural networks (NNs). In general quantum field theories over a p-adic spacetime can be formulated in a rigorous way. Nowadays these theories are considered just mathematical toy models for understanding the problems of the true theories. In this work we show these theories are deeply connected with the deep belief networks (DBNs). Hinton et al. constructed DBNs by stacking several restricted Boltzmann machines (RBMs). The purpose of this construction is to obtain a network with a hierarchical structure (a deep learning architecture). An RBM corresponds to a certain spin glass, we argue that a DBN should correspond to an ultrametric spin glass. A model of such a system can be easily constructed by using p-adic numbers. In our approach, a p-adic SFT corresponds to a p-adic continuous DBN, and a discretization of this theory corresponds to a p-adic discrete DBN. We show that these last machines are universal approximators. In the p-adic framework, the correspondence between SFTs and NNs is not fully developed. We point out several open problems.

Robust AISMC-neural network observer-based control of high-speed autonomous vehicles with unknown dynamics

Substantial challenges still exist in designing path-tracking control systems for autonomous vehicles, particularly at speed limits or under varying operating conditions. Such problems arise for various reasons, such as the nonlinear characteristics of vehicular components, system-component interactions, constraints on the states and control inputs, and more. This paper focuses on designing a robust adaptive control system for high-speed autonomous vehicles in case the system dynamics are unknown or unavailable. For this purpose, an intelligent NN-based estimation system’s universal approximation potential will be leveraged, coupled to an adaptive integral sliding mode controller (AISMC). Unlike previously reported studies, the present paper considers the entire dynamics of the autonomous vehicle unknown rather than solely a part of the system or external disturbances merely. The Lyapunov stability theorem is employed to guarantee the asymptotic stability of the developed framework and to obtain the adaptation laws. A critical maneuver explores the effectiveness and robustness of the suggested framework under severe disturbances, parametric uncertainties, and high speeds. The obtained results indicate that the developed framework holds the capacity to navigate the vehicle alongside the desired trajectory and outperforms other reported studies in the literature subject to various external disturbances.

Neural adaptive with impedance learning control for uncertain cooperative multiple robot manipulators

In this paper, an adaptive radial basis function neural network (RBFNN) control scheme is studied for the desired tracking of multiple robot manipulators with carrying a common object in joint-space. First, a non-zero time-varying parameter is introduced into the RBFNN, as well as a novel universal approximation of RBFNN with this non-zero parameter is introduced. Second, a switching control scheme based on this approximation property is designed. When the states of uncertainties terms leave the universal approximation domains, the suggested adaptive laws established by a sliding surface would pull them back, then the uncertain nonlinear functions are approximated by the RBFNNs in the domains. With this new structure of control technique, the limitation of a finite universal approximation can be broken. Third, the cooperative multiple manipulators can track the desired trajectory determined through impedance learning, which is used to modulate the control input in order to optimize the environment robot interaction. Finally, simulation results are shown to demonstrate the effectiveness in terms of expected tracking performance and position tracking errors constraints.