Universal Approximation Research Articles

Asymptotic properties of neural network sieve estimators

Neural networks have become one of the most popularly used methods in machine learning and artificial intelligence. Due to the universal approximation theorem, a neural network with one hidden layer can approximate any continuous function on compact support as long as the number of hidden units is sufficiently large. Statistically, a neural network can be classified into a nonlinear regression framework. However, if we consider it parametrically, due to the unidentifiability of the parameters, it is difficult to derive its asymptotic properties. Instead, we consider the estimation problem in a nonparametric regression framework and use the results from sieve estimation to establish the consistency, the rates of convergence and the asymptotic normality of the neural network estimators. We also illustrate the validity of the theories via simulations.

Design of restricted normalizing flow towards arbitrary stochastic policy with computational efficiency

This paper proposes a new design method for a stochastic control policy using a normalizing flow (NF). In reinforcement learning (RL), the policy is usually modeled as a distribution model with trainable parameters. When this parameterization has less expressiveness, it would fail to acquiring the optimal policy. A mixture model has capability of a universal approximation, but it with too much redundancy increases the computational cost, which can become a bottleneck when considering the use of real-time robot control. As another approach, NF, which is with additional parameters for invertible transformation from a simple stochastic model as a base, is expected to exert high expressiveness and lower computational cost. However, NF cannot compute its mean analytically due to complexity of the invertible transformation, and it lacks reliability because it retains stochastic behaviors after deployment for robot controller. This paper therefore designs a restricted NF (RNF) that achieves an analytic mean by appropriately restricting the invertible transformation. In addition, the expressiveness impaired by this restriction is regained using bimodal student-t distribution as its base, so-called Bit-RNF. In RL benchmarks, Bit-RNF policy outperformed the previous models. Finally, a real robot experiment demonstrated the applicability of Bit-RNF policy to real world.

Approximating triangulated categories by spaces

We initiate a systematic study of lattices of thick subcategories for arbitrary essentially small triangulated categories. To this end we give several examples illustrating the various properties these lattices may, or may not, have and show that as soon as a lattice of thick subcategories is distributive it is automatically a spatial frame. We then construct two non-commutative spectra, one functorial and one with restricted functoriality, that give universal approximations of these lattices by spaces.

Random vector functional link network: Recent developments, applications, and future directions

Neural networks have been successfully employed in various domains such as classification, regression and clustering, etc. Generally, the back propagation (BP) based iterative approaches are used to train the neural networks, however, it results in the issues of local minima, sensitivity to learning rate and slow convergence. To overcome these issues, randomization based neural networks such as random vector functional link (RVFL) network have been proposed. RVFL model has several characteristics such as fast training speed, direct links, simple architecture, and universal approximation capability, that make it a viable randomized neural network. This article presents the first comprehensive review of the evolution of RVFL model, which can serve as the extensive summary for the beginners as well as practitioners. We discuss the shallow RVFLs, ensemble RVFLs, deep RVFLs and ensemble deep RVFL models. The variations, improvements and applications of RVFL models are discussed in detail. Moreover, we discuss the different hyperparameter optimization techniques followed in the literature to improve the generalization performance of the RVFL model. Finally, we present potential future research directions/opportunities that can inspire the researchers to improve the RVFL’s architecture and learning algorithm further.

The universal approximation theorem for complex-valued neural networks

We generalize the classical universal approximation theorem for neural networks to the case of complex-valued neural networks. Precisely, we consider feedforward networks with a complex activation function σ:C→C in which each neuron performs the operation CN→C,z↦σ(b+wTz) with weights w∈CN and a bias b∈C, and with σ applied componentwise. We completely characterize those activation functions σ for which the associated complex networks have the universal approximation property, meaning that they can uniformly approximate any continuous function on any compact subset of Cd arbitrarily well. Unlike the classical case of real networks, the set of “good activation functions”—which give rise to networks with the universal approximation property—differs significantly depending on whether one considers deep networks or shallow networks: For deep networks with at least two hidden layers, the universal approximation property holds as long as σ is neither a polynomial, a holomorphic function, or an antiholomorphic function. Shallow networks, on the other hand, are universal if and only if the real part or the imaginary part of σ is not a polyharmonic function.

Universal Approximation Power of Deep Residual Neural Networks Through the Lens of Control

In this paper we show that deep residual neural networks have the power of universal approximation by using, in an essential manner, the observation that these networks can be modeled as nonlinear control systems. We first study the problem of using a deep residual neural network to exactly memorize training data by formulating it as a controllability problem for an ensemble control system. Using techniques from geometric control theory, we identify a class of activation functions that allow us to ensure controllability on an open and dense submanifold of sample points. Using this result, and resorting to the notion of monotonicity, we establish that any continuous function can be approximated on a compact set to arbitrary accuracy, with respect to the uniform norm, by this class of neural networks. Moreover, we provide optimal bounds on the number of required neurons.

Fast Finite-Time Control for Nonaffine Stochastic Nonlinear Systems Against Multiple Actuator Constraints via Output Feedback.

This research addresses the finite-time control problem for nonaffine stochastic nonlinear systems with actuator faults and input saturation. Specifically, a new finite-time control scheme is constructed based on the adaptive backstepping framework, with the usage of a state observer and taking advantage of the universal approximation capability of the fuzzy-logic system (FLS). The novelty of this work is that it considers the output feedback problem of a completely nonaffine stochastic system and incorporates the idea of the dynamic surface control (DSC) design. By using the Lyapunov stability theory, all the signals of the controlled system can be semiglobal finite-time stable in probability (SGFSP) while the system is imposed with multiple actuator constraints. In the meantime, the problem of "complexity explosion" is avoided. Two simulation examples are given to demonstrate the validity of the presented strategy.

An explicit-time and explicit-accuracy control for a state-constrained nonstrict-feedback uncertain system based on adaptive fuzzy dynamic-approximation

This article proposes a novel explicit-time and explicit-accuracy adaptive fuzzy control for a state-constrained nonlinear nonstrict-feedback uncertain system. This method can explicitly parameterize the upper bound of settling-time with low initial control input under a bounded initial condition. Meanwhile, this method can also explicitly parameterize the upper bound of accuracy while achieving low control input based on the adaptive fuzzy dynamic-approximation theorem. Firstly, a novel generalized explicit-time stability system is proposed by introducing the boundary gain term to render the time-parameter explicit, this method can solve the input conservatism problem caused by the unbounded-state gain term of traditional fixed/prdefined-time function. Then, according to the universal fuzzy approximation theorem, the novel dynamic relationship of adaptive fuzzy logic system between approximation error and adaptive parameters is presented. This relationship can lead to the adaptive fuzzy dynamic-approximation theorem, and an adaptive law designed by this theorem can realize the Lyapunov stability of adaptive control system under a Lasalle invariant set. In the end, a novel adaptive fuzzy control scheme is proposed by the generalized explicit-time function and adaptive fuzzy dynamic-approximation theorem. This scheme can achieve the explicit-time stability by the human-like activation function, and the accuracy can be parameterized by Lyapunov synthesis. Compared with other existing fixed/prdefined-time adaptive fuzzy control methods, the proposed explicit-time and explicit-accuracy controller achieves a significant reduction in the initial control input. Theoretical analysis and simulation results validate the effectiveness of the proposed method.

Designing universal causal deep learning models: The geometric (Hyper)transformer

AbstractSeveral problems in stochastic analysis are defined through their geometry, and preserving that geometric structure is essential to generating meaningful predictions. Nevertheless, how to design principled deep learning (DL) models capable of encoding these geometric structures remains largely unknown. We address this open problem by introducing a universal causal geometric DL framework in which the user specifies a suitable pair of metric spaces and and our framework returns a DL model capable of causally approximating any “regular” map sending time series in to time series in while respecting their forward flow of information throughout time. Suitable geometries on include various (adapted) Wasserstein spaces arising in optimal stopping problems, a variety of statistical manifolds describing the conditional distribution of continuous‐time finite state Markov chains, and all Fréchet spaces admitting a Schauder basis, for example, as in classical finance. Suitable spaces are compact subsets of any Euclidean space. Our results all quantitatively express the number of parameters needed for our DL model to achieve a given approximation error as a function of the target map's regularity and the geometric structure both of and of . Even when omitting any temporal structure, our universal approximation theorems are the first guarantees that Hölder functions, defined between such and can be approximated by DL models.

Learning CHARME models with neural networks

In this paper, we consider a model called CHARME (Conditional Heteroscedastic Autoregressive Mixture of Experts), a class of generalized mixture of nonlinear (non)parametric AR-ARCH time series. The main objective of this paper is to learn the autoregressive and volatility functions of this model with neural networks (NN). This approach is justified thanks to the universal approximation capacity of neural networks. On the other hand, in order to build the learning theory, it is necessary first to prove the ergodicity of the CHARME model. We therefore show in a general nonparametric framework that under certain Lipschitz-type conditions on the autoregressive and volatility functions, this model is stationary, ergodic and $$\tau $$ -weakly dependent. These conditions are much weaker than those in the existing literature. Moreover, this result forms the theoretical basis for deriving an asymptotic theory of the underlying parametric estimation, which we present for this model in a general parametric framework. Altogether, this allows to develop a learning theory for the NN-based autoregressive and volatility functions of the CHARME model, where strong consistency and asymptotic normality of the considered estimator of the NN weights and biases are guaranteed under weak conditions. Numerical experiments are reported to support our theoretical findings.

Basis operator network: A neural network-based model for learning nonlinear operators via neural basis

It is widely acknowledged that neural networks can approximate any continuous (even measurable) functions between finite-dimensional Euclidean spaces to arbitrary accuracy. Recently, the use of neural networks has started emerging in infinite-dimensional settings. Universal approximation theorems of operators guarantee that neural networks can learn mappings between infinite-dimensional spaces. In this paper, we propose a neural network-based method (BasisONet) capable of approximating mappings between function spaces. To reduce the dimension of an infinite-dimensional space, we propose a novel function autoencoder that can compress the function data. Our model can predict the output function at any resolution using the corresponding input data at any resolution once trained. Numerical experiments demonstrate that the performance of our model is competitive with existing methods on the benchmarks, and our model can address the data on a complex geometry with high precision. We further analyze some notable characteristics of our model based on the numerical results.

Interpretable polynomial neural ordinary differential equations.

Neural networks have the ability to serve as universal function approximators, but they are not interpretable and do not generalize well outside of their training region. Both of these issues are problematic when trying to apply standard neural ordinary differential equations (ODEs) to dynamical systems. We introduce the polynomial neural ODE, which is a deep polynomial neural network inside of the neural ODE framework. We demonstrate the capability of polynomial neural ODEs to predict outside of the training region, as well as to perform direct symbolic regression without using additional tools such as SINDy.

Orthogonal stochastic configuration networks with adaptive construction parameter for data analytics

As a randomized learner model, SCNs are remarkable that the random weights and biases are assigned employing a supervisory mechanism to ensure universal approximation and fast learning. However, the randomness makes SCNs more likely to generate approximate linear correlative nodes that are redundant and low quality, thereby resulting in non-compact network structure. In light of a fundamental principle in machine learning, that is, a model with fewer parameters holds improved generalization. This paper proposes orthogonal SCN, termed OSCN, to filtrate out the low-quality hidden nodes for network structure reduction by incorporating Gram–Schmidt orthogonalization technology. The universal approximation property of OSCN and an adaptive setting for the key construction parameters have been presented in details. In addition, an incremental updating scheme is developed to dynamically determine the output weights, contributing to improved computational efficiency. Finally, experimental results on two numerical examples and several real-world regression and classification datasets substantiate the effectiveness and feasibility of the proposed approach.

Physics informed deep neural network embedded in a chemical transport model for the Amazon rainforest

Secondary organic aerosols (SOA) are fine particles in the atmosphere, which interact with clouds, radiation and affect the Earth’s energy budget. SOA formation involves chemistry in gas phase, aqueous aerosols, and clouds. Simulating these chemical processes involve solving a stiff set of differential equations, which are computationally expensive steps for three-dimensional chemical transport models. Deep neural networks (DNNs) are universal function approximators that could be used to represent the complex nonlinear changes in aerosol physical and chemical processes; however, key challenges such as generalizability to extended time periods, preservation of mass balance, simulating sparse model outputs, and maintaining physical constraints have limited their use in atmospheric chemistry. Here, we develop an approach of using a physics-informed DNN that overcomes previous such challenges and demonstrates its applicability for the chemical formation processes of isoprene epoxydiol SOA (IEPOX-SOA) over the Amazon rainforest. The DNN is trained with data generated by simulating IEPOX-SOA over the entire atmospheric column, using the Weather Research and Forecasting Model coupled with Chemistry (WRF-Chem). The trained DNN is then embedded within WRF-Chem to replace the computationally expensive default solver of IEPOX-SOA formation. The trained DNN predictions generalizes well with the default model simulation of the IEPOX-SOA mass concentrations and its size distribution (20 size bins) over several days of simulations in both dry and wet seasons. The embedded DNN reduces the computational expense of WRF-Chem by a factor of 2. Our approach shows promise in terms of application to other computationally expensive chemistry solvers in climate models.

한의학에서 딥러닝의 뜻밖의 역할: 딥러닝의 과학으로 한의학 이해하기

Deep learning is revolutionizing in many scientific fields today. When it comes to Traditional Korean Medicine(TKM), it is commonly expected that deep learning will be able to assist TKM doctors to diagnose and prescribe treatments. We believe, however, there is another way to revolutionize TKM using deep learning. Mathematically, deep learning is a universal approximation function and can be a powerful model that explains cognitive processes in the brain. Since all of the decision-making processes in TKM are cognitive processes in the TKM doctors’ brain, they could also be modeled and explained using deep learning framework. As the science of deep learning advances, we will be able to better understand TKM through deep learning framework.

Neuroadaptive tracking control for uncertain pure‐feedback systems under dynamic constraints

AbstractThis paper presents a neuroadaptive tracking control method for a class of pure‐feedback nonlinear systems in the presence of dynamic constraints and unmodeled dynamics simultaneously. By introducing a nonlinear mapping (NM), the tracking control problem for constrained pure‐feedback system is recast into a regulation problem of the converted system without constraints. Such transformation allows the states to be confined within given regions directly, this is in contrast to the commonly used Barrier Lyapunov Function method that relies on the upper bound of the virtual control errors. To handle the unmodeled dynamics in the system, a dynamic compensation signal is introduced. It is shown that in the proposed scheme the neural networks (NN) not only act as a universal approximator to deal with unknown nonlinearity, but also function as a decoupler to cope with the coupling effects between state and the new variable arising from the introduction of the NM and the backstepping design. Simulation results also confirm the effectiveness of the proposed method.

Adaptive fuzzy dynamic surface control for nonstrict-feedback nonlinear state constrained systems with input dead-zone via event-triggered sampling

In this paper, a novel adaptive fuzzy dynamic surface control (DSC) via the event-triggered mechanism is developed for nonstrict-feedback nonlinear state constrained systems preceded by actuator dead-zone. The utilization of the dynamic surface technique avoids the issue of repeated differentiation of virtual control law and eliminates the drawback of the explosion of complexity with the backstepping technique. By utilizing the universal approximation property of fuzzy logic systems (FLSs), the unknown function is approximated successfully. By combining the DSC and event-triggered control (ETC) scheme, communication and computation strain are alleviated contemporaneously. An adaptive controller and an auxiliary system are cooperatively constructed to offset the effect of dead-zone nonlinear function. By employing the barrier Lyapunov functions (BLFs), all the states can remain within a prescribed interval. The tracking error can be restricted at the origin and all signals of closed-loop systems can be kept within the boundaries by the designed parameter adaptive laws and event-triggered controller. Finally, a numerical simulation result and a practical simulation result of the pendulum are proposed to guarantee the effectiveness of the presented control theory.

Optimal control of inventory level for perishable goods with uncertain decay factor and uncertain forecast information: a new robust MPC approach

This paper deals with the inventory control in supply chains under the following assumptions: (1) perishable goods with uncertain deteriorating factor, (2) a future uncertain customer demand that, over a limited prediction horizon, belongs to a known compact set. The problem is to define a smooth control policy maximising the fulfilled customer demand and minimising the inventory level. This problem is here solved through a new Robust Model Predictive Control (RMPC) approach. This implies solving a min–max optimisation problem with hard constraints on the control effort (i.e. the sequence of replenishment orders). To drastically reduce the numerical complexity of this problem, the control signal is sought in the space of B-spline functions, which are known to be universal approximators admitting a parsimonious parametric representation. This allows us: (1) to reduce the number of both decision variables and constraints involved in the optimisation procedure, (2) to reformulate the numerically demanding minimisation of the worst case cost functional as a simpler Weighted Constrained Robust Least Squares (WCRLS) estimation problem. The WCRLS algorithm can be efficiently solved using interior point methods. A rigorous analysis of stability and feasibility conditions is provided.

Deep Learning Discrete Calculus (DLDC): a family of discrete numerical methods by universal approximation for STEM education to frontier research

The article proposes formulating and codifying a set of applied numerical methods, coined as Deep Learning Discrete Calculus (DLDC), that uses the knowledge from discrete numerical methods to interpret the deep learning algorithms through the lens of applied mathematics. The DLDC methods aim to leverage the flexibility and ever-increasing resources of deep learning and rich literature on numerical analysis to formulate a general class of numerical methods that can directly use data with uncertainty to predict the behavior of an unknown system as well as elevate the speed and accuracy of numerical solution of the governing equations for known systems. The article is structured into two major sections. In the first section, the building blocks of the DLDC methods are presented and deep learning structures analogous to traditional numerical methods such as finite difference and finite element methods are constructed with a view to incorporate these techniques in Science, Technology, Engineering, Mathematics syllabus for K-12 students. The second section builds upon the building blocks of the previous discussion and proposes new solution schemes for differential and integral equations pertinent to multiscale mechanics. Each section is accompanied by a mathematical formulation of the numerical methods, analogous DLDC formulation, and suitable examples.

Greedy stochastic configuration networks for ill-posed problems

The inequality constraint in stochastic configuration networks (SCNs) is key to their universal approximation capability. SCNs incrementally add new units to neural networks under a supervisory mechanism. However, with an increasing number of hidden nodes, the output weights which are evaluated by the least squares method, are frequently unstable. This instability phenomenon (arbitrarily small perturbations leading to large changes in the solutions) can be classified as an ill-posed problem. Instability is one of the most important manifestations of ill-posed problems, and the Tikhonov regularization method, which is widely used to address ill-posed problems, is undesirable. This study examined the supervisory mechanism of SCNs and algebraic properties of the hidden output matrix, thereby proposing a new greedy stochastic configuration network (GSCN) for ill-posed problems. First, under the premise of ensuring universal approximation capability, GSCN tries to optimize the randomly assigned input weights and biases based on the hunter–prey optimization (HPO) algorithm. The optimized hidden parameters which can reduce residual error the most are selected for the newly added hidden nodes. Second, singular value decomposition (SVD) and orthogonal–triangular (QR) decomposition with column pivoting are introduced to extract a linearly independent subset of the hidden output matrix for ill-posed problems in SCNs. Finally, one function approximation, six benchmark regressions, two classification datasets, and bearing fault diagnosis are employed to evaluate the performance of GSCN. Experimental results indicate that GSCN shows superior performance with respect to fast convergence, generalization, and stability compared with other contrast methods. This code is available at https://github.com/Gzu-public-bigdata/GSCN.