Universal Approximation Property Research Articles

Universal approximation property of stochastic configuration networks for time series

For the purpose of processing sequential data, such as time series, and addressing the challenge of manually tuning the architecture of traditional recurrent neural networks (RNNs), this paper introduces a novel approach-the Recurrent Stochastic Configuration Network (RSCN). This network is constructed based on the random incremental algorithm of stochastic configuration networks. Leveraging the foundational structure of recurrent neural networks, our learning model commences with a modest-scale recurrent neural network featuring a single hidden layer and a solitary hidden node. Subsequently, the node parameters of the hidden layer undergo incremental augmentation through a random configuration process, with corresponding weights assigned structurally. This iterative expansion continues until the network satisfies predefined termination criteria. Noteworthy is the adaptability of this algorithm to handle time series data, exhibiting superior performance compared to traditional recurrent neural networks with similar architectures. The experimental results presented in this paper underscore the efficacy of the proposed RSCN for sequence data processing, showcasing its advantages over conventional recurrent neural networks in the context of the performed experiments.

SHoP: A Deep Learning Framework for Solving High-Order Partial Differential Equations

Solving partial differential equations (PDEs) has been a fundamental problem in computational science and of wide applications for both scientific and engineering research. Due to its universal approximation property, neural network is widely used to approximate the solutions of PDEs. However, existing works are incapable of solving high-order PDEs due to insufficient calculation accuracy of higher-order derivatives, and the final network is a black box without explicit explanation. To address these issues, we propose a deep learning framework to solve high-order PDEs, named SHoP. Specifically, we derive the high-order derivative rule for neural network, to get the derivatives quickly and accurately; moreover, we expand the network into a Taylor series, providing an explicit solution for the PDEs. We conduct experimental validations four high-order PDEs with different dimensions, showing that we can solve high-order PDEs efficiently and accurately. The source code can be found at https://github.com/HarryPotterXTX/SHoP.git.

Neural networks taking probability distributions as input: A framework for analyzing exchangeable networks

In recent years, exchangeable network structures that take datasets as input have been widely used to obtain representations of various datasets. Although they perform well, analyzing exchangeable network with a dataset as input is challenging. Given that this type of network can be viewed as a function acting on probability measures since datasets are drawn from various distributions, this paper theoretically analyzes exchangeable network structures from a probabilistic perspective. This paper proposes a probabilistic analytical framework that neural networks acting on probability measures, which is an extension of Multi-Layer Perceptrons (MLP). When only samples from each distribution are available, in this new analytical framework, neural networks acting on probability measures correspond to the traditional exchangeable structure defined on datasets. Using this new analytical framework, we can demonstrate the ability of exchangeable network structures to capture complex patterns, as it provides the universal approximation property of exchangeable network structures. Furthermore, we derive a consistency result that shows the parameter estimation of exchangeable network structures is consistent statistically under certain conditions.

What Observables Are Needed for Precision Data-Enabled Learning of Inverse Operators?

Abstract The advent of easy access to large amount of data has sparked interest in directly developing the relationships between input and output of dynamic systems. A challenge is that in addition to the applied input and the measured output, the dynamics can also depend on hidden states that are not directly measured. In general, it is unclear what type of data, such as past input and or past output is needed, to learn inverse operators (that predict the input needed to track a desired output for control purposes) with a desired precision. The main contribution of this work is to show that, irrespective of the selected model, removing the hidden-state dependence and achieving a desired precision of inverse operators require (i) a sufficiently-long past history of the output and (ii) sufficiently-precise estimates of the output's instantaneous time derivatives that are necessary and sufficient for linear systems, and under some conditions, for nonlinear systems. This insight, about the required observables (output history and derivative) for removing the hidden-state dependence and achieving precision, is used to develop a data-enabled algorithm to learn the inverse operator for multi-input multi-output square systems. Simulation examples are used to illustrate that neural nets (with universal approximation property) can learn the inverse operator with sufficient precision only if the required observables, identified in this work, are included in training.

On the approximation of bi-Lipschitz maps by invertible neural networks

Invertible neural networks (INNs) represent an important class of deep neural network architectures that have been widely used in applications. The universal approximation properties of INNs have been established recently. However, the approximation rate of INNs is largely missing. In this work, we provide an analysis of the capacity of a class of coupling-based INNs to approximate bi-Lipschitz continuous mappings on a compact domain, and the result shows that it can well approximate both forward and inverse maps simultaneously. Furthermore, we develop an approach for approximating bi-Lipschitz maps on infinite-dimensional spaces that simultaneously approximate the forward and inverse maps, by combining model reduction with principal component analysis and INNs for approximating the reduced map, and we analyze the overall approximation error of the approach. Preliminary numerical results show the feasibility of the approach for approximating the solution operator for parameterized second-order elliptic problems.

Geometric constructive network with block increments for lightweight data-driven industrial process modeling

Industrial data-driven models may require frequent reconstruction to maintain model performance due to the dynamics, uncertainty, and complexity of industrial processes. The infrastructure of the industrial processes is usually distributed control systems (DCS) with energy-sensitive and resource-constrained. In this context, this article proposes a geometric constructive network with block increments (BI-GCN) to reduce the modeling consumption while achieving comparable accuracy. First, this article proposes a geometric control strategy with block increments, which is capable of adding multiple nodes to the BI-GCN simultaneously. Second, this article demonstrates the universal approximation property of BI-GCN, which in turn guarantees the potential high performance of BI-GCN for modeling tasks. Finally, experiments on benchmark datasets and the grinding process show that BI-GCN can effectively reduce the number of iterations in the modeling process while maintaining comparable accuracy.

Universal approximation properties for an ODENet and a ResNet: Mathematical analysis and numerical experiments

We prove a universal approximation property (UAP) for a class of ODENet and a class of ResNet, which are simplified mathematical models for deep learning systems with skip connections. The UAP can be stated as follows. Let $ n $ and $ m $ be the dimension of input and output data, and assume $ m\leq n $. Then we show that ODENet of width $ n+m $ with any non-polynomial continuous activation function can approximate any continuous function on a compact subset on $ \mathbb{R}^n $. We also show that ResNet has the same property as the depth tends to infinity. Furthermore, we derive the gradient of a loss function explicitly with respect to a certain tuning variable. We use this to construct a learning algorithm for ODENet. To demonstrate the usefulness of this algorithm, we apply it to a regression problem, a binary classification, and a multinomial classification in MNIST.

Adaptive Neural Cooperative Control of Multirobot Systems With Input Quantization.

This article develops the adaptive neural cooperative control scheme for a group of mobile robots with a limited sensing range in presence of input quantization by a dynamic surface control technique. First, to make the controller design feasible, the original robotic system is transformed into a new fully actuated system using a transverse function. Then, taking into consideration the effects of a hysteresis quantizer, an adaptive neural cooperative controller is developed based on the universal approximation property of the radial basis function neural networks and the connectivity preservation strategy. Furthermore, the proposed control scheme can guarantee that all closed-loop signals are semi-globally uniformly ultimately bounded. Meanwhile, desired constraints are not breached and tracking errors are within the predefined domains. Finally, several simulation results are carried out to testify the feasibility and efficiency of the theoretical findings revealed in this article.

Designing Universally-Approximating Deep Neural Networks: A First-Order Optimization Approach.

Universal approximation capability, also referred to as universality, is an important property of deep neural networks, endowing them with the potency to accurately represent the underlying target function in learning tasks. In practice, the architecture of deep neural networks largely influences the performance of the models. However, most existing methodologies for designing neural architectures, such as the heuristic manual design or neural architecture search, ignore the universal approximation property, thus losing a potential safeguard about the performance. In this paper, we propose a unified framework to design the architectures of deep neural networks with a universality guarantee based on first-order optimization algorithms, where the forward pass is interpreted as the updates of an optimization algorithm. The (explicit or implicit) network is designed by replacing each gradient term in the algorithm with a learnable module similar to a two-layer network or its derivatives Specifically, we explore the realm of width-bounded neural networks, a common practical scenario, showcasing their universality. Moreover, adding operations of normalization, downsampling, and upsampling does not hurt the universality. To the best of our knowledge, this is the first work that width-bounded networks with universal approximation guarantee can be designed in a principled way. Our framework can inspire a variety of neural architectures including some renowned structures such as ResNet and DenseNet, as well as novel innovations. The experimental results on image classification problems demonstrate that the newly inspired networks are competitive and surpass the baselines of ResNet, DenseNet, as well as the advanced ConvNeXt and ViT, testifying to the effectiveness of our framework.

Deep Convolutional Tables: Deep Learning Without Convolutions.

We propose a novel formulation of deep networks that do not use dot-product neurons and rely on a hierarchy of voting tables instead, denoted as convolutional tables (CTs), to enable accelerated CPU-based inference. Convolutional layers are the most time-consuming bottleneck in contemporary deep learning techniques, severely limiting their use in the Internet of Things and CPU-based devices. The proposed CT performs a fern operation at each image location: it encodes the location environment into a binary index and uses the index to retrieve the desired local output from a table. The results of multiple tables are combined to derive the final output. The computational complexity of a CT transformation is independent of the patch (filter) size and grows gracefully with the number of channels, outperforming comparable convolutional layers. It is shown to have a better capacity:compute ratio than dot-product neurons, and that deep CT networks exhibit a universal approximation property similar to neural networks. As the transformation involves computing discrete indices, we derive a soft relaxation and gradient-based approach for training the CT hierarchy. Deep CT networks have been experimentally shown to have accuracy comparable to that of CNNs of similar architectures. In the low-compute regime, they enable an error:speed tradeoff superior to alternative efficient CNN architectures.

A flexible likelihood-based neural network extension of the classic spatio-temporal model

The inclusion of the geographic information into regression models is becoming increasingly popular due to the increased availability of corresponding geo-referenced data. In this paper, a novel framework for combining spatio-temporal regression techniques and artificial neural network (ANN) regression models is presented. The key idea is to use the universal approximation property of the ANN function to account for an arbitrary spatial pattern in the dependent variable by including geographic coordinate variables as regressors. Moreover, the implicit location-specific effects are allowed to exhibit arbitrary interaction effects with other regressors such as a time variable. In contrast to other machine learning approaches for spatio-temporal data, the likelihood framework of the classic (linear) spatio-temporal regression model is preserved. This allows, inter alia, for inference regarding marginal effects and associated confidence. The framework also allows for non-normal conditional distributions, conditional spatial correlation, arbitrary trend and seasonality. These features are demonstrated in a simulation section and two data examples, using linear spatio-temporal models as a reference.

Approximation error of single hidden layer neural networks with fixed weights

Neural networks with finitely many fixed weights have the universal approximation property under certain conditions on compact subsets of the d-dimensional Euclidean space, where approximation process is considered. Such conditions were delineated in our paper [26]. But for many compact sets it is impossible to approximate multivariate functions with arbitrary precision and the question on estimation or efficient computation of approximation error arises. This paper provides an explicit formula for the approximation error of single hidden layer neural networks with two fixed weights.

Alternating Transfer Functions to Prevent Overfitting in Non-Linear Regression with Neural Networks

ABSTRACT In nonlinear regression with machine learning methods, neural networks (NNs) are ideally suited due to their universal approximation property, which states that arbitrary nonlinear functions can thereby be approximated arbitrarily well. Unfortunately, this property also poses the problem that data points with measurement errors can be approximated too well and unknown parameter subspaces in the estimation can deviate far from the actual value (so-called overfitting). Various developed methods aim to reduce overfitting through modifications in several areas of the training. In this work, we pursue the question of how an NN behaves in training with respect to overfitting when linear and nonlinear transfer functions (TF) are alternated in different hidden layers (HL). The presented approach is applied to a generated dataset and contrasted to established methods from the literature, both individually and in combination. Comparable results are obtained, whereby the common use of purely nonlinear transfer functions proves to be not recommended generally.

A unified and constructive framework for the universality of neural networks

Abstract One of the reasons why many neural networks are capable of replicating complicated tasks or functions is their universal approximation property. Though the past few decades have seen tremendous advances in theories of neural networks, a single constructive and elementary framework for neural network universality remains unavailable. This paper is an effort to provide a unified and constructive framework for the universality of a large class of activation functions including most of the existing ones. At the heart of the framework is the concept of neural network approximate identity (nAI). The main result is as follows: any nAI activation function is universal in the space of continuous functions on compacta. It turns out that most of the existing activation functions are nAI, and thus universal. The framework induces several advantages over the contemporary counterparts. First, it is constructive with elementary means from functional analysis, probability theory, and numerical analysis. Second, it is one of the first unified and constructive attempts that is valid for most of the existing activation functions. Third, it provides new proofs for most activation functions. Fourth, for a given activation and error tolerance, the framework provides precisely the architecture of the corresponding one-hidden neural network with a predetermined number of neurons and the values of weights/biases. Fifth, the framework allows us to abstractly present the first universal approximation with a favorable non-asymptotic rate. Sixth, our framework also provides insights into the developments, and hence providing constructive derivations, of some of the existing approaches.

Adaptive control of a class of uncertain nonlinear systems using brain emotional learning and Legendre polynomials

In this paper, an adaptive controller for a class of uncertain nonlinear systems is presented using a combination of Legendre polynomials and brain emotional learning-based intelligent controller (BELBIC). Recently, some versions of BELBIC have been presented with the aim of satisfying the universal approximation property using Gaussian basis function. However, the size of regressor vector is too large that imposes a heavy computational load to the processor. The novelty of this paper is presenting a new version of BELBIC with less computational burden using Legendre polynomials. Moreover, there are very few tuning parameters in Legendre polynomials. Another contribution of this paper is editing the stability analysis presented in recent related works. Due to the intrinsic non-differentiability of the adaptation rules of BELBIC, the second time derivative of Lyapunov function is undefined and thus, the Barbalat’s lemma cannot be applied to verify the asymptotic convergence of the error function. Therefore, bounded-input-bounded-output (BIBO) stability can only be claimed for this controller. Simulation results on different case studies show that Legendre polynomials can improve the universal approximation property of BELBIC with less tuning parameters. Moreover, in the absence of the robust control term in the control law, the performance Legendre polynomials will not deteriorate, while the performance degrade in Gaussian basis function is quite considerable.

An extreme learning machine-based method for computational PDEs in higher dimensions

We present two effective methods for solving high-dimensional partial differential equations (PDE) based on randomized neural networks. Motivated by the universal approximation property of this type of networks, both methods extend the extreme learning machine (ELM) approach from low to high dimensions. With the first method the unknown solution field in d dimensions is represented by a randomized feed-forward neural network, in which the hidden-layer parameters are randomly assigned and fixed while the output-layer parameters are trained. The PDE and the boundary/initial conditions, as well as the continuity conditions (for the local variant of the method), are enforced on a set of random interior/boundary collocation points. The resultant linear or nonlinear algebraic system, through its least squares solution, provides the trained values for the network parameters. With the second method the high-dimensional PDE problem is reformulated through a constrained expression based on an Approximate variant of the Theory of Functional Connections (A-TFC), which avoids the exponential growth in the number of terms of TFC as the dimension increases. The free field function in the A-TFC constrained expression is represented by a randomized neural network and is trained by a procedure analogous to the first method. We present ample numerical simulations for a number of high-dimensional linear/nonlinear stationary/dynamic PDEs to demonstrate their performance. These methods can produce accurate solutions to high-dimensional PDEs, in particular with their errors reaching levels not far from the machine accuracy for relatively lower dimensions. Compared with the physics-informed neural network (PINN) method, the current method is both cost-effective and more accurate for high-dimensional PDEs.

Universal approximation property of a continuous neural network based on a nonlinear diffusion equation

Recently, differential equation-based neural networks have been actively studied. This paper discusses the universal approximation property of a neural network that is based on a nonlinear partial differential equation (PDE) of the parabolic type.Based on the assumption that the activation function is non-polynomial and Lipschitz continuous, and applying the theory of the difference method, we show that an arbitrary continuous function on any compact set can be approximated using the output of the network with arbitrary precision. Additionally, we present an estimate of the order of accuracy with respect to △t and △x.

On the universal approximation property of radial basis function neural networks

On the universal approximation property of radial basis function neural networks

Mesh-Informed Neural Networks for Operator Learning in Finite Element Spaces

Thanks to their universal approximation properties and new efficient training strategies, Deep Neural Networks are becoming a valuable tool for the approximation of mathematical operators. In the present work, we introduce Mesh-Informed Neural Networks (MINNs), a class of architectures specifically tailored to handle mesh based functional data, and thus of particular interest for reduced order modeling of parametrized Partial Differential Equations (PDEs). The driving idea behind MINNs is to embed hidden layers into discrete functional spaces of increasing complexity, obtained through a sequence of meshes defined over the underlying spatial domain. The approach leads to a natural pruning strategy which enables the design of sparse architectures that are able to learn general nonlinear operators. We assess this strategy through an extensive set of numerical experiments, ranging from nonlocal operators to nonlinear diffusion PDEs, where MINNs are compared against more traditional architectures, such as classical fully connected Deep Neural Networks, but also more recent ones, such as DeepONets and Fourier Neural Operators. Our results show that MINNs can handle functional data defined on general domains of any shape, while ensuring reduced training times, lower computational costs, and better generalization capabilities, thus making MINNs very well-suited for demanding applications such as Reduced Order Modeling and Uncertainty Quantification for PDEs.

Model-free adaptive backstepping control for a class of uncertain nonlinear systems

Most nonlinear dynamic systems are characterized by uncertainties in models and parameters. Deterministic models cannot account for these uncertainties; therefore, model-based control using such models cannot provide the required performance. It is crucial to establish a practical concept of model-free control as a powerful alternative to model-based control. This paper develops a model-free adaptive backstepping control (MFABC) based on type 2 fuzzy Petri nets for a class of uncertain nonlinear systems. To provide valuable robustness to the MFABC structure, we have exploited the universal approximation property of type 2 fuzzy Petri nets to approximate the different nonlinear functions of the uncertain nonlinear system. The parameter adaptive laws are designed by the Lyapunov function; the stability and error convergence can be guaranteed. The simulation tests show that the proposed MFABC can provide good performance and high accuracy compared with the backstepping control. Moreover, the stability of this control scheme is affirmed.