Universal Approximation Research Articles

Insights on the different convergences in Extreme Learning Machine

Neural Networks (NN) are a powerful tool in approximation theory because of the existence of Universal Approximation (UA) results. In the last decades, a significant attention has been given to Extreme Learning Machines (ELMs), typically employed for the training of single layer NNs, and for which a UA result can also be proven. In a generic NN, the design of the optimal approximator can be recast as a non-convex optimization problem that turns out to be particularly demanding from the computational viewpoint. However, under the adoption of ELM, the optimization task reduces to a – possibly rectangular – linear problem. In this work, we detail how to design a sequence of ELM networks trained via a target dataset. Different convergence procedures are proposed and tested for some reference datasets constructed to be equivalent to approximation problems.

Stochastic configuration networks with improved supervisory mechanism

The stochastic configuration networks (SCNs) have been shown to have great potential for developing fast learning models, making them especially suitable for industrial devices. To achieve low consumption during the SCNs training process, this paper proposes an improved version of SCNs (ISCN-III). First, guided by the function dictionary set and the objective function space, a dynamic hyperparameter is proposed to replace the nonnegative sequence of the supervisory mechanism. Second, based on the dynamic hyperparameters and function dictionary set, an improved supervisory mechanism is proposed to further reduce computational consumption. Finally, the improved supervisory mechanism with the dynamic hyperparameter proves the universal approximation property of ISCN-III. ISCN-III is compared with SCNs and its improved versions using six benchmark datasets and subsequently applied to a grinding process and a continuous stirred tank reactor. The experimental results prove that ISCN-III has the advantages of low computational cost and fast convergence speed.

Combining finite-time control and prescribed tracking performance for uncertain PMSM driven steer-by-wire system with unknown disturbance

To solve the uncertainties in permanent magnet synchronous motor (PMSM) driven steer-by-wire (SbW) system, a prescribed-performance-control (PPC) based finite-time fuzzy controller is proposed. Specifically, the dynamics of SbW system driven by PMSM is analyzed, and combine PPC technology to transform the tracking error coordinates. Then, the velocity signal is obtained from the sigmoid function-based tracking differentiator by using the position signal. To solve the problem of mismatched disturbance, fuzzy logic system (FLS) are used as a universal approximator to estimate the lumped nonlinear friction and other unknown functions, the boundary of lumped disturbance is estimated by the designed adaptive updating law. By fusing the designed virtual controller and the intermediate control law, the controller needs no precise parameters of system, under which the effect of uncertainty of control gain can be compensated completely. Finally, rigorous theoretical analysis based on the Lyapunov stability theory is provided to demonstrate the finite-time stability of the closed-loop system under consideration. Simulation and experiment results are presented to show the effectiveness of the developed control approach, and some comparisons are given to show the rapid and accurate position tracking control performance.

Missing Data Imputation with Uncertainty-Driven Network

We study the problem of missing data imputation, which is a fundamental task in the area of data quality that aims to impute the missing data to achieve the completeness of datasets. Though the recent distribution-modeling-based techniques (e.g., distribution generation and distribution matching) can achieve state-of-the-art performance in terms of imputation accuracy, we notice that (1) they deploy a sophisticated deep learning model that tends to be overfitting for missing data imputation; (2) they directly rely on a global data distribution while overlooking the local information. Driven by the inherent variability in both missing data and missing mechanisms, in this paper, we explore the uncertain nature of this task and aim to address the limitations of existing works by proposing an u<u>N</u>certainty-driven netw<u>O</u>rk for <u>M</u>issing data <u>I</u>mputation, termed NOMI. NOMI has three key components, i.e., the retrieval module, the neural network gaussian process imputator (NNGPI) and the uncertainty-based calibration module. NOMI~ runs these components sequentially and in an iterative manner to achieve a better imputation performance. Specifically, in the retrieval module, NOMI~ retrieves local neighbors of the incomplete data samples based on the pre-defined similarity metric. Subsequently, we design NNGPI~ that merges the advantages of both the Gaussian Process and the universal approximation capacity of neural networks. NNGPI~ models the uncertainty by learning the posterior distribution over the data to impute missing values while alleviating the overfitting issue. Moreover, we further propose an uncertainty-based calibration module that utilizes the uncertainty of the imputator on its prediction to help the retrieval module obtain more reliable local information, thereby further enhancing the imputation performance. We also demonstrate that our NOMI~ can be reformulated as an instance of the well-known Expectation Maximization (EM) algorithm, highlighting the strong theoretical foundation of our proposed methods. Extensive experiments are conducted over 12 real-world datasets. The results demonstrate the excellent performance of NOMI in terms of both accuracy and efficiency.

Adaptive predefined-time tracking control of uncertain permanent-magnet synchronous motor drive chaotic system

A predefined-time tracking control algorithm is formulated for the Permanent Magnet Synchronous Motor(PMSM) driver system subjected to chaotic oscillations and undeterministic parameters, which can determine the upper bound of settling-time by controller gains. Radial Basis Function Neural Networks(RBFNNs) are employed to approximate the unknown nonlinear functions as universal approximators. The innovative predefined-time tracking differentiators are constructed to estimate the derivatives of the virtual inputs, resolving “the explosion of complexity” in traditional backstepping method. Then, an adaptive predefined-time controller is proposed in the backstepping framework. Lyapunov’s theoretical analysis proves that all signals in the closed-loop systems are uniformly ultimately bounded and the output tracking errors can converge to a small neighborhood of the origin within the predefined-time. Finally, simulation studies are performed to demonstrate that the proposed control scheme can quash the chaotic behaviors of the PMSM drive system and ensure fast-tracking performance even under unknown parameters.

Unifying O(3) equivariant neural networks design with tensor-network formalism

Many learning tasks, including learning potential energy surfaces from ab initio calculations, involve global spatial symmetries and permutational symmetry between atoms or general particles. Equivariant graph neural networks are a standard approach to such problems, with one of the most successful methods employing tensor products between various tensors that transform under the spatial group. However, as the number of different tensors and the complexity of relationships between them increase, maintaining parsimony and equivariance becomes increasingly challenging. In this paper, we propose using fusion diagrams, a technique widely employed in simulating SU(2)-symmetric quantum many-body problems, to design new spatial equivariant components for neural networks. This results in a diagrammatic approach to constructing novel neural network architectures. When applied to particles within a given local neighborhood, the resulting components, which we term ‘fusion blocks,’ serve as universal approximators of any continuous equivariant function defined on the neighborhood. We incorporate a fusion block into pre-existing equivariant architectures (Cormorant and MACE), leading to improved performance with fewer parameters on a range of challenging chemical problems. Furthermore, we apply group-equivariant neural networks to study non-adiabatic molecular dynamics of stilbene cis-trans isomerization. Our approach, which combines tensor networks with equivariant neural networks, suggests a potentially fruitful direction for designing more expressive equivariant neural networks.

Physics-Informed Neural Operator for Learning Partial Differential Equations

In this article, we propose physics-informed neural operators (PINO) that combine training data and physics constraints to learn the solution operator of a given family of parametric Partial Differential Equations (PDE). PINO is the first hybrid approach incorporating data and PDE constraints at different resolutions to learn the operator. Specifically, in PINO, we combine coarse-resolution training data with PDE constraints imposed at a higher resolution. The resulting PINO model can accurately approximate the ground-truth solution operator for many popular PDE families and shows no degradation in accuracy even under zero-shot super-resolution, that is, being able to predict beyond the resolution of training data. PINO uses the Fourier neural operator (FNO) framework that is guaranteed to be a universal approximator for any continuous operator and discretization convergent in the limit of mesh refinement. By adding PDE constraints to FNO at a higher resolution, we obtain a high-fidelity reconstruction of the ground-truth operator. Moreover, PINO succeeds in settings where no training data is available and only PDE constraints are imposed, while previous approaches, such as the Physics-Informed Neural Network (PINN), fail due to optimization challenges, for example, in multi-scale dynamic systems such as Kolmogorov flows.

Data-Driven Initial Gap Identification of Piecewise-Linear Systems Using Sparse Regression and Universal Approximation Theorem

Abstract This paper proposes a method for identifying an initial gap in piecewise-linear (PWL) systems from data. Piecewise-linear systems appear in many engineered systems such as degraded mechanical systems and infrastructures and are known to show strong nonlinearities. To analyze the behavior of such piecewise-linear systems, it is necessary to identify the initial gap, at which the system behavior switches. The proposed method identifies the initial gap by discovering the governing equations using sparse regression and calculating the gap based on the universal approximation theorem. A key step to achieve this is to approximate a piecewise-linear function by a finite sum of piecewise-linear functions in sparse regression. The equivalent gap is then calculated from the coefficients of the multiple piecewise-linear functions and their respective switching points in the obtained equation. The proposed method is first applied to a numerical model to confirm its applicability to piecewise-linear systems. Experimental validation of the proposed method has then been conducted with a simple mass-spring-hopping system, where the method successfully identifies the initial gap in the system with high accuracy.

A Novel Consolidation Analysis Framework: Universal Function Approximators Regularized by Physical Principles

Analytical and numerical techniques are widely used to analyse and interpret soil consolidation problems. An important limitation is the requirement for significant geotechnical knowledge and expertise to derive ‘true’ solutions. This study proposes an alternative: a universal function approximator regularized with known physical principles. The proposed approach here advances previous work to solve one-dimensional (1D) consolidation considering both self-weight and large strains, as well as two-dimensional (2D) consolidation by vertical drains. Initial and boundary conditions of the studied consolidation problems are first strongly and weakly enforced for comparison. A neural network is adopted here as the function approximator. To boost prediction accuracy, a novel strategy is proposed to adaptively sample data points for training. An estimate of epistemic uncertainty is achieved using the confidence interval of ensembled multiple outputs. The results show that the proposed approach accurately predicts the behaviours of complex consolidation processes. Results also indicate that regularization using weak physical constraints can alleviate the imbalance of back-propagated gradients of different loss terms and, in turn, achieve higher accuracy. The proposed method is generic, mesh-free, more robust and can be applied to a wide range of geotechnical problems.

Curvature perturbations from preheating with scale dependence

We extend the formalism to calculate non-Gaussianity of primordial curvature perturbations produced by preheating in the presence of a light scalar field. The calculation is carried out in the separate universe approximation using the non-perturbative delta N formalism and lattice field theory simulations. Initial conditions for simulations are drawn from a statistical ensemble determined by modes that left the horizon during inflation, with the time-dependence of Hubble rate during inflation taken into account. Our results show that cosmic variance, i.e., the contribution from modes with wavelength longer than the size of the observable universe today, plays a key role in determining the dominant contribution. We illustrate our formalism by applying it to an observationally-viable preheating model motivated by non-minimal coupling to gravity, and study its full parameter dependence.

Generalization of Direct Adaptive Control Using Fractional Calculus Applied to Nonlinear Systems

This paper presents a new direct adaptive control (DAC) technique using Caputo’s definition of the fractional-order derivative. This is the first time a fractional-order adaptive law is introduced to work together with an integer-order stable manifold for approximating the uncertainty of a class of nonlinear systems. The DAC approach uses universal function approximators such as multi-layer perceptrons with one hidden layer or fuzzy systems to approximate the controller. This paper presents a new lemma, which elucidates and clarifies the link between the Caputo and the Riemann–Liouville definitions. The introduced lemma is useful in developing a Lyapunov candidate to prove the stability of using the proposed fractional-order adaptive law. This is further explained by a numerical example, which is provided to elucidate the practicality of using the fractional-order derivative for updating the approximator parameters. The main novelty of the results in this paper is a rigorous stability proof of the fractional DAC approach for a class of nonlinear systems that is subjected to unstructured uncertainty and deals with the adaptation mechanism using a traditional integer-order stable manifold. This makes the control scheme easier to implement in practice. The fractional-order adaptation law provides greater degrees of freedom and a potentially larger functional control structure than the conventional adaptive control. Finally, the paper demonstrates that traditional integer-order DAC is a special case of the more general fractional-order DAC scheme introduced here.

Learning in Deep Radial Basis Function Networks.

Learning in neural networks with locally-tuned neuron models such as radial Basis Function (RBF) networks is often seen as instable, in particular when multi-layered architectures are used. Furthermore, universal approximation theorems for single-layered RBF networks are very well established; therefore, deeper architectures are theoretically not required. Consequently, RBFs are mostly used in a single-layered manner. However, deep neural networks have proven their effectiveness on many different tasks. In this paper, we show that deeper RBF architectures with multiple radial basis function layers can be designed together with efficient learning schemes. We introduce an initialization scheme for deep RBF networks based on k-means clustering and covariance estimation. We further show how to make use of convolutions to speed up the calculation of the Mahalanobis distance in a partially connected way, which is similar to the convolutional neural networks (CNNs). Finally, we evaluate our approach on image classification as well as speech emotion recognition tasks. Our results show that deep RBF networks perform very well, with comparable results to other deep neural network types, such as CNNs.

GRINN: a physics-informed neural network for solving hydrodynamic systems in the presence of self-gravity

Modeling self-gravitating gas flows is essential to answering many fundamental questions in astrophysics. This spans many topics including planet-forming disks, star-forming clouds, galaxy formation, and the development of large-scale structures in the Universe. However, the nonlinear interaction between gravity and fluid dynamics offers a formidable challenge to solving the resulting time-dependent partial differential equations (PDEs) in three dimensions (3D). By leveraging the universal approximation capabilities of a neural network within a mesh-free framework, physics informed neural networks (PINNs) offer a new way of addressing this challenge. We introduce the gravity-informed neural network (GRINN), a PINN-based code, to simulate 3D self-gravitating hydrodynamic systems. Here, we specifically study gravitational instability and wave propagation in an isothermal gas. Our results match a linear analytic solution to within 1% in the linear regime and a conventional grid code solution to within 5% as the disturbance grows into the nonlinear regime. We find that the computation time of the GRINN does not scale with the number of dimensions. This is in contrast to the scaling of the grid-based code for the hydrodynamic and self-gravity calculations as the number of dimensions is increased. Our results show that the GRINN computation time is longer than the grid code in one- and two- dimensional calculations but is an order of magnitude lesser than the grid code in 3D with similar accuracy. Physics-informed neural networks like GRINN thus show promise for advancing our ability to model 3D astrophysical flows.

Random vector functional link networks for function approximation on manifolds

The learning speed of feed-forward neural networks is notoriously slow and has presented a bottleneck in deep learning applications for several decades. For instance, gradient-based learning algorithms, which are used extensively to train neural networks, tend to work slowly when all of the network parameters must be iteratively tuned. To counter this, both researchers and practitioners have tried introducing randomness to reduce the learning requirement. Based on the original construction of Igelnik and Pao, single layer neural-networks with random input-to-hidden layer weights and biases have seen success in practice, but the necessary theoretical justification is lacking. In this study, we begin to fill this theoretical gap. We then extend this result to the non-asymptotic setting using a concentration inequality for Monte-Carlo integral approximations. We provide a (corrected) rigorous proof that the Igelnik and Pao construction is a universal approximator for continuous functions on compact domains, with approximation error squared decaying asymptotically like O(1/n) for the number n of network nodes. We then extend this result to the non-asymptotic setting, proving that one can achieve any desired approximation error with high probability provided n is sufficiently large. We further adapt this randomized neural network architecture to approximate functions on smooth, compact submanifolds of Euclidean space, providing theoretical guarantees in both the asymptotic and non-asymptotic forms. Finally, we illustrate our results on manifolds with numerical experiments.

Expressive power of ReLU and step networks under floating-point operations

The study of the expressive power of neural networks has investigated the fundamental limits of neural networks. Most existing results assume real-valued inputs and parameters as well as exact operations during the evaluation of neural networks. However, neural networks are typically executed on computers that can only represent a tiny subset of the reals and apply inexact operations, i.e., most existing results do not apply to neural networks used in practice. In this work, we analyze the expressive power of neural networks under a more realistic setup: when we use floating-point numbers and operations as in practice. Our first set of results assumes floating-point operations where the significand of a float is represented by finite bits but its exponent can take any integer value. Under this setup, we show that neural networks using a binary threshold unit or ReLU can memorize any finite input/output pairs and can approximate any continuous function within an arbitrary error. In particular, the number of parameters in our constructions for universal approximation and memorization coincides with that in classical results assuming exact mathematical operations. We also show similar results on memorization and universal approximation when floating-point operations use finite bits for both significand and exponent; these results are applicable to many popular floating-point formats such as those defined in the IEEE 754 standard (e.g., 32-bit single-precision format) and bfloat16.

An indirect training approach for implicit constitutive modelling using recurrent neural networks and the virtual fields method

Accurate material description is crucial to achieve high-quality results in computational analysis software. Phenomenological constitutive laws generalize the material behaviour observed in simple mechanical tests. The resulting empirical expressions contain parameters that need to be calibrated through an inverse optimization process. Advancements in Digital Image Correlation (DIC) techniques have enabled the extraction of non-uniform multi-axial displacement fields, facilitating the development of heterogeneous mechanical specimens and unlocking access to richer material behaviour data. Despite these advancements, constitutive models are still limited by mathematical formulations and biases due to simplifying assumptions. Artificial Neural Networks (ANNs), as universal function approximators, could potentially drive a paradigm shift in the field. ANNs do not require explicit pre-formulations, hence avoiding the need for identifying unknown parameters. Moreover, ANNs are able to implicitly learn patterns purely from data, allowing to circumvent the biases induced by the simplifying assumptions of first-order principles. However, the training of ANNs for implicit constitutive modelling is not straightforward, given the impossibility of obtaining direct measurements of stress. This paper explores the integration of Recurrent Neural Networks (RNNs) with the Virtual Fields Method (VFM) for material modelling. This approach uses global force and displacement data to indirectly train the neural network, with the equilibrium being evaluated globally through the VFM loss function. The proposed method is (i) computationally efficient by not requiring Finite Element Analysis (FEA), (ii) compatible with both full-field measurements and numerically generated data, and (iii) able to handle experimental boundary conditions.

Universal approximation property of stochastic configuration networks for time series

For the purpose of processing sequential data, such as time series, and addressing the challenge of manually tuning the architecture of traditional recurrent neural networks (RNNs), this paper introduces a novel approach-the Recurrent Stochastic Configuration Network (RSCN). This network is constructed based on the random incremental algorithm of stochastic configuration networks. Leveraging the foundational structure of recurrent neural networks, our learning model commences with a modest-scale recurrent neural network featuring a single hidden layer and a solitary hidden node. Subsequently, the node parameters of the hidden layer undergo incremental augmentation through a random configuration process, with corresponding weights assigned structurally. This iterative expansion continues until the network satisfies predefined termination criteria. Noteworthy is the adaptability of this algorithm to handle time series data, exhibiting superior performance compared to traditional recurrent neural networks with similar architectures. The experimental results presented in this paper underscore the efficacy of the proposed RSCN for sequence data processing, showcasing its advantages over conventional recurrent neural networks in the context of the performed experiments.

Neural networks taking probability distributions as input: A framework for analyzing exchangeable networks

In recent years, exchangeable network structures that take datasets as input have been widely used to obtain representations of various datasets. Although they perform well, analyzing exchangeable network with a dataset as input is challenging. Given that this type of network can be viewed as a function acting on probability measures since datasets are drawn from various distributions, this paper theoretically analyzes exchangeable network structures from a probabilistic perspective. This paper proposes a probabilistic analytical framework that neural networks acting on probability measures, which is an extension of Multi-Layer Perceptrons (MLP). When only samples from each distribution are available, in this new analytical framework, neural networks acting on probability measures correspond to the traditional exchangeable structure defined on datasets. Using this new analytical framework, we can demonstrate the ability of exchangeable network structures to capture complex patterns, as it provides the universal approximation property of exchangeable network structures. Furthermore, we derive a consistency result that shows the parameter estimation of exchangeable network structures is consistent statistically under certain conditions.

SHoP: A Deep Learning Framework for Solving High-Order Partial Differential Equations

Solving partial differential equations (PDEs) has been a fundamental problem in computational science and of wide applications for both scientific and engineering research. Due to its universal approximation property, neural network is widely used to approximate the solutions of PDEs. However, existing works are incapable of solving high-order PDEs due to insufficient calculation accuracy of higher-order derivatives, and the final network is a black box without explicit explanation. To address these issues, we propose a deep learning framework to solve high-order PDEs, named SHoP. Specifically, we derive the high-order derivative rule for neural network, to get the derivatives quickly and accurately; moreover, we expand the network into a Taylor series, providing an explicit solution for the PDEs. We conduct experimental validations four high-order PDEs with different dimensions, showing that we can solve high-order PDEs efficiently and accurately. The source code can be found at https://github.com/HarryPotterXTX/SHoP.git.

Abstract 2305: Label-free high dimensional single cell morphological profiling of different hematological malignancies by ghost cytometry

Abstract Characterizing the heterogeneity in human leukemias is critical for understanding mechanisms of disease onset, uncovering biomarkers for disease diagnosis, and guiding research into novel treatment approaches. While extensive research focus has been placed on characterizing the molecular heterogeneity in leukemia by modern omics technologies, comparatively less emphasis has been placed on studying the diversity in holistic morphological changes that occur in single cells in the context of disease. Here, we report on a novel approach to understanding and characterizing different hematological malignancies using ghost cytometry, a recently developed high-content flow cytometric method that leverages high-speed, artificial intelligence (AI) driven morphological characterization and analysis of single cells. We performed morphometric profiling on human bone marrow samples from patients with T and B-cell acute lymphoblastic leukemia (ALL, n=5), acute myeloid leukemia (AML, n=5), multiple myeloma (MM, n=5) and healthy bone marrow mononuclear cell (BM-MNC) controls (n=6). Morphological profiles from individual cells were analyzed by unsupervised machine learning (universal manifold approximation and projection, UMAP) and compared within disease subsets and against controls. We identified disease-specific cell populations from a mixture of healthy control and disease samples without any labeling, based on completely label-free information. For validation of the specific cell subsets, we overlaid known cell surface markers (e.g. CD45 for blast cells, CD10/CD19 for B-ALL, CD33/CD34 for AML, CD38/CD138 for MM). The intensity of CD45 was dimmer and disease-specific markers were highly expressed in the disease-specific cell population. We also identified a unique cell subpopulation in the disease-specific cell population that could not be distinguished by existing markers. We also analyzed three human myeloma cell lines (AMO1, KMS, and L363) for comparison to the primary MM samples. In the UMAP analysis using LF-GC data, in all three types of cell lines, Bortezomib-resistant cell lines exhibited different distributions from the control parental cell lines. In addition, we observed distinct morphological profiles between primary myeloma samples and myeloma cell lines. Here we present a novel cell characterization approach for human hematological diseases that leverages AI-based, label-free, high-speed morphological characterization of single cells. We demonstrate that the approach can be used to identify subtle morphological differences in blast cells from a range of blood cancer subtypes. Application of morphological profiling and AI can be used to identify measurable, disease-related changes in these diseases that have diagnostic, drug screening, and therapeutic monitoring potential. Citation Format: Asako Tsubouchi, Juho J. Miettinen, Keisuke Wagatsuma, Satoru Akai, Yuri An, Mika Uematsu, Philipp Sergeev, Dimitrios Tsallos, Markus J. Vähä-Koskela, Sadao Ota, Caroline A. Heckman. Label-free high dimensional single cell morphological profiling of different hematological malignancies by ghost cytometry [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 2305.