Deep ReLU Research Articles

A data-driven control method for ground locomotion on sloped terrain of a hybrid aerial-ground robot

In this work, we present a data-driven solution for the attitude control of DoubleBee on slopes. DoubleBee is a novel hybrid aerial-ground robot with two rotors and two active wheels. Inspired by the physcics modeling of the system, we add a channel-separated attention head to a deep ReLU neural network to predict disturbances from ground effects, motor torques and rotation axis shift. The proposed neural network is Lipschitz continuous, has fewer parameters and performs better for disturbance estimation than the baseline deep ReLU neural network. Then, we design a sliding mode controller using these predictions and establish its input-to-state stability and error bounds. Experiments show improvements of the proposed neural network in training speed and robustness over a baseline ReLU network, and a 40% reduction in tracking error compared to a baseline PID controller.

An error analysis for deep binary classification with sigmoid loss

Deep neural networks have demonstrated remarkable efficacy in diverse classification tasks. In this paper, we specifically focus on the predictive performance in deep binary classification problems with the sigmoid loss. Given that sigmoid loss is categorized as a non-convex and bounded loss function, it exhibits potential resilience against the disruptive impact of outlier noises. We first derive the convergence rate of the excess misclassification risk for deep ReLU neural networks with the sigmoid loss, a result that attains minimax optimality. To the best of our acknowledge, we are the first to derive the convergence rate for the sigmoid loss. Moreover, we extend our analysis to derive a faster convergence rate under margin assumptions. This achievement renders our findings comparable to those of commonly employed convex loss functions operating under analogous assumptions. Lastly, we undertake a comprehensive validation of the robustness inherent in the sigmoid loss across diverse datasets.

Magnitude and angle dynamics in training single ReLU neurons

Understanding the training dynamics of deep ReLU networks is a significant area of interest in deep learning. However, there remains a lack of complete elucidation regarding the weight vector dynamics, even for single ReLU neurons. To bridge this gap, our study delves into the training dynamics of the gradient flow w(t) for single ReLU neurons under the square loss, dissecting it into its magnitude ‖w(t)‖ and angle φ(t) components. Through this decomposition, we establish upper and lower bounds on these components to elucidate the convergence dynamics. Furthermore, we demonstrate the empirical extension of our findings to general two-layer multi-neuron networks. All theoretical results are generalized to the gradient descent method and rigorously verified through experiments.

Robust nonparametric regression based on deep ReLU neural networks

In this paper, we consider robust nonparametric regression using deep neural networks with ReLU activation function. While several existing theoretically justified methods are geared towards robustness against identical heavy-tailed noise distributions, the rise of adversarial attacks has emphasized the importance of safeguarding estimation procedures against systematic contamination. We approach this statistical issue by shifting our focus towards estimating conditional distributions. To address it robustly, we introduce a novel estimation procedure based on ℓ-estimation. Under a mild model assumption, we establish general non-asymptotic risk bounds for the resulting estimators, showcasing their robustness against contamination, outliers, and model misspecification. We then delve into the application of our approach using deep ReLU neural networks. When the model is well-specified and the regression function belongs to an α-Hölder class, employing ℓ-type estimation on suitable networks enables the resulting estimators to achieve the minimax optimal rate of convergence. Additionally, we demonstrate that deep ℓ-type estimators can circumvent the curse of dimensionality by assuming the regression function closely resembles the composition of several Hölder functions. To attain this, new deep fully-connected ReLU neural networks have been designed to approximate this composition class. This approximation result can be of independent interest.

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

The expressive power of deep neural networks is manifested by their remarkable ability to approximate multivariate functions in a way that appears to overcome the curse of dimensionality. This ability is exemplified by their success in solving high-dimensional problems where traditional numerical solvers fail due to their limitations in accurately representing high-dimensional structures. To provide a theoretical framework for explaining this phenomenon, we analyze the approximation of Hölder functions defined on a d-dimensional smooth manifold M embedded in RD, with d≪D, using deep neural networks. We prove that the uniform convergence estimates of the approximation and generalization errors by deep neural networks with ReLU activation functions do not depend on the ambient dimension D of the function but only on its lower manifold dimension d, in a precise sense. Our result improves existing results from the literature where approximation and generalization errors were shown to depend weakly on D.

Universal consistency of deep ReLU neural networks

Universal consistency of deep ReLU neural networks

On Centralization and Unitization of Batch Normalization for Deep ReLU Neural Networks

On Centralization and Unitization of Batch Normalization for Deep ReLU Neural Networks

Convergence of deep ReLU networks

We explore convergence of deep neural networks with the popular ReLU activation function, as the depth of the networks tends to infinity. To this end, we introduce the notion of activation domains and activation matrices of a ReLU network. By replacing applications of the ReLU activation function by multiplications with activation matrices on activation domains, we obtain an explicit expression of the ReLU network. We then identify the convergence of the ReLU networks as convergence of a class of infinite products of matrices. Sufficient and necessary conditions for convergence of these infinite products of matrices are studied. As a result, we establish necessary conditions for ReLU networks to converge that the sequence of weight matrices converges to the identity matrix and the sequence of the bias vectors converges to zero as the depth of ReLU networks increases to infinity. Moreover, we obtain sufficient conditions in terms of the weight matrices and bias vectors at hidden layers for pointwise convergence of deep ReLU networks. These results provide mathematical insights to convergence of deep neural networks. Experiments are conducted to mathematically verify the results and to illustrate their potential usefulness in initialization of deep neural networks.

How deep convolutional neural networks lose spatial information with training

A central question of machine learning is how deep nets manage to learn tasks in high dimensions. An appealing hypothesis is that they achieve this feat by building a representation of the data where information irrelevant to the task is lost. For image datasets, this view is supported by the observation that after (and not before) training, the neural representation becomes less and less sensitive to diffeomorphisms acting on images as the signal propagates through the network. This loss of sensitivity correlates with performance and surprisingly correlates with a gain of sensitivity to white noise acquired during training. Which are the mechanisms learned by convolutional neural networks (CNNs) responsible for the these phenomena? In particular, why is the sensitivity to noise heightened with training? Our approach consists of two steps. (1) Analyzing the layer-wise representations of trained CNNs, we disentangle the role of spatial pooling in contrast to channel pooling in decreasing their sensitivity to image diffeomorphisms while increasing their sensitivity to noise. (2) We introduce model scale-detection tasks, which qualitatively reproduce the phenomena reported in our empirical analysis. In these models we can assess quantitatively how spatial pooling affects these sensitivities. We find that the increased sensitivity to noise observed in deep ReLU networks is a mechanistic consequence of the perturbing noise piling up during spatial pooling, after being rectified by ReLU units. Using odd activation functions like tanh drastically reduces the CNNs’ sensitivity to noise.

Construction of Deep ReLU Nets for Spatially Sparse Learning.

Training an interpretable deep net to embody its theoretical advantages is difficult but extremely important in the community of machine learning. In this article, noticing the importance of spatial sparseness in signal and image processing, we develop a constructive approach to generate a deep net to capture the spatial sparseness feature. We conduct both theoretical analysis and numerical verifications to show the power of the constructive approach. Theoretically, we prove that the constructive approach can yield a deep net estimate that achieves the optimal generalization error bounds in the framework of learning theory. Numerically, we show that the constructive approach is essentially better than shallow learning in the sense that it provides better prediction accuracy with less training time.

Correction: Approximation of Nonlinear Functionals Using Deep ReLU Networks

Correction: Approximation of Nonlinear Functionals Using Deep ReLU Networks

Deep ReLU neural network approximation in Bochner spaces and applications to parametric PDEs

We investigate non-adaptive methods of deep ReLU neural network approximation in Bochner spaces L2(U∞,X,μ) of functions on U∞ taking values in a separable Hilbert space X, where U∞ is either R∞ equipped with the standard Gaussian probability measure, or [−1,1]∞ equipped with the Jacobi probability measure. Functions to be approximated are assumed to satisfy a certain weighted ℓ2-summability of the generalized chaos polynomial expansion coefficients with respect to the measure μ. We prove the convergence rate of this approximation in terms of the size of approximating deep ReLU neural networks. These results then are applied to approximation of the solution to parametric elliptic PDEs with random inputs for the lognormal and affine cases.

Collocation approximation by deep neural ReLU networks for parametric and stochastic PDEs with lognormal inputs

We find the convergence rates of the collocation approximation by deep ReLU neural networks of solutions to elliptic PDEs with lognormal inputs, parametrized by $\boldsymbol{y}$ in the noncompact set ${\mathbb R}^\infty$. The approximation error is measured in the norm of the Bochner space $L_2({\mathbb R}^\infty, V, \gamma)$, where $\gamma$ is the infinite tensor-product standard Gaussian probability measure on ${\mathbb R}^\infty$ and $V$ is the energy space. We also obtain similar dimension-independent results in the case when the lognormal inputs are parametrized by ${\mathbb R}^M$ of very large dimension $M$, and the approximation error is measured in the $\sqrt{g_M}$-weighted uniform norm of the Bochner space $L_\infty^{\sqrt{g}}({\mathbb R}^M, V)$, where $g_M$ is the density function of the standard Gaussian probability measure on ${\mathbb R}^M$. Bibliography: 62 titles.

Deep learning-based security situational awareness and detection technology for power networks in the context of big data

Abstract With the comprehensive promotion of “big data + energy”, new power network security threats are also more prominent, and the traditional security system mainly based on “protection” will face great challenges. Firstly, this paper proposes four kinds of network security situational awareness detection techniques based on distributed data analysis by combining the characteristics of big data in power networks. Secondly, the CRIT-LSTM power network security situational awareness model is constructed by improving its loss evaluation process using the cross entropy (CE) function and improving the LSTM unit structure using linear unit (ReLU). Finally, the performance of the three models is compared and analyzed under two aspects of neural network training and testing and various metrics to verify the models’ effectiveness. The results show that the improved CRIT-LSTM model based on deep learning, combining LSTM and ReLU algorithms, has an RMSE of 0.717 for the training set and 0.806 for the test set. 7.32% accuracy and 10.51% improvement in recall compared to the LSTM-only model. The power network security situational awareness model based on the CRIT-LSTM model proposed in this paper integrates various security system functions to maximize the defense against attacks and reduce unnecessary security risk losses.

Neural Network Approximation of Refinable Functions

In the desire to quantify the success of neural networks in deep learning and other applications, there is a great interest in understanding which functions are efficiently approximated by the outputs of neural networks. By now, there exists a variety of results which show that a wide range of functions can be approximated with sometimes surprising accuracy by these outputs. For example, it is known that the set of functions that can be approximated with exponential accuracy (in terms of the number of parameters used) includes, on one hand, very smooth functions such as polynomials and analytic functions and, on the other hand, very rough functions such as the Weierstrass function, which is nowhere differentiable. In this paper, we add to the latter class of rough functions by showing that it also includes refinable functions. Namely, we show that refinable functions are approximated by the outputs of deep ReLU neural networks with a fixed width and increasing depth with accuracy exponential in terms of their number of parameters. Our results apply to functions used in the standard construction of wavelets as well as to functions constructed via subdivision algorithms in Computer Aided Geometric Design.

Spline representation and redundancies of one-dimensional ReLU neural network models

We analyze the structure of a one-dimensional deep ReLU neural network (ReLU DNN) in comparison to the model of continuous piecewise linear (CPL) spline functions with arbitrary knots. In particular, we give a recursive algorithm to transfer the parameter set determining the ReLU DNN into the parameter set of a CPL spline function. Using this representation, we show that after removing the well-known parameter redundancies of the ReLU DNN, which are caused by the positive scaling property, all remaining parameters are independent. Moreover, we show that the ReLU DNN with one, two or three hidden layers can represent CPL spline functions with K arbitrarily prescribed knots (breakpoints), where K is the number of real parameters determining the normalized ReLU DNN (up to the output layer parameters). Our findings are useful to fix a priori conditions on the ReLU DNN to achieve an output with prescribed breakpoints and function values.

Wavelet transform based deep residual neural network and ReLU based Extreme Learning Machine for skin lesion classification

Skin cancer is one of the most widespread threats to human health worldwide. Therefore, early-stage recognition and detection of these diseases are crucial for patients' lives.Computer-aided methods can be used to solve this problem with high performance. We presented a wavelet transform-based deep residual neural network (WT-DRNNet) for skin lesion classification. In the proposed model, wavelet transformation, pooling, and normalization reveal more refined details and eliminate unwanted details from skin lesion images. Then, deep features are extracted with the residual neural network based on transfer learning as a feature extractor. Finally, these deep features were combined with the global average pooling approach, and the training phase was carried out using the Extreme Learning Machine based on the ReLu activation function. The ISIC2017 and HAM10000 datasets were used in the experimental works to test the proposed model's performance. The performance metrics of accuracy, specificity, precision, and F1-Score of the proposed model for the ISIC2017 dataset were 96.91%, 97.68%, 96.43%, and 95.79%, respectively, while these metrics for the HAM10000 dataset were 95.73%, 98.8%, 95.84%, and 93.44%, respectively. These results outperform the state-of-the-art to classify skin lesions. As a result, the proposed model can assist specialist physicians in automatically classifying cancer-based on skin lesion images.

Deep ReLU neural networks overcome the curse of dimensionality for partial integrodifferential equations

Deep neural networks (DNNs) with ReLU activation function are proved to be able to express viscosity solutions of linear partial integrodifferential equations (PIDEs) on state spaces of possibly high dimension d. Admissible PIDEs comprise Kolmogorov equations for high-dimensional diffusion, advection, and for pure jump Lévy processes. We prove for such PIDEs arising from a class of jump-diffusions on [Formula: see text], that for any suitable measure [Formula: see text] on [Formula: see text], there exist constants [Formula: see text] such that for every [Formula: see text] and for every [Formula: see text] the DNN [Formula: see text]-expression error of viscosity solutions of the PIDE is of size [Formula: see text] with DNN size bounded by [Formula: see text]. In particular, the constant [Formula: see text] is independent of [Formula: see text] and of [Formula: see text] and depends only on the coefficients in the PIDE and the measure used to quantify the error. This establishes that ReLU DNNs can break the curse of dimensionality (CoD for short) for viscosity solutions of linear, possibly degenerate PIDEs corresponding to suitable Markovian jump-diffusion processes. As a consequence of the employed techniques, we also obtain that expectations of a large class of path-dependent functionals of the underlying jump-diffusion processes can be expressed without the CoD.

Model-Centric Data Manifold: The Data Through the Eyes of the Model

We discover that deep ReLU neural network classifiers can see a low-dimensional Riemannian manifold structure on data. Such structure comes via the local data matrix, a variation of the Fisher information matrix, where the role of the model parameters is taken by the data variables. We obtain a foliation of the data domain and we show that the dataset on which the model is trained lies on a leaf, the data leaf, whose dimension is bounded by the number of classification labels. We validate our results with some experiments with the MNIST dataset: paths on the data leaf connect valid images, while other leaves cover noisy images.

Simultaneous neural network approximation for smooth functions

We establish in this work approximation results of deep neural networks for smooth functions measured in Sobolev norms, motivated by recent development of numerical solvers for partial differential equations using deep neural networks. Our approximation results are nonasymptotic in the sense that the error bounds are explicitly characterized in terms of both the width and depth of the networks simultaneously with all involved constants explicitly determined. Namely, for f∈Cs([0,1]d), we show that deep ReLU networks of width O(NlogN) and of depth O(LlogL) can achieve a nonasymptotic approximation rate of O(N−2(s−1)/dL−2(s−1)/d) with respect to the W1,p([0,1]d) norm for p∈[1,∞). If either the ReLU function or its square is applied as activation functions to construct deep neural networks of width O(NlogN) and of depth O(LlogL) to approximate f∈Cs([0,1]d), the approximation rate is O(N−2(s−n)/dL−2(s−n)/d) with respect to the Wn,p([0,1]d) norm for p∈[1,∞).