Gradient-based Learning Algorithms Research Articles

Random vector functional link networks for function approximation on manifolds

The learning speed of feed-forward neural networks is notoriously slow and has presented a bottleneck in deep learning applications for several decades. For instance, gradient-based learning algorithms, which are used extensively to train neural networks, tend to work slowly when all of the network parameters must be iteratively tuned. To counter this, both researchers and practitioners have tried introducing randomness to reduce the learning requirement. Based on the original construction of Igelnik and Pao, single layer neural-networks with random input-to-hidden layer weights and biases have seen success in practice, but the necessary theoretical justification is lacking. In this study, we begin to fill this theoretical gap. We then extend this result to the non-asymptotic setting using a concentration inequality for Monte-Carlo integral approximations. We provide a (corrected) rigorous proof that the Igelnik and Pao construction is a universal approximator for continuous functions on compact domains, with approximation error squared decaying asymptotically like O(1/n) for the number n of network nodes. We then extend this result to the non-asymptotic setting, proving that one can achieve any desired approximation error with high probability provided n is sufficiently large. We further adapt this randomized neural network architecture to approximate functions on smooth, compact submanifolds of Euclidean space, providing theoretical guarantees in both the asymptotic and non-asymptotic forms. Finally, we illustrate our results on manifolds with numerical experiments.

Forward layer-wise learning of convolutional neural networks through separation index maximizing

This paper proposes a forward layer-wise learning algorithm for CNNs in classification problems. The algorithm utilizes the Separation Index (SI) as a supervised complexity measure to evaluate and train each layer in a forward manner. The proposed method explains that gradually increasing the SI through layers reduces the input data’s uncertainties and disturbances, achieving a better feature space representation. Hence, by approximating the SI with a variant of local triplet loss at each layer, a gradient-based learning algorithm is suggested to maximize it. Inspired by the NGRAD (Neural Gradient Representation by Activity Differences) hypothesis, the proposed algorithm operates in a forward manner without explicit error information from the last layer. The algorithm’s performance is evaluated on image classification tasks using VGG16, VGG19, AlexNet, and LeNet architectures with CIFAR-10, CIFAR-100, Raabin-WBC, and Fashion-MNIST datasets. Additionally, the experiments are applied to text classification tasks using the DBPedia and AG’s News datasets. The results demonstrate that the proposed layer-wise learning algorithm outperforms state-of-the-art methods in accuracy and time complexity.

A hybrid complex spectral conjugate gradient learning algorithm for complex-valued data processing

Complex-valued neural networks (CVNNs) have become a powerful modelling tool for complex-valued data processing. Because most of the critical points of CVNNs are saddle points, the gradient-based learning algorithms for CVNNs enjoy more chances to reach the global minima while suffering from slow convergence. To this end, we propose a hybrid complex spectral conjugate gradient learning algorithm for fast training CVNNs in this paper. The proposed algorithm combines the scaled negative gradient with a Barzilai–Borwein stepsize and an optimized conjugate term to define a new training direction, thus providing an accurate approximation of the second-order curvature of the objective function. The complex Wolfe conditions are employed to adaptively determine the optimal training stepsize. Under mild conditions, the descent property of the training direction and the convergence of the proposed algorithm are theoretically established. Simulation results on a number of benchmark complex-valued data processing problems demonstrate the efficiency of the proposed algorithm.

Anytime Guarantees under Heavy-Tailed Data

Under data distributions which may be heavy-tailed, many stochastic gradient-based learning algorithms are driven by feedback queried at points with almost no performance guarantees on their own. Here we explore a modified "anytime online-to-batch" mechanism which for smooth objectives admits high-probability error bounds while requiring only lower-order moment bounds on the stochastic gradients. Using this conversion, we can derive a wide variety of "anytime robust" procedures, for which the task of performance analysis can be effectively reduced to regret control, meaning that existing regret bounds (for the bounded gradient case) can be robustified and leveraged in a straightforward manner. As a direct takeaway, we obtain an easily implemented stochastic gradient-based algorithm for which all queried points formally enjoy sub-Gaussian error bounds, and in practice show noteworthy gains on real-world data applications.

COV-ELM classifier: An extreme learning machine based identification of COVID-19 using chest X-ray images

Coronaviruses constitute a family of viruses that gives rise to respiratory diseases. COVID-19 is an infectious disease caused by a newly discovered coronavirus also termed Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). As COVID-19 is highly contagious, early diagnosis of COVID-19 is crucial for an effective treatment strategy. However, the reverse transcription-polymerase chain reaction (RT-PCR) test which is considered to be a gold standard in the diagnosis of COVID-19 suffers from a high false-negative rate. Therefore, the research community is exploring alternative diagnostic mechanisms. Chest X-ray (CXR) image analysis has emerged as a feasible and effective diagnostic technique towards this objective. In this work, we propose the COVID-19 classification problem as a three-class classification problem to distinguish between COVID-19, normal, and pneumonia classes. We propose a three-stage framework, named COV-ELM based on extreme learning machine (ELM). Our dataset comprises CXR images in a frontal view, namely Posteroanterior (PA) and Erect anteroposterior (AP). Stage one deals with preprocessing and transformation while stage two deals with feature extraction. These extracted features are passed as an input to the ELM at the third stage, resulting in the identification of COVID-19. The choice of ELM in this work has been motivated by its faster convergence, better generalization capability, and shorter training time in comparison to the conventional gradient-based learning algorithms. As bigger and diverse datasets become available, ELM can be quickly retrained as compared to its gradient-based competitor models. We use 10-fold cross-validation to evaluate the results of COV-ELM. The proposed model achieved a macro average F1-score of 0.95 and the overall sensitivity of 0.94 ± 0.02 at a 95% confidence interval. When compared to state-of-the-art machine learning algorithms, the COV-ELM is found to outperform its competitors in this three-class classification scenario. Further, LIME has been integrated with the proposed COV-ELM model to generate annotated CXR images. The annotations are based on the superpixels that have contributed to distinguish between the different classes. It was observed that the superpixels correspond to the regions of the human lungs that are clinically observed in COVID-19 and Pneumonia cases.

Development of two-phase logic-oriented fuzzy AND/OR network

The architecture of AND/OR fuzzy neural networks exhibits outstanding learning abilities and significant interpretation capabilities. However, AND/OR networks suffer from structure-related problems namely low efficiency and slow convergence of learning due to several reasons such as high dimensionality and gradient-based learning algorithms which lead to a visible computing overhead. In this paper, we present a two-phase fuzzy logic-oriented network design that is composed of AND/OR neurons. This design takes advantages of Randomized Neural Network (RNN) to achieve higher convergence while exhibiting good nonlinear approximation capabilities. A gradient–based learning algorithm is implemented in the second phase of the design to further reduce values of performance index. The quality of the proposed design and resulting architecture is quantified through the use of numeric data along with fuzzy sets (information granules). Experimental results meet the research’s objectives and the proposed design methodology opens up new future directions for proceeding with more improvements.

Spatial-Temporal Hybrid Neural Network With Computing-in-Memory Architecture

Deep learning (DL) has gained unprecedented success in many real-world applications. However, DL poses difficulties for efficient hardware implementation due to the needs of a complex gradient-based learning algorithm and the required high memory bandwidth for synaptic weight storage, especially in today's data-intensive environment. Computing-in-memory (CIM) strategies have emerged as an alternative for realizing energy-efficient neuromorphic applications in silicon, reducing resources and energy required for neural computations. In this work, we exploit a CIM-based spatial-temporal hybrid neural network (STHNN) with a unique learning algorithm. To be specific, we integrate both multilayer perceptron and recurrent-based delay-dynamical system, making the network becomes linear separable while processing information in both spatial and temporal domains, better yet, reducing the memory bandwidth and hardware overhead through the CIM architecture. The prototype fabricated in 180 nm CMOS process is built of fully-analog components, yielding an average on-chip classification accuracy up to 86.9% on handprinted alphabet characters with a power consumption of 33 mW. Beyond that, through the handwritten digit database and the radio frequency fingerprinting dataset, software-based numerical evaluations offer 1.6 -to- 9.8 × and 1.9 -to- 4.4 × speedup, respectively, without significantly degrading its classification accuracy compared to the cutting-edge DL approaches.

Event-based backpropagation can compute exact gradients for spiking neural networks

Spiking neural networks combine analog computation with event-based communication using discrete spikes. While the impressive advances of deep learning are enabled by training non-spiking artificial neural networks using the backpropagation algorithm, applying this algorithm to spiking networks was previously hindered by the existence of discrete spike events and discontinuities. For the first time, this work derives the backpropagation algorithm for a continuous-time spiking neural network and a general loss function by applying the adjoint method together with the proper partial derivative jumps, allowing for backpropagation through discrete spike events without approximations. This algorithm, EventProp, backpropagates errors at spike times in order to compute the exact gradient in an event-based, temporally and spatially sparse fashion. We use gradients computed via EventProp to train networks on the Yin-Yang and MNIST datasets using either a spike time or voltage based loss function and report competitive performance. Our work supports the rigorous study of gradient-based learning algorithms in spiking neural networks and provides insights toward their implementation in novel brain-inspired hardware.

Brain-Inspired Learning on Neuromorphic Substrates

Neuromorphic hardware strives to emulate brain-like neural networks and thus holds the promise for scalable, low-power information processing on temporal data streams. Yet, to solve real-world problems, these networks need to be trained. However, training on neuromorphic substrates creates significant challenges due to the offline character and the required nonlocal computations of gradient-based learning algorithms. This article provides a mathematical framework for the design of practical online learning algorithms for neuromorphic substrates. Specifically, we show a direct connection between real-time recurrent learning (RTRL), an online algorithm for computing gradients in conventional recurrent neural networks (RNNs), and biologically plausible learning rules for training spiking neural networks (SNNs). Furthermore, we motivate a sparse approximation based on block-diagonal Jacobians, which reduces the algorithm's computational complexity, diminishes the nonlocal information requirements, and empirically leads to good learning performance, thereby improving its applicability to neuromorphic substrates. In summary, our framework bridges the gap between synaptic plasticity and gradient-based approaches from deep learning and lays the foundations for powerful information processing on future neuromorphic hardware systems.

Artificial Neural Network Hysteresis Operators for the Identification of Hammerstein Hysteretic Systems

Abstract This paper introduces explicit neural representations of fundamental hysteresis operators such as the play and stop operators. The hysteresis neurons are represented by recurrent artificial neural networks (ANN) using classical activation functions and are trained with gradient-based learning algorithms. The hysteresis neurons are combined into a single ANN hysteretic layer which, from a mathematical point of view, is equivalent to a classical Prandtl-Ishlinskii model expression, but has all the advantages of ANNs. One such benefit is the flexibility it offers to be combined with other ANN layers to obtain various complex model structures. In this paper, this is illustrated by combining it with a linear recurrent neural network to obtain a Hammerstein model structure, which is often considered for the characterization of systems driven by a class of smart material-based actuators. The input nonlinearity in the Hammerstein model is represented by a Prandtl-Ishlinskii ANN hysteretic layer while the linear dynamics are captured using a linear recurrent ANN representing a single-input multiple-output linear time-invariant state-space model. The identification approach of the Hammerstein model with a hysteretic layer is illustrated both numerically and experimentally.

Convergence of a Gradient-Based Learning Algorithm With Penalty for Ridge Polynomial Neural Networks

Recently there have been renewed interests in high order neural networks (HONNs) for its powerful mapping capability. Ridge polynomial neural network (RPNN) is an important kind of HONNs, which always occupies a key position as an efficient instrument in the tasks of classification or regression. In order to make the convergence speed faster and the network generalization ability stronger, we introduce a regularization model for RPNN with Group Lasso penalty, which deals with the structural sparse problem at the group level in this paper. Nevertheless, there are two main obstacles for introducing the Group Lasso penalty, one is numerical oscillation and the other is convergence analysis challenge. In doing so, we adopt smoothing function to approximate the Group Lasso penalty to overcome these drawbacks. Meanwhile, strong and weak convergence theorems, and monotonicity theorems are provided for this novel algorithm. We also demonstrate the efficiency of our proposed algorithm by numerical experiments, and compare it to the no regularizer, $L_{2}$ regularizer, $L_{1/2}$ regularizer, smoothing $L_{1/2}$ regularizer, and the Group Lasso regularizer, and also the relevant theoretical analysis has been verified.

Gradient-Adaptive Spline-Interpolated LUT Methods for Low-Complexity Digital Predistortion

In this paper, new digital predistortion (DPD) solutions for power amplifier (PA) linearization are proposed, with particular emphasis on reduced processing complexity in future 5G and beyond wideband radio systems. The first proposed method, referred to as the spline-based Hammerstein (SPH) approach, builds on complex spline-interpolated lookup table (LUT) followed by a linear finite impulse response (FIR) filter. The second proposed method, the spline-based memory polynomial (SMP) approach, contains multiple parallel complex spline-interpolated LUTs together with an input delay line such that more versatile memory modeling can be achieved. For both structures, gradient-based learning algorithms are derived to efficiently estimate the LUT control points and other related DPD parameters. Large set of experimental results are provided, with specific focus on 5G New Radio (NR) systems, showing successful linearization of multiple PA samples as well as a 28 GHz active antenna array, incorporating channel bandwidths up to 200 MHz. Explicit performance-complexity comparisons are also reported between the SPH and SMP DPD systems and the widely-applied ordinary memory-polynomial (MP) DPD solution. The results show that the linearization capabilities of the proposed methods are very close to that of the ordinary MP DPD, particularly with the proposed SMP approach, while having substantially lower processing complexity.

Gradient-based Learning Methods Extended to Smooth Manifolds Applied to Automated Clustering

Grassmann manifold based sparse spectral clustering is a classification technique that consists in learning a latent representation of data, formed by a subspace basis, which is sparse. In order to learn a latent representation, spectral clustering is formulated in terms of a loss minimization problem over a smooth manifold known as Grassmannian. Such minimization problem cannot be tackled by one of traditional gradient-based learning algorithms, which are only suitable to perform optimization in absence of constraints among parameters. It is, therefore, necessary to develop specific optimization/learning algorithms that are able to look for a local minimum of a loss function under smooth constraints in an efficient way. Such need calls for manifold optimization methods. In this paper, we extend classical gradient-based learning algorithms on at parameter spaces (from classical gradient descent to adaptive momentum) to curved spaces (smooth manifolds) by means of tools from manifold calculus. We compare clustering performances of these methods and known methods from the scientific literature. The obtained results confirm that the proposed learning algorithms prove lighter in computational complexity than existing ones without detriment in clustering efficacy.

Control of the nonlinear building using an optimum inverse TSK model of MR damper based on modified grey wolf optimizer

This paper proposes an inverse model of the magnetorheological (MR) damper, based on a new Takagi-Sugeno-Kang (TSK) model to estimate the required voltage for producing the MR force. Usually, a MATLAB toolbox function, adaptive neuro-fuzzy inference systems (ANFIS), is used to train the TSK models, which uses the gradient-based learning algorithms to tune the weights or membership functions parameters. The main drawback is that, these algorithms most often find themselves trapped in local minima, depending on the initial estimations. To overcome this issue, a grey wolf optimizer (GWO) is selected and modified to achieve the training task and called the model as an optimum modified grey wolf-TSK model (OMGT). Also, the superiority of the modified grey wolf optimizer over its standard one is investigated using some mathematical benchmark test functions. Moreover, the linear quadratic regulator (LQR) controller is designed to estimate the optimal control force of an MR damper. The effectiveness of this optimum inverse model in structural control is illustrated and verified using an eight-story nonlinear benchmark building. The performance of the designed OMGT model is compared with the different control algorithms such on (PON), clipped optimal control (COC), active control, and ANFIS under different earthquakes, which demonstrate an acceptable performance of the OMGT over these control algorithms.

Options of Interest: Temporal Abstraction with Interest Functions

Temporal abstraction refers to the ability of an agent to use behaviours of controllers which act for a limited, variable amount of time. The options framework describes such behaviours as consisting of a subset of states in which they can initiate, an internal policy and a stochastic termination condition. However, much of the subsequent work on option discovery has ignored the initiation set, because of difficulty in learning it from data. We provide a generalization of initiation sets suitable for general function approximation, by defining an interest function associated with an option. We derive a gradient-based learning algorithm for interest functions, leading to a new interest-option-critic architecture. We investigate how interest functions can be leveraged to learn interpretable and reusable temporal abstractions. We demonstrate the efficacy of the proposed approach through quantitative and qualitative results, in both discrete and continuous environments.

On Gradient-Based Learning in Continuous Games

We formulate a general framework for competitive gradient-based learning that encompasses a wide breadth of multi-agent learning algorithms, and analyze the limiting behavior of competitive gradient-based learning algorithms using dynamical systems theory. For both general-sum and potential games, we characterize a non-negligible subset of the local Nash equilibria that will be avoided if each agent employs a gradient-based learning algorithm. We also shed light on the issue of convergence to non-Nash strategies in general- and zero-sum games, which may have no relevance to the underlying game, and arise solely due to the choice of algorithm. The existence and frequency of such strategies may explain some of the difficulties encountered when using gradient descent in zero-sum games as, e.g., in the training of generative adversarial networks. To reinforce the theoretical contributions, we provide empirical results that highlight the frequency of linear quadratic dynamic games (a benchmark for multi-agent reinforcement learning) that admit global Nash equilibria that are almost surely avoided by policy gradient.

Neurofuzzy-Based Plant-Wide Hierarchical Coordinating Optimization and Control: An Application to Zinc Hydrometallurgy Plant

In this paper, a novel three-layer coordinating optimization and control strategy involving fuzzy neural networks (FNNs) is proposed for industrial plant-wide control. This strategy includes a coordinator (top layer), local optimizers (middle layer), and local controllers (bottom layer). The coordinator determines and coordinates set-points of production indices for each process in the plant, aiming to achieve plant-wide optimal control. The local optimizers optimize individual unit and deliver the information to the coordinator. The controllers calculate optimal control inputs for the processes according to the set-points. Model predictive control (MPC) is used in this layer. Also used are FNNs with a gradient-based learning algorithm, which are trained to represent the models of the processes that are controlled by MPC. By constructing a Lyapunov function, convergence of the FNNs is established. Experiments are carried out in the largest zinc hydrometallurgy plant in China. The results demonstrate performance of our strategy in tracking the set-points and saving processing costs. Moreover, the strategy achieves a more optimal overall plant operation when compared to the same FNN-based MPC scheme that does not use the coordinator.

Differentiable learning of quantum circuit Born machines

Quantum circuit Born machines are generative models which represent the probability distribution of classical dataset as quantum pure states. Computational complexity considerations of the quantum sampling problem suggest that the quantum circuits exhibit stronger expressibility compared to classical neural networks. One can efficiently draw samples from the quantum circuits via projective measurements on qubits. However, similar to the leading implicit generative models in deep learning, such as the generative adversarial networks, the quantum circuits cannot provide the likelihood of the generated samples, which poses a challenge to the training. We devise an efficient gradient-based learning algorithm for the quantum circuit Born machine by minimizing the kerneled maximum mean discrepancy loss. We simulated generative modeling of the Bars-and-Stripes dataset and Gaussian mixture distributions using deep quantum circuits. Our experiments show the importance of circuit depth and gradient-based optimization algorithm. The proposed learning algorithm is runnable on near-term quantum device and can exhibit quantum advantages for generative modeling.

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

Based on a novel algorithm, known as the upper-layer-solution-aware (USA), a new algorithm, in which the penalty method is introduced into the empirical risk, is studied for training feed-forward neural networks in this paper, named as USA with penalty. Both theoretical analysis and numerical results show that it can control the magnitude of weights of the networks. Moreover, the deterministic theoretical analysis of the new algorithm is proved. The monotonicity of the empirical risk with penalty term is guaranteed in the training procedure. The weak and strong convergence results indicate that the gradient of the total error function with respect to weights tends to zero, and the weight sequence goes to a fixed point when the iterations approach positive infinity. Numerical experiment has been implemented and effectively verifies the proved theoretical results.

When Waiting Is Not an Option: Learning Options With a Deliberation Cost

Recent work has shown that temporally extended actions (options) can be learned fully end-to-end as opposed to being specified in advance. While the problem of how to learn options is increasingly well understood, the question of what good options should be has remained elusive. We formulate our answer to what good options should be in the bounded rationality framework (Simon, 1957) through the notion of deliberation cost. We then derive practical gradient-based learning algorithms to implement this objective. Our results in the Arcade Learning Environment (ALE) show increased performance and interpretability.