Standard Stochastic Gradient Research Articles

Device Specifications for Neural Network Training with Analog Resistive Cross‐Point Arrays Using Tiki‐Taka Algorithms

Recently, specialized training algorithms for analog cross‐point array‐based neural network accelerators have been introduced to counteract device non‐idealities such as update asymmetry and cycle‐to‐cycle variation, achieving software‐level performance in neural network training. However, a quantitative analysis of how these algorithms affect the relaxation of device specifications is yet to be conducted. This study provides a detailed analysis by elucidating the device prerequisites for training with the Tiki‐Taka algorithm versions 1 (TTv1) and 2 (TTv2), which leverage the dynamics between multiple arrays to compensate for device non‐idealities. A multiparameter simulation is conducted to assess the impact of device non‐idealities, including asymmetry, retention, number of pulses, and cycle‐to‐cycle variation, on neural network training. Using pattern‐recognition accuracy as a performance metric, the required device specifications for each algorithm are revealed. The results demonstrate that the standard stochastic gradient descent algorithm requires stringent device specifications. Conversely, TTv2 permits more lenient device specifications than the TTv1 across all examined non‐idealities. The analysis provides guidelines for the development, optimization, and utilization of devices for high‐performance neural network training using Tiki‐Taka algorithms.

Variance-Reduced Accelerated First-Order Methods: Central Limit Theorems and Confidence Statements

In this paper, we consider a strongly convex stochastic optimization problem and propose three classes of variable sample-size stochastic first-order methods: (i) the standard stochastic gradient descent method, (ii) its accelerated variant, and (iii) the stochastic heavy-ball method. In each scheme, the exact gradients are approximated by averaging across an increasing batch size of sampled gradients. We prove that when the sample size increases at a geometric rate, the generated estimates converge in mean to the optimal solution at an analogous geometric rate for schemes (i)–(iii). Based on this result, we provide central limit statements, whereby it is shown that the rescaled estimation errors converge in distribution to a normal distribution with the associated covariance matrix dependent on the Hessian matrix, the covariance of the gradient noise, and the step length. If the sample size increases at a polynomial rate, we show that the estimation errors decay at a corresponding polynomial rate and establish the associated central limit theorems (CLTs). Under certain conditions, we discuss how both the algorithms and the associated limit theorems may be extended to constrained and nonsmooth regimes. Finally, we provide an avenue to construct confidence regions for the optimal solution based on the established CLTs and test the theoretical findings on a stochastic parameter estimation problem. Funding: This work was supported by the Office of Naval Research [Grant N00014-22-1-2589] and Basic Energy Sciences [Grant DE-SC0023303]. The work of J. Lei was partially supported by the National Natural Science Foundation of China [Grants 72271187 and 62088101]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/moor.2021.0068 .

The continuous stochastic gradient method: part II–application and numerics

In this contribution, we present a numerical analysis of the continuous stochastic gradient (CSG) method, including applications from topology optimization and convergence rates. In contrast to standard stochastic gradient optimization schemes, CSG does not discard old gradient samples from previous iterations. Instead, design dependent integration weights are calculated to form a convex combination as an approximation to the true gradient at the current design. As the approximation error vanishes in the course of the iterations, CSG represents a hybrid approach, starting off like a purely stochastic method and behaving like a full gradient scheme in the limit. In this work, the efficiency of CSG is demonstrated for practically relevant applications from topology optimization. These settings are characterized by both, a large number of optimization variables and an objective function, whose evaluation requires the numerical computation of multiple integrals concatenated in a nonlinear fashion. Such problems could not be solved by any existing optimization method before. Lastly, with regards to convergence rates, first estimates are provided and confirmed with the help of numerical experiments.

Pengembangan Stochastic Gradient Descent dengan Penambahan Variabel Tetap

Stochastic Gradient Descent (SGD) is one of the commonly used optimizers in deep learning. Therefore, in this work, we modify stochastic gradient descent (SGD) by adding a fixed variable. We will then look at the differences between standard stochastic gradient descent (SGD) and stochastic gradient descent (SGD) with additional variables. The phases performed in this study were: (1) optimization analysis, (2) fix design, (3) fix implementation, (4) fix test, (5) reporting. The results of this study aim to show the additional impact of fixed variables on the performance of stochastic gradient descent (SGD).

Coherent optical neuron control based on reinforcement learning.

Optical neural networks take optical neurons as the cornerstone to achieve complex functions. The coherent optical neuron has become one of the mainstream implementations because it can effectively perform natural and even complex number calculations. However, its state variability and requirement for reliability and effectiveness render traditional control methods no longer applicable. In this Letter, deep reinforcement coherent optical neuron control (DRCON) is proposed, and its effectiveness is experimentally demonstrated. Compared with the standard stochastic gradient descent, the average convergence rate of DRCON is 33% faster, while the effective number of bits increases from less than 2 bits to 5.5 bits. DRCON is a promising first step for large-scale optical neural network control.

Modelling continual learning in humans with Hebbian context gating and exponentially decaying task signals.

Humans can learn several tasks in succession with minimal mutual interference but perform more poorly when trained on multiple tasks at once. The opposite is true for standard deep neural networks. Here, we propose novel computational constraints for artificial neural networks, inspired by earlier work on gating in the primate prefrontal cortex, that capture the cost of interleaved training and allow the network to learn two tasks in sequence without forgetting. We augment standard stochastic gradient descent with two algorithmic motifs, so-called "sluggish" task units and a Hebbian training step that strengthens connections between task units and hidden units that encode task-relevant information. We found that the "sluggish" units introduce a switch-cost during training, which biases representations under interleaved training towards a joint representation that ignores the contextual cue, while the Hebbian step promotes the formation of a gating scheme from task units to the hidden layer that produces orthogonal representations which are perfectly guarded against interference. Validating the model on previously published human behavioural data revealed that it matches performance of participants who had been trained on blocked or interleaved curricula, and that these performance differences were driven by misestimation of the true category boundary.

A hybrid artificial bee colony algorithmic approach for classification using neural networks

Artificial neural networks are an integral component of most corporate and research functions across different platforms. However, depending upon the nature of the problem and quality of initialisation values, the usage of standard stochastic gradient descent always risks the possibility of getting trapped in local minima and saddle points for smaller neural networks in particular. One way to overcome this is by using algorithms with proven global search capabilities to train the network. This allows the neural net to reach the optimum values for weights regardless of the initialisation parameters used during training. Two algorithms are proposed based on modifications to the original artificial bee colony algorithm and their performances are analysed extensively on three benchmark datasets of increasing complexity. The first (NMABC), employs neural network appropriate initialisation and linear search space expansion. This is integrated into the second (LHABC), and incorporates stochastic gradient descent into the employed phase of the bees for faster convergence. It is found that the proposed algorithms consistently outperform standard approaches in all cases.

Natural Evolutionary Gradient Descent Strategy for Variational Quantum Algorithms

Recent research has demonstrated that parametric quantum circuits (PQCs) are affected by gradients that progressively vanish to zero as a function of the number of qubits. We show that using a combination of gradient-free natural evolutionary strategy and gradient descent can mitigate the possibility of optimizing barren plateaus in the landscape. We implemented 2 specific methods: natural evolutionary strategy stochastic gradient descent (NESSGD) and natural evolutionary strategy adapting the step size according to belief in observed gradients (NESAdaBelief) to optimize PQC parameter values. They were compared with standard stochastic gradient descent, adaptive moment estimation, and a version of adaptive moment estimation adapting the step size according to belief in observed gradients in 5 classification tasks. NESSGD and NESAdaBelief demonstrated some superiority in 4 of the tasks. NESAdaBelief showed higher accuracy than AdaBelief in all 5 tasks. In addition, we investigated the applicability of NESSGD under the parameter shift rule and demonstrated that NESSGD can adapt to this rule, which means that our proposed method could also optimize the parameters of PQCs on quantum computers.

Differentially private stochastic gradient descent via compression and memorization

We propose a novel approach for achieving differential privacy for neural network training models through compression and memorization of gradients. The compression technique, which makes gradient vectors sparse, reduces the sensitivity so that differential privacy can be achieved with less noise; whereas the memorization technique, which remembers unused gradient parts, keeps track of the descent direction and thereby maintains the accuracy of the proposed algorithm. Our differentially private algorithm, called dp-memSGD for short, converges mathematically at the same rate of 1/T as standard stochastic gradient descent (SGD) algorithm, where T is the number of training iterations. Experimentally, we demonstrate that dp-memSGD converges with reasonable privacy losses on many benchmark datasets.

A Generative Model for Texture Synthesis based on Optimal Transport Between Feature Distributions

We propose GOTEX, a general framework for texture synthesis by optimization that constrains the statistical distribution of local features. While our model encompasses several existing texture models, we focus on the case where the comparison between feature distributions relies on optimal transport distances. We show that the semi-dual formulation of optimal transport allows to control the distribution of various possible features, even if these features live in a high-dimensional space. We then study the resulting minimax optimization problem, which corresponds to a Wasserstein generative model, for which the inner concave maximization problem can be solved with standard stochastic gradient methods. The alternate optimization algorithm is shown to be versatile in terms of applications, features and architecture; in particular, it allows to produce high-quality synthesized textures with different sets of features. We analyze the results obtained by constraining the distribution of patches or the distribution of responses to a pre-learned VGG neural network. We show that the patch representation can retrieve the desired textural aspect in a more precise manner. We also provide a detailed comparison with state-of-the-art texture synthesis methods. The GOTEX model based on patch features is also adapted to texture inpainting and texture interpolation. Finally, we show how to use our framework to learn a feed-forward neural network that can synthesize on-the-fly new textures of arbitrary size in a very fast manner. Experimental results and comparisons with the mainstream methods from the literature illustrate the relevance of the generative models learned with GOTEX.

Learning to refine source representations for neural machine translation

Machine translation is one of the most classic application technologies in artificial intelligence and natural language processing. Neural machine translation models generally adopt an encoder–decoder architecture for modeling the entire translation process. However, without considering target context (e.g., decoding state) to guide the encoding, encoded source representations struggle to put great emphasis on important information for predicting some target word, yielding the weakness in generating more discriminative attentive representations across different decoding steps. Towards tackling this issue, we propose a novel encoder–refiner–decoder framework, which dynamically refines the source representations based on the generated target-side information at each decoding step. Since the refining operations are time-consuming, we propose a policy network to decide when to refine at specific decoding steps. We solve such a problem using the Gumbel-Softmax reparameterization, which makes our network differentiable and trainable through standard stochastic gradient methods. Experimental results on both Chinese–English and English–German translation tasks show that the proposed approach significantly and consistently improves translation performance over the standard encoder–decoder framework. Furthermore, when refining strategy is applied, experimental results still show a reasonable improvement over the baseline with much decrease in decoding speed.

A distributed optimisation framework combining natural gradient with Hessian-free for discriminative sequence training

This paper presents a novel natural gradient and Hessian-free (NGHF) optimisation framework for neural network training that can operate efficiently in a distributed manner. It relies on the linear conjugate gradient (CG) algorithm to combine the natural gradient (NG) method with local curvature information from Hessian-free (HF). A solution to a numerical issue in CG allows effective parameter updates to be generated with far fewer CG iterations than usually used (e.g. 5-8 instead of 200). This work also presents a novel preconditioning approach to improve the progress made by individual CG iterations for models with shared parameters. Although applicable to other training losses and model structures, NGHF is investigated in this paper for lattice-based discriminative sequence training for hybrid hidden Markov model acoustic models using a standard recurrent neural network, long short-term memory, and time delay neural network models for output probability calculation. Automatic speech recognition experiments are reported on the multi-genre broadcast data set for a range of different acoustic model types. These experiments show that NGHF achieves larger word error rate reductions than standard stochastic gradient descent or Adam, while requiring orders of magnitude fewer parameter updates.

Zeroth-Order Nonconvex Stochastic Optimization: Handling Constraints, High Dimensionality, and Saddle Points

In this paper, we propose and analyze zeroth-order stochastic approximation algorithms for nonconvex and convex optimization, with a focus on addressing constrained optimization, high-dimensional setting, and saddle point avoiding. To handle constrained optimization, we first propose generalizations of the conditional gradient algorithm achieving rates similar to the standard stochastic gradient algorithm using only zeroth-order information. To facilitate zeroth-order optimization in high dimensions, we explore the advantages of structural sparsity assumptions. Specifically, (i) we highlight an implicit regularization phenomenon where the standard stochastic gradient algorithm with zeroth-order information adapts to the sparsity of the problem at hand by just varying the step size and (ii) propose a truncated stochastic gradient algorithm with zeroth-order information, whose rate of convergence depends only poly-logarithmically on the dimensionality. We next focus on avoiding saddle points in nonconvex setting. Toward that, we interpret the Gaussian smoothing technique for estimating gradient based on zeroth-order information as an instantiation of first-order Stein’s identity. Based on this, we provide a novel linear-(in dimension) time estimator of the Hessian matrix of a function using only zeroth-order information, which is based on second-order Stein’s identity. We then provide a zeroth-order variant of cubic regularized Newton method for avoiding saddle points and discuss its rate of convergence to local minima.

Fully asynchronous stochastic coordinate descent: a tight lower bound on the parallelism achieving linear speedup

We seek tight bounds on the viable parallelism in asynchronous implementations of coordinate descent that achieves linear speedup. We focus on asynchronous coordinate descent (ACD) algorithms on convex functions which consist of the sum of a smooth convex part and a possibly non-smooth separable convex part. We quantify the shortfall in progress compared to the standard sequential stochastic gradient descent. This leads to a simple yet tight analysis of the standard stochastic ACD in a partially asynchronous environment, generalizing and improving the bounds in prior work. We also give a considerably more involved analysis for general asynchronous environments in which the only constraint is that each update can overlap with at most q others. The new lower bound on the maximum degree of parallelism attaining linear speedup is tight and improves the best prior bound almost quadratically.

Ritz-like values in steplength selections for stochastic gradient methods

The steplength selection is a crucial issue for the effectiveness of the stochastic gradient methods for large-scale optimization problems arising in machine learning. In a recent paper, Bollapragada et al. (SIAM J Optim 28(4):3312–3343, 2018) propose to include an adaptive subsampling strategy into a stochastic gradient scheme, with the aim to assure the descent feature in expectation of the stochastic gradient directions. In this approach, theoretical convergence properties are preserved under the assumption that the positive steplength satisfies at any iteration a suitable bound depending on the inverse of the Lipschitz constant of the objective function gradient. In this paper, we propose to tailor for the stochastic gradient scheme the steplength selection adopted in the full-gradient method knows as limited memory steepest descent method. This strategy, based on the Ritz-like values of a suitable matrix, enables to give a local estimate of the inverse of the local Lipschitz parameter, without introducing line search techniques, while the possible increase in the size of the subsample used to compute the stochastic gradient enables to control the variance of this direction. An extensive numerical experimentation highlights that the new rule makes the tuning of the parameters less expensive than the trial procedure for the efficient selection of a constant step in standard and mini-batch stochastic gradient methods.

Learning Semisupervised Multilabel Fully Convolutional Network for Hierarchical Object Parsing.

This article presents a semisupervised multilabel fully convolutional network (FCN) for hierarchical object parsing of images. We consider each object part (e.g., eye and head) as a class label and learn to assign every image pixel to multiple coherent part labels. Different from previous methods that consider part labels as independent classes, our method explicitly models the internal relationships between object parts, e.g., that a pixel highly scored for eyes should be highly scored for heads as well. Such relationships directly reflect the structure of the semantic space and thus should be respected while learning the deep representation. We achieve this objective by introducing a multilabel softmax loss function over both labeled and unlabeled images and regularizing it with two pairwise ranking constraints. The first constraint is based on a manifold assumption that image pixels being visually and spatially close to each other should be collaboratively classified as the same part label. The other constraint is used to enforce that no pixel receives significant scores from more than one label that are semantically conflicting with each other. The proposed loss function is differentiable with respect to network parameters and hence can be optimized by standard stochastic gradient methods. We evaluate the proposed method on two public image data sets for hierarchical object parsing and compare it with the alternative parsing methods. Extensive comparisons showed that our method can achieve state-of-the-art performance while using 50% less labeled training samples than the alternatives.

Stochastic Gradient Descent–Whale Optimization Algorithm-Based Deep Convolutional Neural Network To Crowd Emotion Understanding

AbstractCrowd emotion understanding is an interesting research area that assists the security personnel to read the emotion/activity of the crowd in the locality. Most of the traditional methods utilize the low-level visual features to understand the crowd emotions that extend the gap between the low- and the high-level features. With the aim to develop an automatic method for emotion recognition, this paper utilizes the deep convolutional neural network (deep CNN). For the effective emotion recognition, it is essential to select the key frames of the video using the wavelet-based Bhattacharya distance. The key frames are fed to the space-time interest points descriptor that extracts the features and forms the input vector to the classifier. Deep CNN is trained using the proposed Stochastic Gradient Descent–Whale Optimization Algorithm, which is the unification of the standard stochastic gradient descent algorithm with whale optimization algorithm. The proposed classifier recognizes the emotions of the crowd, such as angry, escape, fight, happy, normal, running/walking and violence. The analysis of the method proved that the proposed approaches acquired a maximal accuracy, specificity and sensitivity of 0.9693, 0.9936 and 0.9675, respectively.

On Intra-Class Variance for Deep Learning of Classifiers

Abstract A novel technique for deep learning of image classifiers is presented. The learned CNN models higher offer better separation of deep features (also known as embedded vectors) measured by Euclidean proximity and also no deterioration of the classification results by class membership probability. The latter feature can be used for enhancing image classifiers having the classes at the model’s exploiting stage different from from classes during the training stage. While the Shannon information of SoftMax probability for target class is extended for mini-batch by the intra-class variance, the trained network itself is extended by the Hadamard layer with the parameters representing the class centers. Contrary to the existing solutions, this extra neural layer enables interfacing of the training algorithm to the standard stochastic gradient optimizers, e.g. AdaM algorithm. Moreover, this approach makes the computed centroids immediately adapting to the updating embedded vectors and finally getting the comparable accuracy in less epochs.

Clustering-driven unsupervised deep hashing for image retrieval

Unsupervised deep hash functions are complicated due to the challenges of learning discriminative clusters and the absence of similarity-sensitive objectives. Existing approaches predominately handle these challenges independently, while neglecting the fact that cluster centroids and similarity-sensitive binary codes are correlated and can be learned simultaneously. In this paper, we propose a novel end-to-end deep framework for image retrieval, namely Clustering-driven Unsupervised Deep Hashing (CUDH), to recursively learn discriminative clusters by soft clustering model and produce binary code with high similarity responds. We employ the aggregated clusters as an auxiliary distribution to generate hashing codes. With imposing binary constraints loss and reconstruction loss of auto-encoder, our CUDH can be jointly optimized by standard stochastic gradient descent (SGD). Comprehensive experiments on three popular datasets are conducted and the results show that our CUDH can outperform the state-of-the-art methods by large margins.

Particle filtering‐based recursive identification for controlled auto‐regressive systems with quantised output

Recursive prediction error method is one of the main tools for analysis of controlled auto-regressive systems with quantised output. In this study, a recursive identification algorithm is proposed based on the auxiliary model principle by modifying the standard stochastic gradient algorithm. To improve the convergence performance of the algorithm, a particle filtering technique, which approximates the posterior probability density function with a weighted set of discrete random sampling points is utilised to correct the linear output estimates. It can exclude those invalid particles according to their corresponding weights. The performance of the particle filtering technique-based algorithm is much better than that of the auxiliary model-based one. Finally, results are verified by examples from simulation and engineering.