Standard Stochastic Gradient Descent Research Articles

Variance-Reduced Accelerated First-Order Methods: Central Limit Theorems and Confidence Statements

In this paper, we consider a strongly convex stochastic optimization problem and propose three classes of variable sample-size stochastic first-order methods: (i) the standard stochastic gradient descent method, (ii) its accelerated variant, and (iii) the stochastic heavy-ball method. In each scheme, the exact gradients are approximated by averaging across an increasing batch size of sampled gradients. We prove that when the sample size increases at a geometric rate, the generated estimates converge in mean to the optimal solution at an analogous geometric rate for schemes (i)–(iii). Based on this result, we provide central limit statements, whereby it is shown that the rescaled estimation errors converge in distribution to a normal distribution with the associated covariance matrix dependent on the Hessian matrix, the covariance of the gradient noise, and the step length. If the sample size increases at a polynomial rate, we show that the estimation errors decay at a corresponding polynomial rate and establish the associated central limit theorems (CLTs). Under certain conditions, we discuss how both the algorithms and the associated limit theorems may be extended to constrained and nonsmooth regimes. Finally, we provide an avenue to construct confidence regions for the optimal solution based on the established CLTs and test the theoretical findings on a stochastic parameter estimation problem. Funding: This work was supported by the Office of Naval Research [Grant N00014-22-1-2589] and Basic Energy Sciences [Grant DE-SC0023303]. The work of J. Lei was partially supported by the National Natural Science Foundation of China [Grants 72271187 and 62088101]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/moor.2021.0068 .

A Nonlinear PID-Incorporated Adaptive Stochastic Gradient Descent Algorithm for Latent Factor Analysis

High-dimensional and incomplete (HDI) data are commonly encountered in various big data-related applications concerning the complex interactions among numerous nodes, such as the user-item iterations in a recommender system. A stochastic gradient descent (SGD)-based latent factor analysis (LFA) model can perform efficient representation learning to such HDI data, thereby extracting useful knowledge from them. However, a standard SGD algorithm updates a latent factor based on the current stochastic gradient only, without the considerations on the past information, making a resultant model suffer from slow convergence. To address this critical issue, this paper proposes an Adaptive Non-linear PID-incorporated SGD (ANPS) algorithm with two-fold ideas: 1) rebuilding the instant learning error when computing the stochastic gradient following the principle of a nonlinear PID controller to incorporate past update information into the learning scheme efficiently, and 2) implementing gain parameter adaptation following the principle of particle swarm optimization (PSO). Experiments on six widely-adopted HDI datasets demonstrate that compared with state-of-the-art LFA models, an ANPS-based LFA model achieves significant advantage in both efficiency and accuracy. Moreover, its flexible gain parameter adaptation mechanism greatly boosts its practicability for real issues. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —In many industrial applications like recommender systems, social network systems, and cloud service systems, people usually encounter numerous nodes and their highly-incomplete relationships. An HDI matrix is commonly adopted to describe such specific relationships. One of the major challenges is to acquire useful knowledge from an HDI matrix efficiently and accurately for various data analysis tasks, e.g., accurate recommendation, community detection, and web service selection. An SGD-based LFA model has been widely adopted to tackle this issue. However, it suffers from slow convergence that leads to considerable time cost on large-scale datasets. This study proposes an ANPS algorithm following the principle of a nonlinear PID controller. With it, an ANPS-based LFA model is achieved, which possesses fast convergence rate on an industrial HDI matrix. The proposed ANPS algorithm can be leveraged for different types of various machine learning models, thereby improving their utility and scalability in practice.

Pengembangan Stochastic Gradient Descent dengan Penambahan Variabel Tetap

Stochastic Gradient Descent (SGD) is one of the commonly used optimizers in deep learning. Therefore, in this work, we modify stochastic gradient descent (SGD) by adding a fixed variable. We will then look at the differences between standard stochastic gradient descent (SGD) and stochastic gradient descent (SGD) with additional variables. The phases performed in this study were: (1) optimization analysis, (2) fix design, (3) fix implementation, (4) fix test, (5) reporting. The results of this study aim to show the additional impact of fixed variables on the performance of stochastic gradient descent (SGD).

Coherent optical neuron control based on reinforcement learning.

Optical neural networks take optical neurons as the cornerstone to achieve complex functions. The coherent optical neuron has become one of the mainstream implementations because it can effectively perform natural and even complex number calculations. However, its state variability and requirement for reliability and effectiveness render traditional control methods no longer applicable. In this Letter, deep reinforcement coherent optical neuron control (DRCON) is proposed, and its effectiveness is experimentally demonstrated. Compared with the standard stochastic gradient descent, the average convergence rate of DRCON is 33% faster, while the effective number of bits increases from less than 2 bits to 5.5 bits. DRCON is a promising first step for large-scale optical neural network control.

Modelling continual learning in humans with Hebbian context gating and exponentially decaying task signals.

Humans can learn several tasks in succession with minimal mutual interference but perform more poorly when trained on multiple tasks at once. The opposite is true for standard deep neural networks. Here, we propose novel computational constraints for artificial neural networks, inspired by earlier work on gating in the primate prefrontal cortex, that capture the cost of interleaved training and allow the network to learn two tasks in sequence without forgetting. We augment standard stochastic gradient descent with two algorithmic motifs, so-called "sluggish" task units and a Hebbian training step that strengthens connections between task units and hidden units that encode task-relevant information. We found that the "sluggish" units introduce a switch-cost during training, which biases representations under interleaved training towards a joint representation that ignores the contextual cue, while the Hebbian step promotes the formation of a gating scheme from task units to the hidden layer that produces orthogonal representations which are perfectly guarded against interference. Validating the model on previously published human behavioural data revealed that it matches performance of participants who had been trained on blocked or interleaved curricula, and that these performance differences were driven by misestimation of the true category boundary.

Natural Evolutionary Gradient Descent Strategy for Variational Quantum Algorithms

Recent research has demonstrated that parametric quantum circuits (PQCs) are affected by gradients that progressively vanish to zero as a function of the number of qubits. We show that using a combination of gradient-free natural evolutionary strategy and gradient descent can mitigate the possibility of optimizing barren plateaus in the landscape. We implemented 2 specific methods: natural evolutionary strategy stochastic gradient descent (NESSGD) and natural evolutionary strategy adapting the step size according to belief in observed gradients (NESAdaBelief) to optimize PQC parameter values. They were compared with standard stochastic gradient descent, adaptive moment estimation, and a version of adaptive moment estimation adapting the step size according to belief in observed gradients in 5 classification tasks. NESSGD and NESAdaBelief demonstrated some superiority in 4 of the tasks. NESAdaBelief showed higher accuracy than AdaBelief in all 5 tasks. In addition, we investigated the applicability of NESSGD under the parameter shift rule and demonstrated that NESSGD can adapt to this rule, which means that our proposed method could also optimize the parameters of PQCs on quantum computers.

A hybrid artificial bee colony algorithmic approach for classification using neural networks

Artificial neural networks are an integral component of most corporate and research functions across different platforms. However, depending upon the nature of the problem and quality of initialisation values, the usage of standard stochastic gradient descent always risks the possibility of getting trapped in local minima and saddle points for smaller neural networks in particular. One way to overcome this is by using algorithms with proven global search capabilities to train the network. This allows the neural net to reach the optimum values for weights regardless of the initialisation parameters used during training. Two algorithms are proposed based on modifications to the original artificial bee colony algorithm and their performances are analysed extensively on three benchmark datasets of increasing complexity. The first (NMABC), employs neural network appropriate initialisation and linear search space expansion. This is integrated into the second (LHABC), and incorporates stochastic gradient descent into the employed phase of the bees for faster convergence. It is found that the proposed algorithms consistently outperform standard approaches in all cases.

Differentially private stochastic gradient descent via compression and memorization

We propose a novel approach for achieving differential privacy for neural network training models through compression and memorization of gradients. The compression technique, which makes gradient vectors sparse, reduces the sensitivity so that differential privacy can be achieved with less noise; whereas the memorization technique, which remembers unused gradient parts, keeps track of the descent direction and thereby maintains the accuracy of the proposed algorithm. Our differentially private algorithm, called dp-memSGD for short, converges mathematically at the same rate of 1/T as standard stochastic gradient descent (SGD) algorithm, where T is the number of training iterations. Experimentally, we demonstrate that dp-memSGD converges with reasonable privacy losses on many benchmark datasets.

An Improved Bees Algorithm for Training Deep Recurrent Networks for Sentiment Classification

Recurrent neural networks (RNNs) are powerful tools for learning information from temporal sequences. Designing an optimum deep RNN is difficult due to configuration and training issues, such as vanishing and exploding gradients. In this paper, a novel metaheuristic optimisation approach is proposed for training deep RNNs for the sentiment classification task. The approach employs an enhanced Ternary Bees Algorithm (BA-3+), which operates for large dataset classification problems by considering only three individual solutions in each iteration. BA-3+ combines the collaborative search of three bees to find the optimal set of trainable parameters of the proposed deep recurrent learning architecture. Local learning with exploitative search utilises the greedy selection strategy. Stochastic gradient descent (SGD) learning with singular value decomposition (SVD) aims to handle vanishing and exploding gradients of the decision parameters with the stabilisation strategy of SVD. Global learning with explorative search achieves faster convergence without getting trapped at local optima to find the optimal set of trainable parameters of the proposed deep recurrent learning architecture. BA-3+ has been tested on the sentiment classification task to classify symmetric and asymmetric distribution of the datasets from different domains, including Twitter, product reviews, and movie reviews. Comparative results have been obtained for advanced deep language models and Differential Evolution (DE) and Particle Swarm Optimization (PSO) algorithms. BA-3+ converged to the global minimum faster than the DE and PSO algorithms, and it outperformed the SGD, DE, and PSO algorithms for the Turkish and English datasets. The accuracy value and F1 measure have improved at least with a 30–40% improvement than the standard SGD algorithm for all classification datasets. Accuracy rates in the RNN model trained with BA-3+ ranged from 80% to 90%, while the RNN trained with SGD was able to achieve between 50% and 60% for most datasets. The performance of the RNN model with BA-3+ has as good as for Tree-LSTMs and Recursive Neural Tensor Networks (RNTNs) language models, which achieved accuracy results of up to 90% for some datasets. The improved accuracy and convergence results show that BA-3+ is an efficient, stable algorithm for the complex classification task, and it can handle the vanishing and exploding gradients problem of deep RNNs.

A distributed optimisation framework combining natural gradient with Hessian-free for discriminative sequence training

This paper presents a novel natural gradient and Hessian-free (NGHF) optimisation framework for neural network training that can operate efficiently in a distributed manner. It relies on the linear conjugate gradient (CG) algorithm to combine the natural gradient (NG) method with local curvature information from Hessian-free (HF). A solution to a numerical issue in CG allows effective parameter updates to be generated with far fewer CG iterations than usually used (e.g. 5-8 instead of 200). This work also presents a novel preconditioning approach to improve the progress made by individual CG iterations for models with shared parameters. Although applicable to other training losses and model structures, NGHF is investigated in this paper for lattice-based discriminative sequence training for hybrid hidden Markov model acoustic models using a standard recurrent neural network, long short-term memory, and time delay neural network models for output probability calculation. Automatic speech recognition experiments are reported on the multi-genre broadcast data set for a range of different acoustic model types. These experiments show that NGHF achieves larger word error rate reductions than standard stochastic gradient descent or Adam, while requiring orders of magnitude fewer parameter updates.

Latent Factor-Based Recommenders Relying on Extended Stochastic Gradient Descent Algorithms

High-dimensional and sparse (HiDS) matrices generated by recommender systems contain rich knowledge regarding various desired patterns like users' potential preferences and community tendency. Latent factor (LF) analysis proves to be highly efficient in extracting such knowledge from an HiDS matrix efficiently. Stochastic gradient descent (SGD) is a highly efficient algorithm for building an LF model. However, current LF models mostly adopt a standard SGD algorithm. Can SGD be extended from various aspects in order to improve the resultant models' convergence rate and prediction accuracy for missing data? Are such SGD extensions compatible with an LF model? To answer them, this paper carefully investigates eight extended SGD algorithms to propose eight novel LF models. Experimental results on two HiDS matrices generated by real recommender systems show that compared with an LF model with a standard SGD algorithm, an LF model with extended ones can achieve: 1) higher prediction accuracy for missing data; 2) faster convergence rate; and 3) model diversity.

A proportional-integral-derivative-incorporated stochastic gradient descent-based latent factor analysis model

Large-scale relationships like user-item preferences in a recommender system are mostly described by a high-dimensional and sparse (HiDS) matrix. A latent factor analysis (LFA) model extracts useful knowledge from an HiDS matrix efficiently, where stochastic gradient descent (SGD) is frequently adopted as the learning algorithm. However, a standard SGD algorithm updates a decision parameter with the stochastic gradient on the instant loss only, without considering information described by prior updates. Hence, an SGD-based LFA model commonly consumes many iterations to converge, which greatly affects its practicability. On the other hand, a proportional-integral-derivative (PID) controller makes a learning model converge fast with the consideration of its historical errors from the initial state till the current moment. Motivated by this discovery, this paper proposes a PID-incorporated SGD-based LFA (PSL) model. Its main idea is to rebuild the instant error on a single instance following the principle of PID, and then substitute this rebuilt error into an SGD algorithm for accelerating model convergence. Empirical studies on six widely-accepted HiDS matrices indicate that compared with state-of-the-art LFA models, a PSL model achieves significantly higher computational efficiency as well as highly competitive prediction accuracy for missing data of an HiDS matrix.

Large-scale and Scalable Latent Factor Analysis via Distributed Alternative Stochastic Gradient Descent for Recommender Systems

Latent factor analysis (LFA) via stochastic gradient descent (SGD) is highly efficient in discovering user and item patterns from high-dimensional and sparse (HiDS) matrices from recommender systems. However, most LFA-based recommender systems adopt a standard SGD algorithm, which suffers limited scalability when addressing big data. On the other hand, most existing parallel SGD solvers are either under the memory-sharing framework designed for a bare machine or suffering high communicational costs, which also greatly limits their applications in large-scale systems. To address the above issues, this article proposes a distributed alternative stochastic gradient descent (DASGD) solver for an LFA-based recommender. Its training-dependences among latent features are decoupled via alternatively fixing one-half of the features to learn the other half following the principle of SGD but in parallel. It's distribution mechanism consists of efficient data partition, allocation and task parallelization strategies, which greatly reduces its communicational cost for high scalability. Experimental results on three large-scale HiDS matrices generated by real-world applications demonstrate that the proposed DASGD algorithm outperforms state-of-the-art distributed SGD solvers for recommender systems in terms of prediction accuracy as well as scalability. Hence, it is highly useful for training LFA-based recommenders on large scale HiDS matrices with the help of cloud computing facilities.

Stochastic Gradient Descent–Whale Optimization Algorithm-Based Deep Convolutional Neural Network To Crowd Emotion Understanding

AbstractCrowd emotion understanding is an interesting research area that assists the security personnel to read the emotion/activity of the crowd in the locality. Most of the traditional methods utilize the low-level visual features to understand the crowd emotions that extend the gap between the low- and the high-level features. With the aim to develop an automatic method for emotion recognition, this paper utilizes the deep convolutional neural network (deep CNN). For the effective emotion recognition, it is essential to select the key frames of the video using the wavelet-based Bhattacharya distance. The key frames are fed to the space-time interest points descriptor that extracts the features and forms the input vector to the classifier. Deep CNN is trained using the proposed Stochastic Gradient Descent–Whale Optimization Algorithm, which is the unification of the standard stochastic gradient descent algorithm with whale optimization algorithm. The proposed classifier recognizes the emotions of the crowd, such as angry, escape, fight, happy, normal, running/walking and violence. The analysis of the method proved that the proposed approaches acquired a maximal accuracy, specificity and sensitivity of 0.9693, 0.9936 and 0.9675, respectively.

Design of normalized fractional SGD computing paradigm for recommender systems

Fast and effective recommender systems are fundamental to fulfill the growing requirements of the e-commerce industry. The strength of matrix factorization procedure based on stochastic gradient descent (SGD) algorithm is exploited widely to solve the recommender system problem. Modern computing paradigms are designed by utilizing the concept of fractional gradient in standard SGD and outperform the standard counterpart. The performance of fractional SGD improves considerably by adaptively tuning the learning rate parameter. A nonlinear computing paradigm based on normalized version of fractional SGD is developed in this paper to investigate the adaptive behavior of learning rate with novel application to recommender systems. The accuracy of the proposed approach is verified through root mean square error metric by using different latent features, learning rates, fractional orders and datasets. The superiority of the designed method is validated through comparison with the state-of-the-art counterparts.

Clustering-driven unsupervised deep hashing for image retrieval

Unsupervised deep hash functions are complicated due to the challenges of learning discriminative clusters and the absence of similarity-sensitive objectives. Existing approaches predominately handle these challenges independently, while neglecting the fact that cluster centroids and similarity-sensitive binary codes are correlated and can be learned simultaneously. In this paper, we propose a novel end-to-end deep framework for image retrieval, namely Clustering-driven Unsupervised Deep Hashing (CUDH), to recursively learn discriminative clusters by soft clustering model and produce binary code with high similarity responds. We employ the aggregated clusters as an auxiliary distribution to generate hashing codes. With imposing binary constraints loss and reconstruction loss of auto-encoder, our CUDH can be jointly optimized by standard stochastic gradient descent (SGD). Comprehensive experiments on three popular datasets are conducted and the results show that our CUDH can outperform the state-of-the-art methods by large margins.

Active Mini-Batch Sampling Using Repulsive Point Processes

The convergence speed of stochastic gradient descent (SGD) can be improved by actively selecting mini-batches. We explore sampling schemes where similar data points are less likely to be selected in the same mini-batch. In particular, we prove that such repulsive sampling schemes lower the variance of the gradient estimator. This generalizes recent work on using Determinantal Point Processes (DPPs) for mini-batch diversification (Zhang et al., 2017) to the broader class of repulsive point processes. We first show that the phenomenon of variance reduction by diversified sampling generalizes in particular to non-stationary point processes. We then show that other point processes may be computationally much more efficient than DPPs. In particular, we propose and investigate Poisson Disk sampling—frequently encountered in the computer graphics community—for this task. We show empirically that our approach improves over standard SGD both in terms of convergence speed as well as final model performance.

Effects of Extended Stochastic Gradient Descent Algorithms on Improving Latent Factor-Based Recommender Systems

High-dimensional and sparse (HiDS) matrices from recommender systems contain various useful patterns. A latent factor (LF) analysis is highly efficient in grasping these patterns. Stochastic gradient descent (SGD) is a widely adopted algorithm to train an LF model. Can its extensions be capable of further improving an LF models’ convergence rate and prediction accuracy for missing data? To answer this question, this work selects two of representative extended SGD algorithms to propose two novel LF models. Experimental results from two HiDS matrices generated by real recommender systems show that compared standard SGD, extended SGD algorithms enable an LF model to achieve a higher prediction accuracy for missing data of an HiDS matrix, a faster convergence rate, and a larger model diversity.

SlimML: Removing Non-critical Input Data in Large-scale Iterative Machine Learning

The core of many large-scale machine learning (ML) applications, such as neural networks (NN), support vector machine (SVM), and convolutional neural network (CNN), is the training algorithm that iteratively updates model parameters by processing massive datasets. From a plethora of studies aiming at accelerating ML, being data parallelization and parameter server, the prevalent assumption is that all data points are equivalently relevant to model parameter updating. In this article, we challenge this assumption by proposing a criterion to measure a data point's effect on model parameter updating, and experimentally demonstrate that the majority of data points are non-critical in the training process. We develop a slim learning framework, termed SlimML, which trains the ML models only on the critical data and thus significantly improves training performance. To such an end, SlimML efficiently leverages a small number of aggregated data points per iteration to approximate the criticalness of original input data instances. The proposed approach can be used by changing a few lines of code in a standard stochastic gradient descent (SGD) procedure, and we demonstrate experimentally, on NN regression, SVM classification, and CNN training, that for large datasets, it accelerates model training process by an average of 3.61 times while only incurring accuracy losses of 0.37 percent.

Design of Momentum Fractional Stochastic Gradient Descent for Recommender Systems

The demand for recommender systems in E-commerce industry has increased tremendously. Efficient recommender systems are being proposed by different E-business companies with the intention to give users accurate and most relevant recommendation of products from huge amount of information. To improve the performance of recommender systems, various stochastic variants of gradient descent based algorithms have been reported. The scalability requirement of recommender systems needs algorithms with fast convergence to generate recommendations of specific items. Using the concepts of fractional calculus, an efficient variant of the stochastic gradient descent (SGD) was developed for fast convergence. Such fractional SGD (F-SGD) is further accelerated by adding a momentum term, thus termed as momentum fractional stochastic gradient descent (mF-SGD). The proposed mF-SGD method is shown to offer improved estimation accuracy and convergence rate, as compared to F-SGD and standard momentum SGD for different proportions of previous gradients, fractional orders, learning rates and number of features.