Convergence Of Stochastic Gradient Descent Research Articles

Approximating Hessian matrices using Bayesian inference: a new approach for quasi-Newton methods in stochastic optimization

Using quasi-Newton methods in stochastic optimization is not a trivial task given the difficulty of extracting curvature information from the noisy gradients. Moreover, pre-conditioning noisy gradient observations tend to amplify the noise. We propose a Bayesian approach to obtain a Hessian matrix approximation for stochastic optimization that minimizes the secant equations residue while retaining the extreme eigenvalues between a specified range. Thus, the proposed approach assists stochastic gradient descent to converge to local minima without augmenting gradient noise. We propose maximizing the log posterior using the Newton-CG method. Numerical results on a stochastic quadratic function and an ℓ 2 -regularized logistic regression problem are presented. In all the cases tested, our approach improves the convergence of stochastic gradient descent, compensating for the overhead of solving the log posterior maximization. In particular, pre-conditioning the stochastic gradient with the inverse of our Hessian approximation becomes more advantageous the larger the condition number of the problem is.

Optimized convergence of stochastic gradient descent by weighted averaging

Under mild assumptions stochastic gradient methods asymptotically achieve an optimal rate of convergence if the arithmetic mean of all iterates is returned as an approximate optimal solution. However, in the absence of stochastic noise, the arithmetic mean of all iterates converges considerably slower to the optimal solution than the iterates themselves. And also in the presence of noise, when a termination of the stochastic gradient method after a finite number of steps is considered, the arithmetic mean is not necessarily the best possible approximation to the unknown optimal solution. This paper aims at identifying optimal strategies in a particularly simple case, the minimization of a strongly convex function with i. i. d. noise terms and termination after a finite number of steps. Explicit formulas for the stochastic error and the optimality error are derived in dependence of certain parameters of the SGD method. The aim was to choose parameters such that both stochastic error and optimality error are reduced compared to arithmetic averaging. This aim could not be achieved; however, by allowing a slight increase of the stochastic error it was possible to select the parameters such that a significant reduction of the optimality error could be achieved. This reduction of the optimality error has a strong effect on the approximate solution generated by the stochastic gradient method in case that only a moderate number of iterations is used or when the initial error is large. The numerical examples confirm the theoretical results and suggest that a generalization to non-quadratic objective functions may be possible.

Modeling nonlinear flutter behavior of long‐span bridges using knowledge‐enhanced long short‐term memory network

AbstractThe nonlinear characteristics of bridge aerodynamics preclude a closed‐form solution of limit‐cycle oscillation (LCO) amplitude and frequency in the post‐flutter stage. To address this issue, a long short‐term memory (LSTM) network is utilized as the reduced‐order modeling of nonlinear aeroelastic forces on the bridge deck section, and it is repeatedly employed to generate force inputs at spanwise nodes of a three‐dimensional (3D) finite element model (FEM) of the long‐span bridge (using spatial beam elements). All LSTM networks are dynamically coupled through FEM, and the 3D nonlinear flutter response is accordingly obtained. To improve the simulation accuracy and reduce the required training data of the standard LSTM network, both general knowledge (motivated by the gating mechanism and mathematical models for information processing) and domain knowledge (resulting from the basic understanding of bridge aerodynamics) are leveraged to, respectively, customize the LSTM cell and network architecture. In addition, a fast‐training algorithm effectively combining the linear convergence of stochastic gradient descent and superlinear convergence of modified Broyden–Fletcher–Goldfarb–Shanno is developed to improve the training efficiency of the obtained knowledge‐enhanced LSTM network. To further advance the computational efficiency of the coupled LSTM‐FEM nonlinear flutter analysis, the convolution‐based numerical integration is adopted in the finite element modeling of long‐span bridge dynamics. A case study of a long‐span suspension bridge under strong winds demonstrates the proposed 3D nonlinear flutter analysis presents high simulation efficiency and accuracy and can be utilized to effectively obtain the nonlinear LCO characteristics in a wide range of post‐flutter wind speeds.

Accelerating variance-reduced stochastic gradient methods

Variance reduction is a crucial tool for improving the slow convergence of stochastic gradient descent. Only a few variance-reduced methods, however, have yet been shown to directly benefit from Nesterov’s acceleration techniques to match the convergence rates of accelerated gradient methods. Such approaches rely on “negative momentum”, a technique for further variance reduction that is generally specific to the SVRG gradient estimator. In this work, we show for the first time that negative momentum is unnecessary for acceleration and develop a universal acceleration framework that allows all popular variance-reduced methods to achieve accelerated convergence rates. The constants appearing in these rates, including their dependence on the number of functions n, scale with the mean-squared-error and bias of the gradient estimator. In a series of numerical experiments, we demonstrate that versions of SAGA, SVRG, SARAH, and SARGE using our framework significantly outperform non-accelerated versions and compare favourably with algorithms using negative momentum.

Variance Counterbalancing for Stochastic Large-scale Learning

Stochastic Gradient Descent (SGD) is perhaps the most frequently used method for large scale training. A common example is training a neural network over a large data set, which amounts to minimizing the corresponding mean squared error (MSE). Since the convergence of SGD is rather slow, acceleration techniques based on the notion of “Mini-Batches” have been developed. All of them however, mimicking SGD, impose diminishing step-sizes as a means to inhibit large variations in the MSE objective. In this article, we introduce random sets of mini-batches instead of individual mini-batches. We employ an objective function that minimizes the average MSE and its variance over these sets, eliminating so the need for the systematic step size reduction. This approach permits the use of state-of-the-art optimization methods, far more efficient than the gradient descent, and yields a significant performance enhancement.

On the Convergence of Stochastic Gradient Descent for Nonlinear Ill-Posed Problems

In this work, we analyze the regularizing property of the stochastic gradient descent for the efficient numerical solution of a class of nonlinear ill-posed inverse problems in Hilbert spaces. At each step of the iteration, the method randomly chooses one equation from the nonlinear system to obtain an unbiased stochastic estimate of the gradient, and then performs a descent step with the estimated gradient. It is a randomized version of the classical Landweber method for nonlinear inverse problems, and it is highly scalable to the problem size and holds significant potentials for solving large-scale inverse problems. Under the canonical tangential cone condition, we prove the regularizing property for a priori stopping rules, and then establish the convergence rates under suitable sourcewise condition and range invariance condition.

An SGD-based meta-learner with “growing” descent

The paper considers the problem of accelerating the convergence of stochastic gradient descent (SGD) in an automatic way. Previous research puts forward such algorithms as Adagrad, Adadelta, RMSprop, Adam and etc. to adapt both the updates and learning rates to the slope of a loss function. However, these adaptive methods do not share the same regret bound as the gradient descent method. Adagrad provably achieves the optimal regret bound on the assumption of convexity but accumulates the squared gradients in the denominator that dramatically shrinks the learning rate. This research is aimed at introducing a generalized logistic map directly into the SGD method in order to automatically set its parameters to the slope of the logistic loss function. The optimizer based on the population may be considered as a meta-learner that learns how to tune both the learning rate and gradient updates with respect to the rate of population growth. The present study yields the “growing” descent method and a series of computational experiments to point out the benefits of the proposed meta-learner.

Influence of feature scaling on convergence of gradient iterative algorithm

Feature scaling is a method to unify self-variables or feature ranges in data. In data processing, it is usually used in data pre-processing. Because in the original data, the range of variables is very different. Feature scaling is a necessary step in the calculation of stochastic gradient descent. This paper takes the computer hardware data set maintained by UCI as an example, and compares the influence of normalization method and interval scaling method on the convergence of stochastic gradient descent by algorithm simulation. The result of study has a certain value on feature scaling.

Primal Averaging: A New Gradient Evaluation Step to Attain the Optimal Individual Convergence.

Many well-known first-order gradient methods have been extended to cope with large-scale composite problems, which often arise as a regularized empirical risk minimization in machine learning. However, their optimal convergence is attained only in terms of the weighted average of past iterative solutions. How to make the individual convergence of stochastic gradient descent (SGD) optimal, especially for strongly convex problems has now become a challenging problem in the machine learning community. On the other hand, Nesterov's recent weighted averaging strategy succeeds in achieving the optimal individual convergence of dual averaging (DA) but it fails in the basic mirror descent (MD). In this paper, a new primal averaging (PA) gradient operation step is presented, in which the gradient evaluation is imposed on the weighted average of all past iterative solutions. We prove that simply modifying the gradient operation step in MD by PA strategy suffices to recover the optimal individual rate for general convex problems. Along this line, the optimal individual rate of convergence for strongly convex problems can also be achieved by imposing the strong convexity on the gradient operation step. Furthermore, we extend PA-MD to solve regularized nonsmooth learning problems in the stochastic setting, which reveals that PA strategy is a simple yet effective extra step toward the optimal individual convergence of SGD. Several real experiments on sparse learning and SVM problems verify the correctness of our theoretical analysis.

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

We obtain an improved finite-sample guarantee on the linear convergence of stochastic gradient descent for smooth and strongly convex objectives, improving from a quadratic dependence on the conditioning $$(L/\mu )^2$$(L/μ)2 (where $$L$$L is a bound on the smoothness and $$\mu $$μ on the strong convexity) to a linear dependence on $$L/\mu $$L/μ. Furthermore, we show how reweighting the sampling distribution (i.e. importance sampling) is necessary in order to further improve convergence, and obtain a linear dependence in the average smoothness, dominating previous results. We also discuss importance sampling for SGD more broadly and show how it can improve convergence also in other scenarios. Our results are based on a connection we make between SGD and the randomized Kaczmarz algorithm, which allows us to transfer ideas between the separate bodies of literature studying each of the two methods. In particular, we recast the randomized Kaczmarz algorithm as an instance of SGD, and apply our results to prove its exponential convergence, but to the solution of a weighted least squares problem rather than the original least squares problem. We then present a modified Kaczmarz algorithm with partially biased sampling which does converge to the original least squares solution with the same exponential convergence rate.