Stochastic Conjugate Gradient Research Articles

Painless Stochastic Conjugate Gradient for Large-Scale Machine Learning.

Conjugate gradient (CG), as an effective technique to speed up gradient descent algorithms, has shown great potential and has widely been used for large-scale machine-learning problems. However, CG and its variants have not been devised for the stochastic setting, which makes them extremely unstable, and even leads to divergence when using noisy gradients. This article develops a novel class of stable stochastic CG (SCG) algorithms with a faster convergence rate via the variance-reduced technique and an adaptive step size rule in the mini-batch setting. Actually, replacing the use of a line search in the CG-type approaches which is time-consuming, or even fails for SCG, this article considers using the random stabilized Barzilai-Borwein (RSBB) method to obtain an online step size. We rigorously analyze the convergence properties of the proposed algorithms and show that the proposed algorithms attain a linear convergence rate for both the strongly convex and nonconvex settings. Also, we show that the total complexity of the proposed algorithms matches that of modern stochastic optimization algorithms under different cases. Scores of numerical experiments on machine-learning problems demonstrate that the proposed algorithms outperform state-of-the-art stochastic optimization algorithms.

Random Learning Leads to Faster Convergence in ‘Model‐Free’ ILC: With Application to MIMO Feedforward in Industrial Printing

ABSTRACTModel‐free iterative learning control (ILC) can lead to high performance by attenuating repeating disturbances completely, using dedicated experiments on the real system to replace the traditional model. The aim of this paper is to develop a fast data‐driven method for MIMO ILC that uses random learning in the form of efficient unbiased gradient estimates. This is achieved by developing a stochastic conjugate gradient algorithm, in which the search direction and optimal step size are generated using dedicated experiments. The approach is applied to MIMO automated feedforward tuning. Simulation and experimental results show that the method is superior to earlier stochastic and deterministic methods.

A Stochastic Algorithm Based on Cable-stayed Bridges Model for Stochastic Optimization in Machine Learning

This paper takes the economic cost of road design as the objective function to study the optimization of road construction cost. It is well-know that stochastic optimization is widely regarded as one of the most important and difficult problems in optimization and machine learning. It is noted that various methods for stochastic optimization problems are sensitive to the learning rate (step-size), which is usually underestimated. Therefore, two new stochastic conjugate gradient algorithms are proposed in this paper. We adapt two modified line search techniques to set the stochastic optimization. One uses the modified Armijo line search technology, the other uses the modified weak Wolfe-Powell line search technology. The new search direction has the properties of sufficient descent and trust region, which makes the establishment of global convergence of the two algorithms very understandable and clear. Finally, numerical experiments show that the two new stochastic conjugate gradient methods are competitive compared with other classical methods.

A Family of Hybrid Stochastic Conjugate Gradient Algorithms for Local and Global Minimization Problems

This paper contains two main parts, Part I and Part II, which discuss the local and global minimization problems, respectively. In Part I, a fresh conjugate gradient (CG) technique is suggested and then combined with a line-search technique to obtain a globally convergent algorithm. The finite difference approximations approach is used to compute the approximate values of the first derivative of the function f. The convergence analysis of the suggested method is established. The comparisons between the performance of the new CG method and the performance of four other CG methods demonstrate that the proposed CG method is promising and competitive for finding a local optimum point. In Part II, three formulas are designed by which a group of solutions are generated. This set of random formulas is hybridized with the globally convergent CG algorithm to obtain a hybrid stochastic conjugate gradient algorithm denoted by HSSZH. The HSSZH algorithm finds the approximate value of the global solution of a global optimization problem. Five combined stochastic conjugate gradient algorithms are constructed. The performance profiles are used to assess and compare the rendition of the family of hybrid stochastic conjugate gradient algorithms. The comparison results between our proposed HSSZH algorithm and four other hybrid stochastic conjugate gradient techniques demonstrate that the suggested HSSZH method is competitive with, and in all cases superior to, the four algorithms in terms of the efficiency, reliability and effectiveness to find the approximate solution of the global optimization problem that contains a non-convex function.

Large-scale machine learning with fast and stable stochastic conjugate gradient

In deterministic optimization, conjugate gradient (CG) type approaches are preferred with a superior convergence rate than the ordinary gradient approaches. The requirement of solving large-scale data, growing exponentially, makes recent works study the effectiveness of the CG-type approaches with stochastic approximation, especially for large-scale machine learning problems. However, it is challenging that how to incorporate the noisy gradients into CG-type approaches. In this paper, we develop a class of fast and robust stochastic conjugate gradient (SCG) type approach via using the stochastic recursive gradient algorithm (SARAH) and the hyper-gradient descent (HD) technique in the mini-batching setting. That the use of the SARAH gradient estimator makes the proposed approaches enjoy the low variance accelerates the convergence rate and saves the gradient complexity of the conventional SCG-type approach. In addition, using HD to determine the learning rate for the SCG-type approach greatly saves the computational burden, comparing with the existing literature that usually works with the line search technique in practice. We rigorously prove that the proposed approach attains a linear convergence rate for strongly convex loss functions and show that its complexity matches modern stochastic optimization approaches. Various experimental results on machine learning problems are provided to demonstrate the property and the effectiveness of the proposed approaches respectively.

A mini-batch stochastic conjugate gradient algorithm with variance reduction

A mini-batch stochastic conjugate gradient algorithm with variance reduction

Adaptive stochastic conjugate gradient for machine learning

Due to their faster convergence rate than gradient descent algorithms and less computational cost than second order algorithms, conjugate gradient (CG) algorithms have been widely used in machine learning. This paper considers conjugate gradient in the mini-batch setting. Concretely, we propose a stable adaptive stochastic conjugate gradient (SCG) algorithm via incorporating both the stochastic recursive gradient algorithm (SARAH) and second order information into the CG-type algorithm. Unlike most of existing CG algorithms that spend a lot of time in determining the step size by using line search and may fail in stochastic optimization, the proposed algorithms use a local quadratic model to estimate the step size sequence, but do not require computing the Hessian information, which make the proposed algorithms attain a low computational cost as first-order algorithms. We establish the linear convergence rate of a class of SCG algorithms, when the loss function is the strongly convex. Moreover, we show that the complexity of the proposed algorithm matches modern stochastic optimization algorithms. As a by-product, we develop a practical variant of the proposed algorithm by setting a stopping criterion for the number of inner loop iterations. Various numerical experiments on machine learning problems demonstrate the efficiency of the proposed algorithms.

Stochastic Conjugate Gradient Algorithm With Variance Reduction.

Conjugate gradient (CG) methods are a class of important methods for solving linear equations and nonlinear optimization problems. In this paper, we propose a new stochastic CG algorithm with variance reduction1 and we prove its linear convergence with the Fletcher and Reeves method for strongly convex and smooth functions. We experimentally demonstrate that the CG with variance reduction algorithm converges faster than its counterparts for four learning models, which may be convex, nonconvex or nonsmooth. In addition, its area under the curve performance on six large-scale data sets is comparable to that of the LIBLINEAR solver for the L2 -regularized L2 -loss but with a significant improvement in computational efficiency.1CGVR algorithm is available on github: https://github.com/xbjin/cgvr.

Building neural network language model with POS-based negative sampling and stochastic conjugate gradient descent

Traditional statistical language model is a probability distribution over sequences of words. It has the problem of curse of dimensionality incurred by the exponentially increasing number of possible sequences of words in training text. To solve this issue, neural network language models are proposed by representing words in a distributed way. Due to computation cost on updating a large number of word vectors’ gradients, neural network model needs much training time to converge. To alleviate this problem, in this paper, we propose a gradient descent algorithm based on stochastic conjugate gradient to accelerate the convergence of the neural network’s parameters. To improve the performance of the neural language model, we also propose a negative sampling algorithm based on POS (part of speech) tagging, which can optimize the negative sampling process and improve the quality of the final language model. A novel evaluation model is also used with perplexity to demonstrate the performance of the improved language model. Experiment results prove the effectiveness of our novel methods.

Plane-wave least-squares reverse time migration with a preconditioned stochastic conjugate gradient method

This study derives a preconditioned stochastic conjugate gradient (CG) method that combines stochastic optimization with singular spectrum analysis (SSA) denoising to improve the efficiency and image quality of plane-wave least-squares reverse time migration (PLSRTM). This method reduces the computational costs of PLSRTM by applying a controlled group-sampling method to a sufficiently large number of plane-wave sections and accelerates the convergence using a hybrid of stochastic descent (SD) iteration and CG iteration. However, the group sampling also produces aliasing artifacts in the migration results. We use SSA denoising as a preconditioner to remove the artifacts. Moreover, we implement the preconditioning on the take-off angle-domain common-image gathers (CIGs) for better results. We conduct numerical tests using the Marmousi model and Sigsbee2A salt model and compare the results of this method with those of the SD method and the CG method. The results demonstrate that our method efficiently eliminates the artifacts and produces high-quality images and CIGs.

Power amplifier behavioral model adaptive pruning using conjugate gradient‐based greedy algorithm

Starting from the greedy theory, this paper presents a novel adaptive greedy scheme for the power amplifier (PA) behavioral model adaptive pruning. The proposed scheme incorporates the stochastic conjugate gradient (SCG) principle into the subspace pursuit (SP) greedy algorithm, which can considerably offer improved tracking capabilities and faster convergence compared to other anterior adaptive greedy algorithms. Compared to conventional nonsparse methods, simulation results show that the proposed scheme can efficiently reduce the model order and computational complexity but almost have the comparable model performance with the full model. © 2017 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

Low‐complexity robust adaptive beamforming algorithms exploiting shrinkage for mismatch estimation

This study proposes low-complexity robust adaptive beamforming (RAB) techniques based on shrinkage methods. The authors first review a low-complexity shrinkage-based mismatch estimation batch algorithm to estimate the desired signal steering vector mismatch, in which the interference-plus-noise covariance matrix is also estimated by a recursive matrix shrinkage method. Then they develop low-complexity adaptive recursive versions of stochastic gradient and conjugate gradient to update the beamforming weights, resulting in low-cost robust adaptive algorithms. An analysis of the effect of shrinkage on the estimation procedure is developed along with a computational complexity study of the proposed and existing algorithms. Simulations are conducted in local scattering scenarios and comparisons to existing RAB techniques are provided.

Least-squares seismic inversion with stochastic conjugate gradient method

With the development of computational power, there has been an increased focus on data-fitting related seismic inversion techniques for high fidelity seismic velocity model and image, such as full-waveform inversion and least squares migration. However, though more advanced than conventional methods, these data fitting methods can be very expensive in terms of computational cost. Recently, various techniques to optimize these data-fitting seismic inversion problems have been implemented to cater for the industrial need for much improved efficiency. In this study, we propose a general stochastic conjugate gradient method for these data-fitting related inverse problems. We first prescribe the basic theory of our method and then give synthetic examples. Our numerical experiments illustrate the potential of this method for large-size seismic inversion application.

Digital Predistortion Using Stochastic Conjugate Gradient Method

This paper is concerned with digital predistortion for linearization of RF high power amplifiers (HPAs). An iterative stochastic conjugate gradient (SCG) method is used in computing the predistorter in the digital predistortion. We also propose an adaptive scheme for selecting basis functions in the SCG computations. The adaptive scheme with SCG has the advantage of reducing the complexity and, at the same time, increasing the convergence rate and stability of iteration. We present some results of using the SCG in a hardware platform with a solid state HPA.

A stochastic conjugate gradient method for the approximation of functions

A stochastic conjugate gradient method for the approximation of a function is proposed. The proposed method avoids computing and storing the covariance matrix in the normal equations for the least squares solution. In addition, the method performs the conjugate gradient steps by using an inner product that is based on stochastic sampling. Theoretical analysis shows that the method is convergent in probability. The method has applications in such fields as predistortion for the linearization of power amplifiers.