Barzilai-Borwein Method Research Articles

Painless Stochastic Conjugate Gradient for Large-Scale Machine Learning.

Conjugate gradient (CG), as an effective technique to speed up gradient descent algorithms, has shown great potential and has widely been used for large-scale machine-learning problems. However, CG and its variants have not been devised for the stochastic setting, which makes them extremely unstable, and even leads to divergence when using noisy gradients. This article develops a novel class of stable stochastic CG (SCG) algorithms with a faster convergence rate via the variance-reduced technique and an adaptive step size rule in the mini-batch setting. Actually, replacing the use of a line search in the CG-type approaches which is time-consuming, or even fails for SCG, this article considers using the random stabilized Barzilai-Borwein (RSBB) method to obtain an online step size. We rigorously analyze the convergence properties of the proposed algorithms and show that the proposed algorithms attain a linear convergence rate for both the strongly convex and nonconvex settings. Also, we show that the total complexity of the proposed algorithms matches that of modern stochastic optimization algorithms under different cases. Scores of numerical experiments on machine-learning problems demonstrate that the proposed algorithms outperform state-of-the-art stochastic optimization algorithms.

Riemannian SVRG Using Barzilai–Borwein Method as Second-Order Approximation for Federated Learning

Riemannian SVRG Using Barzilai–Borwein Method as Second-Order Approximation for Federated Learning

Stabilized BB projection algorithm for large-scale convex constrained nonlinear monotone equations to signal and image processing problems

The Barzilai–Borwein (BB) method is a popular and efficient gradient method for solving large-scale unconstrained optimization problems. In general, it converges much faster than the steepest descent (Cauchy) method. However, it may not converge, even when the objective function is strongly convex. To overcome this, by virtue of bounding the distance between sequential iterates, [Burdakov et al. (2019)] proposed a stabilized BB method. In this paper, by combining the stabilized BB method with a projection approach, we develop a stabilized BB projection (SBBP) method to solve nonlinear monotone equations with convex constraints. SBBP does not involve computing matrices proving the usefulness for solving large-scale problems. Its global convergence and Q-linear convergence rate are also established. We report numerical experiments on recovering sparse signals and restoring blurred images arising from compressive sensing, demonstrating the usefulness of SBBP.

Nonmonotone variable metric Barzilai-Borwein method for composite minimization problem

<abstract><p>In this study, we develop a nonmonotone variable metric Barzilai-Borwein method for minimizing the sum of a smooth function and a convex, possibly nondifferentiable, function. At each step, the descent direction is obtained by taking the difference between the minimizer of the scaling proximal function and the current iteration point. An adaptive nonmonotone line search is proposed for determining the step length along this direction. We also show that the limit point of the iterates sequence is a stationary point. Numerical results with parallel magnetic resonance imaging, Poisson, and Cauchy noise deblurring demonstrate the effectiveness of the new algorithm.</p></abstract>

A Modified Inexact SARAH Algorithm with Stabilized Barzilai-Borwein Step-Size in Machine learning

The Inexact SARAH (iSARAH) algorithm as a variant of SARAH algorithm, which does not require computation of the exact gradient, can be applied to solving general expectation minimization problems rather than only finite sum problems. The performance of iSARAH algorithm is frequently affected by the step size selection, and how to choose an appropriate step size is still a worthwhile problem for study. In this paper, we propose to use the stabilized Barzilai-Borwein (SBB) method to automatically compute step size for iSARAH algorithm, which leads to a new algorithm called iSARAH-SBB. By introducing this adaptive step size in the design of the new algorithm, iSARAH-SBB can take better advantages of both iSARAH and SBB methods. We analyse the convergence rate and complexity of the modified algorithm under the usual assumptions. Numerical experimental results on standard data sets demonstrate the feasibility and effectiveness of our proposed algorithm.

Adaptive learning rate optimization algorithms with dynamic bound based on Barzilai-Borwein method

The training effect of the neural network model is directly influenced by optimization algorithms. The Barzilai-Borwein(BB) method is used in the stochastic gradient descent (SGD) and other deep learning optimization algorithms because of its outstanding performance in terms of convergence speed. In order to improve the stability of BB step size and the output quality of network model in deep learning, this paper presents two improved optimization algorithms based on the BB method: BBbound and AdaBBbound. BBbound reduces the floating BB step size by generating the upper bound of the step in the current iteration. It avoids the occasional occurrence of long steps; AdaBBbound is an adaptive gradient method based on BB method with dynamic bound. The algorithm has a fast convergence effect in the early stage, then calculates learning rate and smoothly transitions to SGD by setting the better conditions of the BB method. We analyzed the performance of the two algorithms based on the initial conditions and the learning rate change curve. Meanwhile, we tested our algorithms on popular network models such as ResNet and DenseNet. The results showed that the new optimization algorithms have achieved high stability and significantly improved the performance of the network model.

A harmonic framework for stepsize selection in gradient methods

We study the use of inverse harmonic Rayleigh quotients with target for the stepsize selection in gradient methods for nonlinear unconstrained optimization problems. This not only provides an elegant and flexible framework to parametrize and reinterpret existing stepsize schemes, but it also gives inspiration for new flexible and tunable families of steplengths. In particular, we analyze and extend the adaptive Barzilai–Borwein method to a new family of stepsizes. While this family exploits negative values for the target, we also consider positive targets. We present a convergence analysis for quadratic problems extending results by Dai and Liao (IMA J Numer Anal 22(1):1–10, 2002), and carry out experiments outlining the potential of the approaches.

Distribution preserving-based deep semi-NMF for data representation

The semi-nonnegative matrix factorization (Semi-NMF) is a promising soft K-means clustering technique. Deep Semi-NMF, which stacks one-layer Semi-NMF into multi-layer, is able to learn the hierarchical projections and can obtain the deep hidden representations according to the unknown attributes of the given data. On the other hand, the inherent structure of the each data cluster can be described by the distribution of the intraclass data. Then one hopes to learn the new deep hidden representations which can preserve the intrinsic structures embedded in the original data space perfectly. Here seamlessly integrating the benefits of the Deep Semi-NMF and the distribution preserving strategy, we propose a novel distribution preserving-based deep semi-nonnegative matrix factorization method (DPNMF) to achieve this goal. In DPNMF, by maintaining the consistency of two distributions that can approximate the manifold structures, we can seek the deep hidden features which reveal the original intrinsic structures. As a result, the manifold structures in the raw data are well preserved in the new feature space. We also devise an adaptive projected Barzilai-Borwein method to optimize the proposed constrained objective function efficiently. The experimental results on the several real-world datasets show that the proposed DPNMF can achieve advantageous clustering performance in terms of accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI).

Second-Order Step-Size Tuning of SGD for Non-Convex Optimization

In view of a direct and simple improvement of vanilla SGD, this paper presents a fine-tuning of its step-sizes in the mini-batch case. For doing so, one estimates curvature, based on a local quadratic model and using only noisy gradient approximations. One obtains a new stochastic first-order method (Step-Tuned SGD), enhanced by second-order information, which can be seen as a stochastic version of the classical Barzilai-Borwein method. Our theoretical results ensure almost sure convergence to the critical set and we provide convergence rates. Experiments on deep residual network training illustrate the favorable properties of our approach. For such networks we observe, during training, both a sudden drop of the loss and an improvement of test accuracy at medium stages, yielding better results than SGD, RMSprop, or ADAM.

On the acceleration of the Barzilai–Borwein method

The Barzilai–Borwein (BB) gradient method is efficient for solving large-scale unconstrained problems to modest accuracy due to its ingenious stepsize which generally yields nonmonotone behavior. In this paper, we propose a new stepsize to accelerate the BB method by requiring finite termination for minimizing the two-dimensional strongly convex quadratic function. Based on this new stepsize, we develop an efficient gradient method for quadratic optimization which adaptively takes the nonmonotone BB stepsizes and certain monotone stepsizes. Two variants using retard stepsizes associated with the new stepsize are also presented. Numerical experiments show that our strategies of properly inserting monotone gradient steps into the nonmonotone BB method could significantly improve its performance and our new methods are competitive with the most successful gradient descent methods developed in the recent literature.

Total Variation Constrained Graph Manifold Learning Strategy for Cerenkov Luminescence Tomography.

Harnessing the power and flexibility of radiolabeled molecules, Cerenkov luminescence tomography (CLT) provides a novel technique for non-invasive visualisation and quantification of viable tumour cells in a living organism. However, owing to the photon scattering effect and the ill-posed inverse problem, CLT still suffers from insufficient spatial resolution and shape recovery in various preclinical applications. In this study, we proposed a total variation constrained graph manifold learning (TV-GML) strategy for achieving accurate spatial location, dual-source resolution, and tumour morphology. TV-GML integrates the isotropic total variation term and dynamic graph Laplacian constraint to make a trade-off between edge preservation and piecewise smooth region reconstruction. Meanwhile, the tetrahedral mesh-Cartesian grid pair method based on the k-nearest neighbour, and the adaptive and composite Barzilai-Borwein method, were proposed to ensure global super linear convergence of the solution of TV-GML. The comparison results of both simulation experiments and in vivo experiments further indicated that TV-GML achieved superior reconstruction performance in terms of location accuracy, dual-source resolution, shape recovery capability, robustness, and in vivo practicability. Significance: We believe that this novel method will be beneficial to the application of CLT for quantitative analysis and morphological observation of various preclinical applications and facilitate the development of the theory of solving inverse problem.

On the Asymptotic Convergence and Acceleration of Gradient Methods

We consider the asymptotic behavior of a family of gradient methods, which include the steepest descent and minimal gradient methods as special instances. It is proved that each method in the family will asymptotically zigzag between two directions. Asymptotic convergence results of the objective value, gradient norm, and stepsize are presented as well. To accelerate the family of gradient methods, we further exploit spectral properties of stepsizes to break the zigzagging pattern. In particular, a new stepsize is derived by imposing finite termination on minimizing two-dimensional strictly convex quadratic function. It is shown that, for the general quadratic function, the proposed stepsize asymptotically converges to the reciprocal of the largest eigenvalue of the Hessian. Furthermore, based on this spectral property, we propose a periodic gradient method by incorporating the Barzilai-Borwein method. Numerical comparisons with some recent successful gradient methods show that our new method is very promising.

On the convergence and mesh-independent property of the Barzilai–Borwein method for PDE-constrained optimization

Abstract Aiming at optimization problems governed by partial differential equations (PDEs), local R-linear convergence of the Barzilai–Borwein (BB) method for a class of twice continuously Fréchet-differentiable functions is proven. Relying on this result, the mesh-independent principle for the BB-method is investigated. The applicability of the theoretical results is demonstrated for two different types of PDE-constrained optimization problems. Numerical experiments are given, which illustrate the theoretical results.

An optimal control method for time-dependent fluid-structure interaction problems

In this article, we derive an adjoint fluid-structure interaction (FSI) system in an arbitrary Lagrangian-Eulerian (ALE) framework, based upon a one-field finite element method. A key feature of this approach is that the interface condition is automatically satisfied and the problem size is reduced since we only solve for one velocity field for both the primary and adjoint system. A velocity (and/or displacement)-matching optimisation problem is considered by controlling a distributed force. The optimisation problem is solved using a gradient descent method, and a stabilised Barzilai-Borwein method is adopted to accelerate the convergence, which does not need additional evaluations of the objective functional. The proposed control method is validated and assessed against a series of static and dynamic benchmark FSI problems, before being applied successfully to solve a highly challenging FSI control problem.

Concurrent Modified Constant Modulus Algorithm and Decision Directed Scheme With Barzilai-Borwein Method.

At present, in robot technology, remote control of robot is realized by wireless communication technology, and data anti-interference in wireless channel becomes a very important part. Any wireless communication system has an inherent multi-path propagation problem, which leads to the expansion of generated symbols on a time scale, resulting in symbol overlap and Inter-symbol Interference (ISI). ISI in the signal must be removed and the signal restores to its original state at the time of transmission or becomes as close to it as possible. Blind equalization is a popular equalization method for recovering transmitted symbols of superimposed noise without any pilot signal. In this work, we propose a concurrent modified constant modulus algorithm (MCMA) and the decision-directed scheme (DDS) with the Barzilai-Borwein (BB) method for the purpose of blind equalization of wireless communications systems (WCS). The BB method, which is two-step gradient method, has been widely employed to solve multidimensional unconstrained optimization problems. Considering the similarity of equalization process and optimization process, the proposed algorithm combines existing blind equalization algorithm and Barzilai-Borwein method, and concurrently operates a MCMA equalizer and a DD equalizer. After that, it modifies the DD equalizer's step size (SS) by the BB method. Theoretical investigation was involved and it demonstrated rapid convergence and improved equalization performance of the proposed algorithm compared with the original one. Additionally, the simulation results were consistent with the proposed technique.

An accelerated first-order method with complexity analysis for solving cubic regularization subproblems

We propose a first-order method to solve the cubic regularization subproblem (CRS) based on a novel reformulation. The reformulation is a constrained convex optimization problem whose feasible region admits an easily computable projection. Our reformulation requires computing the minimum eigenvalue of the Hessian. To avoid the expensive computation of the exact minimum eigenvalue, we develop a surrogate problem to the reformulation where the exact minimum eigenvalue is replaced with an approximate one. We then apply first-order methods such as the Nesterov’s accelerated projected gradient method (APG) and projected Barzilai-Borwein method to solve the surrogate problem. As our main theoretical contribution, we show that when an $$\epsilon$$ -approximate minimum eigenvalue is computed by the Lanczos method and the surrogate problem is approximately solved by APG, our approach returns an $$\epsilon$$ -approximate solution to CRS in $${\tilde{O}}(\epsilon ^{-1/2})$$ matrix-vector multiplications (where $${\tilde{O}}(\cdot )$$ hides the logarithmic factors). Numerical experiments show that our methods are comparable to and outperform the Krylov subspace method in the easy and hard cases, respectively. We further implement our methods as subproblem solvers of adaptive cubic regularization methods, and numerical results show that our algorithms are comparable to the state-of-the-art algorithms.

Distributed stochastic principal component analysis using stabilized Barzilai-Borwein step-size for data compression with WSN

The popularity of diverse IoT-based applications and services continuously generating tremendous amount of data has revealed the significance of data compression (DC). Principal component analysis (PCA) is one of the most commonly employed algorithms for DC. However, when dealing with large-scale matrices, the standard PCA takes a very long time and requires a lot of memory. Therefore, this paper presents a novel distributed stochastic PCA algorithm (DSPCA) for hierarchical sensor network based on gradient-based adaptive PCA (GA-PCA), where the standard PCA is reformulated as a single-pass stochastic setting to find the direction of approximate maximal variance. The step-size in each iteration is obtained by incorporating the stabilized Barzilai-Borwein method with the gradient optimization. This enables DSPCA to be processed with low computational complexity while maintaining a high convergence speed. Computer simulation with two types of datasets displays that the proposed scheme consistently outperforms the representative DC schemes in terms of reconstruction accuracy of original data and explained variance.

AFISTA: Accelerated FISTA for sparse signal recovery and compressive sensing

This paper presents a new fast iterative shrinkage-thresholding algorithm, termed AFISTA. The essential idea is to improve the convergence rate of FISTA using a new continuation strategy leading to a less number of iterations compared to FISTA. The convergence theorem of the AFISTA is proposed. In order to further accelerate the AFISTA method, it is equipped with the Barzilai-Borwein (BB) method. Also, for applications with orthogonal sensing matrix A, we proposed a specialized version of the AFISTA method. AFISTA is tailored for solving the basis pursuit problem which can be applied successfully on a variety of problems arising in signal and image processing issues such as sparse signal recovery, signal and image denoising, image restoration, and compressive sensing. To show the efficiency of the method, we compare our results with generalizations of linearized Bregman and fixed - point continuation (FPC) methods in sparse signal recovery applications, with split Bregman method in compressive sensing for sparse MRI and with Gradient projection for sparse reconstruction (GPSR) method in image deconvolution. Numerical results demonstrate that AFISTA overcomes all of the compared methods in convergence rate and some of them in both convergence rate and quality of reconstructed results.

Accelerating mini-batch SARAH by step size rules

StochAstic Recursive grAdient algoritHm (SARAH), originally proposed for convex optimization and also proven to be effective for general nonconvex optimization, has received great attention because of its simple recursive framework for updating stochastic gradient estimates. The performance of SARAH significantly depends on the choice of the step size sequence. However, SARAH and its variants often manually select a best-tuned step size, which is time consuming in practice. Motivated by this gap, we propose a variant of the Barzilai-Borwein (BB) method, referred to as the Random Barzilai-Borwein (RBB) method, to determine the step size for SARAH in the mini-batch setting, leading to a new SARAH method: MB-SARAH-RBB. We prove that MB-SARAH-RBB converges linearly in expectation for strongly convex objective functions. Moreover, we analyze the gradient complexity of MB-SARAH-RBB and show that it is better than the original method. To further confirm the efficacy of the RBB method, we propose the MB-SARAH+-RBB method, by incorporating it into the MB-SARAH + method. Numerical experiments on standard data sets indicate that our proposed methods outperform or match state-of-the-art algorithms.

The Barzilai–Borwein Method for distributed optimization over unbalanced directed networks

This paper studies optimization problems over multi-agent systems, in which all agents cooperatively minimize a global objective function expressed as a sum of local cost functions. Each agent in the systems uses only local computation and communication in the overall process without leaking their private information. Based on the Barzilai–Borwein (BB) method and multi-consensus inner loops, a distributed algorithm with the availability of larger step-sizes and accelerated convergence, named as ADBB, is proposed. Moreover, owing to the employment of only row-stochastic weight matrices, ADBB can resolve the optimization problems over unbalanced directed networks without requiring the knowledge of neighbors’ out-degree for each agent. Via establishing contraction relationships between the consensus error, the optimality gap, and the gradient tracking error, ADBB is theoretically proved to converge linearly to the global optimal solution. A real-world data set is used in simulations to validate the correctness of the theoretical analysis.