Barzilai-Borwein Step Size Research Articles

Adaptive learning rate algorithms based on the improved Barzilai–Borwein method

Objective:The Barzilai–Borwein(BB) method is essential in solving unconstrained optimization problems. The momentum method accelerates optimization algorithms with exponentially weighted moving average. In order to design reliable deep learning optimization algorithms, this paper proposes applying the BB method in four variants to the optimization algorithm of deep learning. Findings:The momentum method generates the BB step size under different step range limits. We also apply the momentum method and its variants to the stochastic gradient descent with the BB step size. Novelty:The algorithm’s robustness has been demonstrated through experiments on the initial learning rate and random seeds. The algorithm’s sensitivity is tested by choosing different momentum factors until a suitable momentum factor is found. Moreover, we compare our algorithms with popular algorithms in various neural networks. The results show that the new algorithms improve the efficiency of the BB step size in deep learning and provide a variety of optimization algorithm choices.

A hybrid complex spectral conjugate gradient learning algorithm for complex-valued data processing

Complex-valued neural networks (CVNNs) have become a powerful modelling tool for complex-valued data processing. Because most of the critical points of CVNNs are saddle points, the gradient-based learning algorithms for CVNNs enjoy more chances to reach the global minima while suffering from slow convergence. To this end, we propose a hybrid complex spectral conjugate gradient learning algorithm for fast training CVNNs in this paper. The proposed algorithm combines the scaled negative gradient with a Barzilai–Borwein stepsize and an optimized conjugate term to define a new training direction, thus providing an accurate approximation of the second-order curvature of the objective function. The complex Wolfe conditions are employed to adaptively determine the optimal training stepsize. Under mild conditions, the descent property of the training direction and the convergence of the proposed algorithm are theoretically established. Simulation results on a number of benchmark complex-valued data processing problems demonstrate the efficiency of the proposed algorithm.

Modified nonmonotonic projection Barzilai-Borwein gradient method for nonnegative matrix factorization

<p>In this paper, an active set recognition technique is suggested, and then a modified nonmonotonic line search rule is presented to enhance the efficiency of the nonmonotonic line search rule, in which we introduce a new parameter formula to attempt to control the nonmonotonic degree of the line search, and thus improve the chance of discovering the global minimum. By using a modified linear search and an active set recognition technique, a global convergence gradient solution for nonnegative matrix factorization (NMF) based on an alternating nonnegative least squares framework is proposed. We used a Barzilai-Borwein step size and greater step-size tactics to speed up the convergence. Finally, a large number of numerical experiments were carried out on synthetic and image datasets, and the results showed that our presented method was effective in calculating the speed and solution quality.</p>

Distributed Heavy-Ball Over Time-Varying Digraphs With Barzilai-Borwein Step Sizes

Distributed Heavy-Ball Over Time-Varying Digraphs With Barzilai-Borwein Step Sizes

A mini-batch algorithm for large-scale learning problems with adaptive step size

The step size selection in stochastic optimization methods is a crucial aspect that holds significant importance in theoretical analysis and practical applications. We propose two stochastic optimization methods based on the competitive Barzilai-Borwein (BB) step size in both the inner and outer loops of the mini-batch semi-stochastic gradient descent (mS2GD) algorithm. The competitive BB step size is automatically updated using the latest and most accurate secant equation. We introduce two algorithms: mS2GD-CBB, which updates the step size in the outer loop, and mS2GD-RCBB, which updates the step size in the inner loop. We evaluate the performance of the proposed algorithms on the classical optimization problem and compare them against existing methods. Experimental results demonstrate that the methods exhibit favorable convergence properties and can effectively handle the challenges of big data arising from the fields of signal processing, statistics, and machine learning. The methods offer enhanced adaptability, flexibility, and exploration capabilities, making them promising approaches for a wide range of optimization problems.

On Kahan's automatic step-size control and an anadromic gradient descent iteration

Accelerating the basic gradient iteration is a critical issue in numerical optimization and deep-learning. Recently, William Kahan [Kahan, Automatic Step-Size Control for Minimization Iterations, Technical report, University of California, Berkeley CA, USA, 2019] proposed automatic step-size strategies for the gradient descent iteration without explicitly using any prior information of the Hessian; moreover, a new momentum-based gradient acceleration method, namely, an Anadromic Gradient Descent (AGD) iteration, was proposed. Besides the capability of accelerating the basic gradient descent iteration, AGD enables the iteration to return to the past iterates by just reversing steps in sign and order in the same updating formulation. Numerical performances of the automatic step-size strategies and AGD are claimed to be favourable. In this paper, for the quadratic model, through a revisit of some classical momentum-based gradient methods, we perform new analysis on the convergence and their optimal hyper-parameters. We also investigate the convergence behaviour of AGD with the optimal hyper-parameters and connect one Kahan's automatic step-size scheme with the long Barzilai–Borwein step-size. Numerical results are presented to reflect the theoretical convergence behaviours and demonstrate the practical performances of various momentum-based gradient methods.

Adaptive learning rate optimization algorithms with dynamic bound based on Barzilai-Borwein method

The training effect of the neural network model is directly influenced by optimization algorithms. The Barzilai-Borwein(BB) method is used in the stochastic gradient descent (SGD) and other deep learning optimization algorithms because of its outstanding performance in terms of convergence speed. In order to improve the stability of BB step size and the output quality of network model in deep learning, this paper presents two improved optimization algorithms based on the BB method: BBbound and AdaBBbound. BBbound reduces the floating BB step size by generating the upper bound of the step in the current iteration. It avoids the occasional occurrence of long steps; AdaBBbound is an adaptive gradient method based on BB method with dynamic bound. The algorithm has a fast convergence effect in the early stage, then calculates learning rate and smoothly transitions to SGD by setting the better conditions of the BB method. We analyzed the performance of the two algorithms based on the initial conditions and the learning rate change curve. Meanwhile, we tested our algorithms on popular network models such as ResNet and DenseNet. The results showed that the new optimization algorithms have achieved high stability and significantly improved the performance of the network model.

Trace minimization method via penalty for linear response eigenvalue problems

<p style='text-indent:20px;'>In various applications, such as the computation of energy excitation states of electrons and molecules, and the analysis of interstellar clouds, the linear response eigenvalue problem, which is a special type of the Hamiltonian eigenvalue problem, is frequently encountered. However, traditional eigensolvers may not be applicable to this problem owing to its inherently large scale. In fact, we are usually more interested in computing some of the smallest positive eigenvalues. To this end, a trace minimum principle optimization model with orthogonality constraint has been proposed. On this basis, we propose an unconstrained surrogate model called trace minimization via penalty, and we establish its equivalence with the original constrained model, provided that the penalty parameter is larger than a certain threshold. By avoiding the orthogonality constraint, we can use a gradient-type method to solve this model. Specifically, we use the gradient descent method with Barzilai–Borwein step size. Moreover, we develop a restarting strategy for the proposed algorithm whereby higher accuracy and faster convergence can be achieved. This is verified by preliminary experimental results.</p>

A New Quasi-Newton Method with PCG Method for Nonlinear Optimization Problems

The major stationary iterative method used to solve nonlinear optimization problems is the quasi-Newton (QN) method. Symmetric Rank-One (SR1) is a method in the quasi-Newton family. This algorithm converges towards the true Hessian fast and has computational advantages for sparse or partially separable problems [1]. Thus, investigating the efficiency of the SR1 algorithm is significant. It's possible that the matrix generated by SR1 update won't always be positive. The denominator may also vanish or become zero. To overcome the drawbacks of the SR1 method, resulting in better performance than the standard SR1 method, in this work, we derive a new vector <img src=image/13429423_01.gif> depending on the Barzilai-Borwein step size to obtain a new SR1 method. Then using this updating formula with preconditioning conjugate gradient (PCG) method is presented. With the aid of inexact line search procedure by strong Wolfe conditions, the new SR1 method is proposed and its performance is evaluated in comparison to the conventional SR1 method. It is proven that the updated matrix of the new SR1 method, <img src=image/13429423_02.gif>, is symmetric matrix and positive definite matrix, given <img src=image/13429423_03.gif> is initialized to identity matrix. In this study, the proposed method solved 13 problems effectively in terms of the number of iterations (NI) and the number of function evaluations (NF). Regarding NF, the new SR1 method also outperformed the classic SR1 method. The proposed method is shown to be more efficient in solving relatively large-scale problems (5,000 variables) compared to the original method. From the numerical results, the proposed method turned out to be significantly faster, effective and suitable for solving large dimension nonlinear equations.

Modeling the capacity of multimodal and intermodal urban transportation networks that incorporate emerging travel modes

Given the increasing prevalence of new urban transport modes such as ridesharing, e-hailing, and combined transport, it is essential to evaluate their effects on the capacity of transportation networks. Hence, this paper develops a novel transportation network capacity model to capture the travel behaviors of inter-multimodal mobility in an urban transportation system that incorporates emerging travel modes. The novel model is formulated as a bi-level programming problem, in which the lower-level model is a combined modal split and traffic assignment (CMSTA) problem based on mathematical programming. The CMSTA problem adopts the cross-nested logit model to account for intermodal travel behavior in the modal split phase and the path-sized logit model to account for route overlap in the traffic assignment phase. Moreover, the logit-based trip distribution model is used to capture the dispatch of the e-hailing traffic flow and the matching of ridesharing drivers with passengers. Besides, we consider flow interactions (e.g., cars and buses sharing the same link) in the road network. We customize a solution framework for solving this novel model that adopts the recently developed fast path-based algorithm with the Barzilai–Borwein stepsize strategy to efficiently solve the CMSTA problem, and derive a sensitivity analysis-based (SAB) algorithm to solve the entire bi-level programming problem. The effectiveness of the novel model is verified in numerical experiments that demonstrate the effects of intermodal transportation, e-hailing, and ridesharing on the capacity of a multimodal transportation network.

A new inexact stochastic recursive gradient descent algorithm with Barzilai–Borwein step size in machine learning

A new inexact stochastic recursive gradient descent algorithm with Barzilai–Borwein step size in machine learning

A first order reliability method based on hybrid conjugate approach with adaptive Barzilai–Borwein steps

In the first order reliability method (FORM), the Hasofer–Lind and Rackwitz–Flessler (HL–RF) algorithm sometimes encounters numerical instability problems due to the highly nonlinear limit state function (LSF). In this paper, an improved HL–RF algorithm introducing the hybrid conjugate gradient method with adaptive Barzilai–Borwein step sizes is developed to enhance the robustness and efficiency of the original HL–RF method. The proposed algorithm is composed of two stages, the steepest descent method is performed in the first stage to move to the vicinity of the most probable failure point (MPFP) and provide a good initial location for the second stage. Along with the hybrid conjugate search direction defined by the descent direction of the non-differential merit function, the second stage quantizes the adaptive Barzilai–Borwein step sizes to accelerate the process of locating the final MPFP under the nonmonotone line search rule. Eight illustrative examples with nonlinear LSFs are analyzed in detail to validate the performance of the proposed algorithm compared with other first order reliability methods. The results indicate that the proposed algorithm is not only computationally efficient but also robust in terms of convergence, especially for those problems with super nonlinear LSFs and with nonlinear LSFs involving high-frequency noise terms.

A gradient method exploiting the two dimensional quadratic termination property

The quadratic termination property is important to the efficiency of gradient methods. We consider equipping a family of gradient methods, where the stepsize is given by the ratio of two norms, with two dimensional quadratic termination. Such a desired property is achieved by cooperating with a new stepsize which is derived by maximizing the stepsize of the considered family in the next iteration. It is proved that each method in the family will asymptotically alternate in a two dimensional subspace spanned by the eigenvectors corresponding to the largest and smallest eigenvalues. Based on this asymptotic behavior, we show that the new stepsize converges to the reciprocal of the largest eigenvalue of the Hessian. Furthermore, by adaptively taking the long Barzilai–Borwein stepsize and reusing the new stepsize with retard, we propose an efficient gradient method for unconstrained quadratic optimization. We prove that the new method is R-linearly convergent with a rate of \(1-1/\kappa\), where \(\kappa\) is the condition number of Hessian. Numerical experiments show the efficiency of our proposed method.

Distributed and scalable optimization for robust proton treatment planning.

The importance of robust proton treatment planning to mitigate the impact of uncertainty is well understood. However, its computational cost grows with the number of uncertainty scenarios, prolonging the treatment planning process. We developed a fast and scalable distributed optimization platform that parallelizes the robust proton treatment plan computation over the uncertaintyscenarios. We modeled the robust proton treatment planning problem as a weighted least-squares problem. To solve it, we employed an optimization technique called the alternating direction method of multipliers with Barzilai-Borwein step size (ADMM-BB). We reformulated the problem in such a way as to split the main problem into smaller subproblems, one for each proton therapy uncertainty scenario. The subproblems can be solved in parallel, allowing the computational load to be distributed across multiple processors (e.g., CPU threads/cores). We evaluated ADMM-BB on four head-and-neck proton therapy patients, each with 13 scenarios accounting for 3 mm setup and 3.5% range uncertainties. We then compared the performance of ADMM-BB with projected gradient descent (PGD) applied to the sameproblem. For each patient, ADMM-BB generated a robust proton treatment plan that satisfied all clinical criteria with comparable or better dosimetric quality than the plan generated by PGD. However, ADMM-BB's total runtime averaged about 6 to 7 times faster. This speedup increased with the number ofscenarios. ADMM-BB is a powerful distributed optimization method that leverages parallel processing platforms, such as multicore CPUs, GPUs, and cloud servers, to accelerate the computationally intensive work of robust proton treatment planning. This results in (1) a shorter treatment planning process and (2) the ability to consider more uncertainty scenarios, which improves planquality.

An accelerated algorithm for distributed optimization with Barzilai-Borwein step sizes

Distributed optimization algorithms are widely applied in distributed systems where the agents cooperate with each other to find the minimal solution of the problem over a connected network. In this paper, we introduce the adaptive step sizes that are computed automatically with variables and gradients of the last two iterates, which are independent of the underlying network topology and the function property. Moreover, we propose an accelerated algorithm based on the dynamic average consensus approach and the Barzilai-Borwein step sizes and further demonstrate the geometric convergence of the algorithm for the smooth and strongly convex functions. Finally, the numerical experiments are provided to validate the theoretical results and show the efficacy of the proposed algorithm.

Multimodal Urban Transportation Network Capacity Model Considering Intermodal Transportation

Transportation network capacity is an important indicator from measuring the performance of a transportation system. Previous studies on the capacity of multimodal networks have not considered transfers between different transportation modes. In fact, because of the diversified development of the urban transportation system, travelers are no longer limited to a single transportation mode in daily travel. This paper thus proposes an improved network capacity model considering intermodal transportation in urban multimodal transportation systems. The proposed network capacity model is formulated as a bi-level programming problem, in which the lower-level model is a combined modal split and traffic assignment (CMSTA) model. The CMSTA model consists of a cross-nested logit (CNL) in the phase of mode split to account for intermodal transportation and a path-size logit (PSL) in the phase of traffic assignment to account for route overlapping. Also, we consider the flow interaction of different modes (e.g., car and bus) when they share the same links. The Barzilai–Borwein step size method is used to efficiently solve the lower-level CMSTA model (i.e., the CNL-PSL model) of which the objective function is complicated. The numerical results demonstrate the advantages and features of the proposed model. The model is also applied to a real network case to assess the network capacity of the multimodal transportation system.

Accelerating the gradient projection algorithm for solving the non-additive traffic equilibrium problem with the Barzilai-Borwein step size

The non-additive traffic equilibrium problem (NaTEP) overcomes the inadequacies of the additivity assumption in traditional traffic equilibrium models by relaxing the cost incurred on each path that is not a simple sum of the link costs on that path. The computation of the NaTEP heavily depends on the efficiency of the step size determination. This paper aims to accelerate the path-based gradient projection (GP) algorithm for solving the NaTEP using the Barzilai-Borwein (BB) step size scheme. The GP algorithm with the BB step size scheme uses the solution information of the last two iterations to determine a suitable step size and avoids extra evaluations of the mapping value. The proposed algorithm only needs to perform one-time projection onto the nonnegative orthant at each iteration. Two approaches with and without column generation are considered in the GP algorithm implementation. A non-additive shortest path algorithm is adopted for the column generation approach. Numerical results on four transportation networks demonstrate the superior efficiency and robustness of the GP algorithm with the BB step size scheme over the self-adaptive GP method.

On the acceleration of the Barzilai–Borwein method

The Barzilai–Borwein (BB) gradient method is efficient for solving large-scale unconstrained problems to modest accuracy due to its ingenious stepsize which generally yields nonmonotone behavior. In this paper, we propose a new stepsize to accelerate the BB method by requiring finite termination for minimizing the two-dimensional strongly convex quadratic function. Based on this new stepsize, we develop an efficient gradient method for quadratic optimization which adaptively takes the nonmonotone BB stepsizes and certain monotone stepsizes. Two variants using retard stepsizes associated with the new stepsize are also presented. Numerical experiments show that our strategies of properly inserting monotone gradient steps into the nonmonotone BB method could significantly improve its performance and our new methods are competitive with the most successful gradient descent methods developed in the recent literature.

An Accelerated Algorithm for Distributed Optimization with Barzilai-Borwein Step Sizes

An Accelerated Algorithm for Distributed Optimization with Barzilai-Borwein Step Sizes

Achieving geometric convergence for distributed optimization with Barzilai-Borwein step sizes

Achieving geometric convergence for distributed optimization with Barzilai-Borwein step sizes