Barzilai Borwein Research Articles

A new inexact stochastic recursive gradient descent algorithm with Barzilai–Borwein step size in machine learning

A new inexact stochastic recursive gradient descent algorithm with Barzilai–Borwein step size in machine learning

A first order reliability method based on hybrid conjugate approach with adaptive Barzilai–Borwein steps

In the first order reliability method (FORM), the Hasofer–Lind and Rackwitz–Flessler (HL–RF) algorithm sometimes encounters numerical instability problems due to the highly nonlinear limit state function (LSF). In this paper, an improved HL–RF algorithm introducing the hybrid conjugate gradient method with adaptive Barzilai–Borwein step sizes is developed to enhance the robustness and efficiency of the original HL–RF method. The proposed algorithm is composed of two stages, the steepest descent method is performed in the first stage to move to the vicinity of the most probable failure point (MPFP) and provide a good initial location for the second stage. Along with the hybrid conjugate search direction defined by the descent direction of the non-differential merit function, the second stage quantizes the adaptive Barzilai–Borwein step sizes to accelerate the process of locating the final MPFP under the nonmonotone line search rule. Eight illustrative examples with nonlinear LSFs are analyzed in detail to validate the performance of the proposed algorithm compared with other first order reliability methods. The results indicate that the proposed algorithm is not only computationally efficient but also robust in terms of convergence, especially for those problems with super nonlinear LSFs and with nonlinear LSFs involving high-frequency noise terms.

A gradient method exploiting the two dimensional quadratic termination property

The quadratic termination property is important to the efficiency of gradient methods. We consider equipping a family of gradient methods, where the stepsize is given by the ratio of two norms, with two dimensional quadratic termination. Such a desired property is achieved by cooperating with a new stepsize which is derived by maximizing the stepsize of the considered family in the next iteration. It is proved that each method in the family will asymptotically alternate in a two dimensional subspace spanned by the eigenvectors corresponding to the largest and smallest eigenvalues. Based on this asymptotic behavior, we show that the new stepsize converges to the reciprocal of the largest eigenvalue of the Hessian. Furthermore, by adaptively taking the long Barzilai–Borwein stepsize and reusing the new stepsize with retard, we propose an efficient gradient method for unconstrained quadratic optimization. We prove that the new method is R-linearly convergent with a rate of \(1-1/\kappa\), where \(\kappa\) is the condition number of Hessian. Numerical experiments show the efficiency of our proposed method.

Hybrid limited memory gradient projection methods for box-constrained optimization problems

Gradient projection methods represent effective tools for solving large-scale constrained optimization problems thanks to their simple implementation and low computational cost per iteration. Despite these good properties, a slow convergence rate can affect gradient projection schemes, especially when high accurate solutions are needed. A strategy to mitigate this drawback consists in properly selecting the values for the steplength along the negative gradient. In this paper, we consider the class of gradient projection methods with line search along the projected arc for box-constrained minimization problems and we analyse different strategies to define the steplength. It is well known in the literature that steplength selection rules able to approximate, at each iteration, the eigenvalues of the inverse of a suitable submatrix of the Hessian of the objective function can improve the performance of gradient projection methods. In this perspective, we propose an automatic hybrid steplength selection technique that employs a proper alternation of standard Barzilai–Borwein rules, when the final active set is not well approximated, and a generalized limited memory strategy based on the Ritz-like values of the Hessian matrix restricted to the inactive constraints, when the final active set is reached. Numerical experiments on quadratic and non-quadratic test problems show the effectiveness of the proposed steplength scheme.

Spectral representation-based efficient simulation method for fully non-stationary spatially varying ground motions

As one of the most commonly-used simulation methods, the spectral representation method (SRM) has been utilized to generate fully non-stationary spatially varying ground motions. However, this method usually fails to simulate these ground motions at a large number of interesting locations because of the heavy computational cost of the decomposition of spectral matrices and the superposition of harmonic functions, especially for simulations with time-frequency coupled modulation functions. To address this issue, this study develops an efficient SRM-based method based on a simple interpolation strategy and a quadratic regularization projected Barzilai–Borwein assisted non-negative matrix factorization (QRPBB-NMF). Concretely, the distribution of interpolation nodes is predetermined via spectral feature analysis and only the spectral matrix decomposition at these nodes is required. Then, QRPBB-NMF is introduced to decouple the decomposed spectra into the sum of the products of various time and frequency expressions. Finally, a simple spline interpolation is carried out for these expressions, thereby approximately achieving the SRM-required matrix decomposition in a decoupled manner. The developed method can decrease the operand of the matrix decomposition and invoke the Fast Fourier Transform to speed up the harmonic superposition. A case study in which the fully non-stationary spatially varying ground motions of a tunnel are simulated proves the effectiveness of the developed method in terms of accuracy and efficiency.

A Scaled Dai–Yuan Projection-Based Conjugate Gradient Method for Solving Monotone Equations with Applications

In this paper, we propose two scaled Dai–Yuan (DY) directions for solving constrained monotone nonlinear systems. The proposed directions satisfy the sufficient descent condition independent of the line search strategy. We also reasonably proposed two different relations for computing the scaling parameter at every iteration. The first relation is proposed by approaching the quasi-Newton direction, and the second one is by taking the advantage of the popular Barzilai–Borwein strategy. Moreover, we propose a robust projection-based algorithm for solving constrained monotone nonlinear equations with applications in signal restoration problems and reconstructing the blurred images. The global convergence of this algorithm is also provided, using some mild assumptions. Finally, a comprehensive numerical comparison with the relevant algorithms shows that the proposed algorithm is efficient.

A manifold inexact augmented Lagrangian method for nonsmooth optimization on Riemannian submanifolds in Euclidean space

Abstract We develop a manifold inexact augmented Lagrangian framework to solve a family of nonsmooth optimization problem on Riemannian submanifold embedding in Euclidean space, whose objective function is the sum of a smooth function (but possibly nonconvex) and a nonsmooth convex function in Euclidean space. By utilizing the Moreau envelope, we get a smoothing Riemannian minimization subproblem at each iteration of the proposed method. Consequentially, each iteration subproblem is solved by a Riemannian Barzilai–Borwein gradient method. Theoretically, the convergence to critical point of the proposed method is established under some mild assumptions. Numerical experiments on compressed modes problems in physic and sparse principal component analysis demonstrate that the proposed method is a competitive method compared with some state-of-the-art methods.

Multimodal Urban Transportation Network Capacity Model Considering Intermodal Transportation

Transportation network capacity is an important indicator from measuring the performance of a transportation system. Previous studies on the capacity of multimodal networks have not considered transfers between different transportation modes. In fact, because of the diversified development of the urban transportation system, travelers are no longer limited to a single transportation mode in daily travel. This paper thus proposes an improved network capacity model considering intermodal transportation in urban multimodal transportation systems. The proposed network capacity model is formulated as a bi-level programming problem, in which the lower-level model is a combined modal split and traffic assignment (CMSTA) model. The CMSTA model consists of a cross-nested logit (CNL) in the phase of mode split to account for intermodal transportation and a path-size logit (PSL) in the phase of traffic assignment to account for route overlapping. Also, we consider the flow interaction of different modes (e.g., car and bus) when they share the same links. The Barzilai–Borwein step size method is used to efficiently solve the lower-level CMSTA model (i.e., the CNL-PSL model) of which the objective function is complicated. The numerical results demonstrate the advantages and features of the proposed model. The model is also applied to a real network case to assess the network capacity of the multimodal transportation system.

An extended projected residual algorithm for solving smooth convex optimization problems

A projected residual algorithm for solving smooth convex optimization problems is presented. The proposed method is an extension of a residual algorithm for solving systems of nonlinear monotone equations introduced by La Cruz (2017), which uses in a systematic way the residual as a search direction combined with the Barzilai–Borwein’s choice of the step size and a line search globalization strategy that does not impose the condition that the function value to decrease monotonically at every iteration. The global and R-sublinear convergence of the new method is established. With the aim of showing the advantages of the proposed global scheme an extensive set of numerical experiments including standard test problems and some specific applications are reported.

Accelerating the gradient projection algorithm for solving the non-additive traffic equilibrium problem with the Barzilai-Borwein step size

The non-additive traffic equilibrium problem (NaTEP) overcomes the inadequacies of the additivity assumption in traditional traffic equilibrium models by relaxing the cost incurred on each path that is not a simple sum of the link costs on that path. The computation of the NaTEP heavily depends on the efficiency of the step size determination. This paper aims to accelerate the path-based gradient projection (GP) algorithm for solving the NaTEP using the Barzilai-Borwein (BB) step size scheme. The GP algorithm with the BB step size scheme uses the solution information of the last two iterations to determine a suitable step size and avoids extra evaluations of the mapping value. The proposed algorithm only needs to perform one-time projection onto the nonnegative orthant at each iteration. Two approaches with and without column generation are considered in the GP algorithm implementation. A non-additive shortest path algorithm is adopted for the column generation approach. Numerical results on four transportation networks demonstrate the superior efficiency and robustness of the GP algorithm with the BB step size scheme over the self-adaptive GP method.

On the acceleration of the Barzilai–Borwein method

The Barzilai–Borwein (BB) gradient method is efficient for solving large-scale unconstrained problems to modest accuracy due to its ingenious stepsize which generally yields nonmonotone behavior. In this paper, we propose a new stepsize to accelerate the BB method by requiring finite termination for minimizing the two-dimensional strongly convex quadratic function. Based on this new stepsize, we develop an efficient gradient method for quadratic optimization which adaptively takes the nonmonotone BB stepsizes and certain monotone stepsizes. Two variants using retard stepsizes associated with the new stepsize are also presented. Numerical experiments show that our strategies of properly inserting monotone gradient steps into the nonmonotone BB method could significantly improve its performance and our new methods are competitive with the most successful gradient descent methods developed in the recent literature.

Total Variation Constrained Graph Manifold Learning Strategy for Cerenkov Luminescence Tomography.

Harnessing the power and flexibility of radiolabeled molecules, Cerenkov luminescence tomography (CLT) provides a novel technique for non-invasive visualisation and quantification of viable tumour cells in a living organism. However, owing to the photon scattering effect and the ill-posed inverse problem, CLT still suffers from insufficient spatial resolution and shape recovery in various preclinical applications. In this study, we proposed a total variation constrained graph manifold learning (TV-GML) strategy for achieving accurate spatial location, dual-source resolution, and tumour morphology. TV-GML integrates the isotropic total variation term and dynamic graph Laplacian constraint to make a trade-off between edge preservation and piecewise smooth region reconstruction. Meanwhile, the tetrahedral mesh-Cartesian grid pair method based on the k-nearest neighbour, and the adaptive and composite Barzilai-Borwein method, were proposed to ensure global super linear convergence of the solution of TV-GML. The comparison results of both simulation experiments and in vivo experiments further indicated that TV-GML achieved superior reconstruction performance in terms of location accuracy, dual-source resolution, shape recovery capability, robustness, and in vivo practicability. Significance: We believe that this novel method will be beneficial to the application of CLT for quantitative analysis and morphological observation of various preclinical applications and facilitate the development of the theory of solving inverse problem.

Solving the discrete Euler–Arnold equations for the generalized rigid body motion

We propose three iterative methods for solving the Moser–Veselov equation, which arises in the discretization of the Euler–Arnold differential equations governing the motion of a generalized rigid body. We start by formulating the problem as an optimization problem with orthogonal constraints and proving that the objective function is convex. Then, using techniques from optimization on Riemannian manifolds, the three feasible algorithms are designed. The first one splits the orthogonal constraints using the Bregman method, whereas the other two methods are of the steepest-descent type. The second method uses the Cayley-transform to preserve the constraints and a Barzilai–Borwein step size, while the third one involves geodesics, with the step size computed by Armijo’s rule. Finally, a set of numerical experiments is carried out to compare the performance of the proposed algorithms, suggesting that the first algorithm has the best performance in terms of accuracy and number of iterations. An essential advantage of these iterative methods is that they work even when the conditions for applicability of the direct methods available in the literature are not satisfied.

Knowledge Preserving and Distribution Alignment for Heterogeneous Domain Adaptation

Domain adaptation aims at improving the performance of learning tasks in a target domain by leveraging the knowledge extracted from a source domain. To this end, one can perform knowledge transfer between these two domains. However, this problem becomes extremely challenging when the data of these two domains are characterized by different types of features, i.e., the feature spaces of the source and target domains are different, which is referred to as heterogeneous domain adaptation (HDA). To solve this problem, we propose a novel model called Knowledge Preserving and Distribution Alignment (KPDA), which learns an augmented target space by jointly minimizing information loss and maximizing domain distribution alignment. Specifically, we seek to discover a latent space, where the knowledge is preserved by exploiting the Laplacian graph terms and reconstruction regularizations. Moreover, we adopt the Maximum Mean Discrepancy to align the distributions of the source and target domains in the latent space. Mathematically, KPDA is formulated as a minimization problem with orthogonal constraints, which involves two projection variables. Then, we develop an algorithm based on the Gauss–Seidel iteration scheme and split the problem into two subproblems, which are solved by searching algorithms based on the Barzilai–Borwein (BB) stepsize. Promising results demonstrate the effectiveness of the proposed method.

On the convergence and mesh-independent property of the Barzilai–Borwein method for PDE-constrained optimization

Abstract Aiming at optimization problems governed by partial differential equations (PDEs), local R-linear convergence of the Barzilai–Borwein (BB) method for a class of twice continuously Fréchet-differentiable functions is proven. Relying on this result, the mesh-independent principle for the BB-method is investigated. The applicability of the theoretical results is demonstrated for two different types of PDE-constrained optimization problems. Numerical experiments are given, which illustrate the theoretical results.

An alternating nonmonotone projected Barzilai–Borwein algorithm of nonnegative factorization of big matrices

An alternating nonmonotone projected Barzilai–Borwein algorithm of nonnegative factorization of big matrices

Concurrent Modified Constant Modulus Algorithm and Decision Directed Scheme With Barzilai-Borwein Method.

At present, in robot technology, remote control of robot is realized by wireless communication technology, and data anti-interference in wireless channel becomes a very important part. Any wireless communication system has an inherent multi-path propagation problem, which leads to the expansion of generated symbols on a time scale, resulting in symbol overlap and Inter-symbol Interference (ISI). ISI in the signal must be removed and the signal restores to its original state at the time of transmission or becomes as close to it as possible. Blind equalization is a popular equalization method for recovering transmitted symbols of superimposed noise without any pilot signal. In this work, we propose a concurrent modified constant modulus algorithm (MCMA) and the decision-directed scheme (DDS) with the Barzilai-Borwein (BB) method for the purpose of blind equalization of wireless communications systems (WCS). The BB method, which is two-step gradient method, has been widely employed to solve multidimensional unconstrained optimization problems. Considering the similarity of equalization process and optimization process, the proposed algorithm combines existing blind equalization algorithm and Barzilai-Borwein method, and concurrently operates a MCMA equalizer and a DD equalizer. After that, it modifies the DD equalizer's step size (SS) by the BB method. Theoretical investigation was involved and it demonstrated rapid convergence and improved equalization performance of the proposed algorithm compared with the original one. Additionally, the simulation results were consistent with the proposed technique.

Newton’s method for the parameterized generalized eigenvalue problem with nonsquare matrix pencils

The l parameterized generalized eigenvalue problems for the nonsquare matrix pencils, proposed by Chu et al.in 2006, can be formulated as an optimization problem on a corresponding complex product Stiefel manifold. In this paper, an effective and efficient algorithm based on the Riemannian Newton’s method is established to solve the underlying problem. Under our proposed framework, to solve the corresponding Newton’s equation, it can be converted to solve a standard real symmetric linear system with a dimension reduction. By combining the Riemannian curvilinear search method with Barzilai–Borwein steps, a hybrid algorithm with both global and quadratic convergence is obtained. Numerical experiments are provided to illustrate the efficiency of the proposed method. Detailed comparisons with some latest methods are also provided to show the merits of the proposed approach.

AFISTA: Accelerated FISTA for sparse signal recovery and compressive sensing

This paper presents a new fast iterative shrinkage-thresholding algorithm, termed AFISTA. The essential idea is to improve the convergence rate of FISTA using a new continuation strategy leading to a less number of iterations compared to FISTA. The convergence theorem of the AFISTA is proposed. In order to further accelerate the AFISTA method, it is equipped with the Barzilai-Borwein (BB) method. Also, for applications with orthogonal sensing matrix A, we proposed a specialized version of the AFISTA method. AFISTA is tailored for solving the basis pursuit problem which can be applied successfully on a variety of problems arising in signal and image processing issues such as sparse signal recovery, signal and image denoising, image restoration, and compressive sensing. To show the efficiency of the method, we compare our results with generalizations of linearized Bregman and fixed - point continuation (FPC) methods in sparse signal recovery applications, with split Bregman method in compressive sensing for sparse MRI and with Gradient projection for sparse reconstruction (GPSR) method in image deconvolution. Numerical results demonstrate that AFISTA overcomes all of the compared methods in convergence rate and some of them in both convergence rate and quality of reconstructed results.

Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme

Stochastic variance reduced gradient (SVRG) methods are important approaches to minimize the average of a large number of cost functions frequently arising in machine learning and many other applications. In this paper, based on SVRG, we propose a SVRG-TR method which employs a trust-region-like scheme for selecting stepsizes. It is proved that the SVRG-TR method is linearly convergent in expectation for smooth strongly convex functions and enjoys a faster convergence rate than SVRG methods. In order to overcome the difficulty of tuning stepsizes by hand, we propose to combine the Barzilai–Borwein (BB) method to automatically compute stepsizes for the SVRG-TR method, named as the SVRG-TR-BB method. By incorporating mini-batching scheme with SVRG-TR and SVRG-TR-BB, respectively, we further propose two extended methods mSVRG-TR and mSVRG-TR-BB. Linear convergence and complexity of mSVRG-TR are given. Numerical experiments on some standard datasets show that SVRG-TR and SVRG-TR-BB are generally better than or comparable to SVRG with best-tuned stepsizes and some modern stochastic gradient methods, while mSVRG-TR and mSVRG-TR-BB are very competitive with mini-batch variants of recent successful stochastic gradient methods.