Sufficient Descent Research Articles

A modified PRP conjugate gradient method with inertial extrapolation for sparse signal reconstruction

It is widely known that the inertial technique of the heavy-ball method can accelerate its convergence speed. In this paper, by embedding the inertial technique in the famous PRP conjugate gradient method, we propose a modified PRP conjugate gradient method with inertial extrapolation (PRPCG-IE) for sparse signal reconstruction. Its direction satisfies the sufficient descent property, which is independent of any line search. Global convergence of PRPCG-IE is established under some standard conditions. PRPCG-IE is applied to two sparse signal reconstruction problems with noise, and preliminary experimental results demonstrate the effectiveness of PRPCG-IE.

Convergence of a class of algorithms with the level-set subdifferential error bound

In this paper, we study the convergence of algorithms for a class of nonconvex and nonsmooth optimization problems via an level-set subdifferential error bound. Many algorithms (cf. forward-backward splitting algorithm and alternating proximal minimization algorithm) satisfy sufficient descent and relative error conditions. We also find that these algorithms satisfy a quadratic error condition. Under these conditions and the level-set subdifferential error bound, we study the global convergence and the convergence rates of these algorithms without the Kurdyka-Łojasiewicz property which is stronger than the level-set subdifferential error bound. Two examples are given to illustrate the broader applicability of our findings across these algorithms.

A New Hybrid Descent Algorithm for Large-Scale Nonconvex Optimization and Application to Some Image Restoration Problems

Conjugate gradient methods are widely used and attractive for large-scale unconstrained smooth optimization problems, with simple computation, low memory requirements, and interesting theoretical information on the features of curvature. Based on the strongly convergent property of the Dai–Yuan method and attractive numerical performance of the Hestenes–Stiefel method, a new hybrid descent conjugate gradient method is proposed in this paper. The proposed method satisfies the sufficient descent property independent of the accuracy of the line search strategies. Under the standard conditions, the trust region property and the global convergence are established, respectively. Numerical results of 61 problems with 9 large-scale dimensions and 46 ill-conditioned matrix problems reveal that the proposed method is more effective, robust, and reliable than the other methods. Additionally, the hybrid method also demonstrates reliable results for some image restoration problems.

A Modified Wei-Yao-Liu Stochastic Conjugate Gradient Algorithm in Machine Learning

Abstract The Wei-Yao-Liu (WYL) Conjugate Gradient (CG) algorithm exhibits favourable attributes, notably sufficient descent and trust domain characteristics, in the context of solving unconstrained optimization problems. The exploration and enhancement of computational methodologies for stochastic optimization problems have garnered significant attention. In this study, our objective is to improve the Wei-Yao-Liu (WYL) CG algorithm and introduce an effective optimization method tailored for solving stochastic optimization problems. We will present a comprehensive framework for analysing the convergence of this algorithm and, under suitable conditions, establish its sufficient descent properties, trust region features, and global convergence of stationary points. We will also give an improved CG algorithm to calculate a framework for variance reduction. Additionally, we will conduct experimental comparisons to showcase the competitiveness of our proposed algorithm.

A Modified Three-terms CG Method with the MWWP Line Search and its Application to Image Restoration

Abstract In this article, an innovative method, which integrates the optimized three-term Polak–Ribière–Polyak (PRP) conjugate gradient (CG) method with the improved WWP line search rule, is presented and applied in the field of image restoration. The proposed search direction exhibits sufficient descent and trust region characteristics, maintaining its effectiveness regardless of the specific line search rule employed. We established the global convergence of the presented algorithm and eliminated the dependence on the Lipschitz continuity assumption. The numerical comparisons with the PRP CG method revealed the superior performance of the designed algorithm in the field of image restoration. This research outcome provides strong support for further advancements in the field of image recovery.

Two modified conjugate gradient methods for unconstrained optimization

Conjugate gradient (CG) methods are a popular class of iterative methods for solving linear systems of equations and nonlinear optimization problems. Motivated by the construction of some modern CG methods, we propose two modified CG methods, named DHS* and DPRP*. Under the strong Wolfe line search (SWLS), the two presented methods are proven to be sufficient descent and globally convergent. Preliminary numerical results show that the DHS* and DPRP* methods are effective for the given test problems.

On the Dai–Liao conjugate gradient method for vector optimization

Several conjugate gradient (CG) parameters have led to promising methods for optimization problems. However, some parameters, such as the Dai-Liao (DL+) and modified Polak-Ribière-Polyak (EPRP) methods, have not yet been explored in the vector optimization setting. This paper addresses this gap by proposing the DL+ and EPRP+ methods to vector optimization. We start by developing a self-adjusting DL+ algorithm to find critical points of vector-valued functions within a closed, convex, pointed cone with a nonempty interior. The algorithm is designed to satisfy the sufficient descent condition (SDC) with a Wolfe line search; if this condition is not satisfied, the algorithm adjusts by redefining the DL+ to meet the SDC. We establish the global convergence of this algorithm under the assumption of SDC without requiring regular restarts or convexity assumption on the objective functions. Similarly, we developed the EPRP+ algorithm using the same approach as the DL+ algorithm. Furthermore, under exact line search, the DL+ and EPRP+ methods reduce to Hestenes-Stiefel (HS) and Polak-Ribière-Polyak (PRP) methods, respectively. We present numerical experiments on some selected test problems sourced from the multiobjective optimization literature that demonstrate the effectiveness and practical implementation of the proposed algorithms, highlighting their promising potential.

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

In this work, for unconstrained optimization, we introduce an improved Dai‐Liao‐style hybrid conjugate gradient method based on the hybridization‐based self‐adaptive technique, and the search direction generated fulfills the sufficient descent and trust region properties regardless of any line search. The global convergence is established under standard Wolfe line search and common assumptions. Then, combining the hyperplane projection technique and a new self‐adaptive line search, we extend the proposed conjugate gradient method and obtain an improved Dai‐Liao‐style hybrid conjugate gradient projection method to solve constrained nonlinear monotone equations. Under mild conditions, we obtain its global convergence without Lipschitz continuity. In addition, the convergence rates for the two proposed methods are analyzed, respectively. Finally, numerical experiments are conducted to demonstrate the effectiveness of the proposed methods.

An Improved Three-Term Conjugate Gradient Algorithm for Constrained Nonlinear Equations under Non-Lipschitz Conditions and Its Applications

This paper proposes an improved three-term conjugate gradient algorithm designed to solve nonlinear equations with convex constraints. The key features of the proposed algorithm are as follows: (i) It only requires that nonlinear equations have continuous and monotone properties; (ii) The designed search direction inherently ensures sufficient descent and trust-region properties, eliminating the need for line search formulas; (iii) Global convergence is established without the necessity of the Lipschitz continuity condition. Benchmark problem numerical results illustrate the proposed algorithm’s effectiveness and competitiveness relative to other three-term algorithms. Additionally, the algorithm is extended to effectively address the image denoising problem.

A modified PRP conjugate gradient method for unconstrained optimization and nonlinear equations

A modified Polak Ribiere Polyak(PRP) conjugate gradient(CG) method is proposed for solving unconstrained optimization problems. The search direction generated by this method satisfies sufficient descent condition at each iteration and this method inherits one remarkable property of the standard PRP method. Under the standard Armijo line search, the global convergence and the linearly convergent rate of the presented method is established. Some numerical results are given to show the effectiveness of the proposed method by comparing with some existing CG methods.

The Dai–Liao-type conjugate gradient methods for solving vector optimization problems

This paper attempts to propose Dai–Liao (DL)-type nonlinear conjugate gradient (CG) methods for solving vector optimization problems. Four variants of the DL method are extended and analysed from the scalar case to the vector setting. We first give a direct vector extended version of the modified Dai–Liao (DL+) method. Then the second algorithm is presented by combining a new extension form of the well-known Hager–Zhang (HZ) parameter. This new scheme equips a good descent property and its parameter also reduces to the classical one in the scalar case. The last two algorithms extend two sufficient descent DL-type methods. Particularly, two different vector forms of their parameters are considered and compared. As a result, we reveal some differences between scalar and vector cases to a certain extent. Without any convex assumption, the sufficient descent property and global convergence of all algorithms are established under inexact line searches. Finally, preliminary numerical experiments illustrate the practical behaviour of our algorithms by using Dolan and Moré performance profile.

Correction: Two sufficient descent spectral conjugate gradient algorithms for unconstrained optimization with application

Correction: Two sufficient descent spectral conjugate gradient algorithms for unconstrained optimization with application

An improved descent Perry‐type algorithm for large‐scale unconstrained nonconvex problems and applications to image restoration problems

AbstractConjugate gradient methods are much effective for large‐scale unconstrained optimization problems by their simple computations and low memory requirements. The Perry conjugate gradient method has been considered to be one of the most efficient methods in the context of unconstrained minimization. However, a globally convergent result for general functions has not been established yet. In this paper, an improved three‐term Perry‐type algorithm is proposed which automatically satisfies the sufficient descent property independent of the accuracy of line search strategy. Under the standard Wolfe line search technique and a modified secant condition, the proposed algorithm is globally convergent for general nonlinear functions without convexity assumption. Numerical results compared with the Perry method for stability, two modified Perry‐type conjugate gradient methods and two effective three‐term conjugate gradient methods for large‐scale problems up to 300,000 dimensions indicate that the proposed algorithm is more efficient and reliable than the other methods for the testing problems. Additionally, we also apply it to some image restoration problems.

A Convex combination of improved Fletcher-Reeves and Rivaie-Mustafa-Ismail-Leong conjugate gradient methods for unconstrained optimization problems and applications

Conjugate gradient methods are probably the most used methods in solving large scale unconstrained optimization problems. They have become popular because of their simplicity and low memory requirements. In this paper, we propose a hybrid conjugate gradient method based on the improved Fletcher-Reeves (IFR) and Rivaie-Mustafa-Ismail-Leong+ (RMIL+) methods and establish its global convergence under the strong Wolfe line search conditions. The new conjugate gradient direction satisfies the sufficient descent condition. The method is then compared to other methods in the literature and numerical experiments show that it is competent when solving large scale unconstrained optimization problems. Furthermore, the method is applied to solve a problem in portfolio selection.

Hypergraph-based convex semi-supervised unconstraint symmetric matrix factorization for image clustering

Semi-supervised symmetric nonnegative matrix factorization (SNMF) has been extensively utilized in both linear and nonlinear data clustering tasks. However, the current SNMF model's non-convex objective function faces challenges in global optimization and time efficiency. In this study, we leverage label information to propose a convex and unconstrained symmetric matrix factorization (SMF) model that is thoroughly analyzed for its convexity properties. In order to capture high-order relationships among data, a hypergraph is utilized in the model, which is computationally simple, translation invariant, and naturally normalized. Moreover, based on the analysis and the corresponding experiments in the paper, the model exhibits robustness towards outliers to some extent. Due to the convexity of our proposed model without constraint, it can be efficiently optimized using the Conjugate Gradient (CG) method, one of the most efficient methods available. Therefore, we propose a novel Convex Combination-based Sufficient Descent CG (CSDCG) method, which outperforms other methods across 284 optimization problems within the CUTEst library. In order to evaluate the effectiveness of the proposed method, the semi-supervised clustering experiments are conducted on the eight datasets by comparison with ten state-of-the-art matrix factorization (MF) methods. The experiment results demonstrate its superiority over the other compared methods to handle the clustering problem with better performance and less computational time. The code is available at https://github.com/Pokemer/HCSSMF.

Abstract generalized epsilon-descent algorithm

Given the problem of minimizing a possibly nonconvex and nonsmooth function in a real Hilbert space, we present a generalized epsilon-descent algorithm motivated from the abstract descent method introduced by Attouch et al. [Math. Program. 137 (2013) 91–129] with two essential additions, we consider scalar errors on the sufficient descent condition, as well as, on the relative inexact optimality condition. Under general conditions on the function to be minimized, we obtain that all accumulation points of the sequences generated by the algorithm, if they exist, are generalized critical limit points of the objective function.

On diagonally structured scheme for nonlinear least squares and data-fitting problems

Recently, structured nonlinear least-squares (NLS) based algorithms gained considerable emphasis from researchers; this attention may result from increasingly applicable areas of these algorithms in different science and engineering domains. In this article, we coined a new efficient structured-based NLS algorithm. We developed a diagonal Hessian-based formulation for solving NLS problems. We derived the quasi-Newton update based on a diagonal matrix scheme subject to a modified structured secant condition. Also, we show that the algorithm’s search direction satisfies a sufficient descent condition under some standard assumptions. Subsequently, we also prove the global convergence of the algorithm and then eventually show its linear convergence rate for strongly convex functions. Furthermore, to show case the proposed algorithm’s performance, we experimented numerically by comparing it with other approaches on some benchmark test functions available in the literature. Finally, the introduced scheme is applied to solve some data-fitting problems

On Spectral Dai-Yuan Structured Technique for Solving Nonlinear Least-Squares Optimisation

Handling nonlinear least squares (NLS) problems as unconstrained optimisation problems has led to the development and adaptation of numerous numerical approaches. Calculating the Hessian matrix expensively for each iteration is one of the problems with the current numerical approaches for solving NLS problems. Some methods designed to rely solely on first-derivative information while ignoring second-order details tend to underperform when dealing with problems involving non-zero residuals. This paper suggests a modification to the Dai-Yuan type (DY) formula by deriving a spectral parameter for the proposed search direction. The sufficient descent results of the new formula are established using the standard assumptions under the strong Wolfe line search. Furthermore, experiments are conducted on some benchmark functions to demonstrate the computational efficiency of the proposed technique. The results highlight the effectiveness of the algorithm compared to existing methods.Keywords: Conjugate gradient, convergence theory, Hessian matrix, optimisation, Quasi-Newton, spectral, structured secant method

An efficient hybrid conjugate gradient method with an adaptive strategy and applications in image restoration problems

In this study, we introduce a novel hybrid conjugate gradient method with an adaptive strategy called asHCG method. The asHCG method exhibits the following characteristics. (i) Its search direction guarantees sufficient descent property without dependence on any line search. (ii) It possesses strong convergence for the uniformly convex function using a weak Wolfe line search, and under the same line search, it achieves global convergence for the general function. (iii) Employing the Armijo line search, it provides an approximate guarantee for worst-case complexity for the uniformly convex function. The numerical results demonstrate promising and encouraging performances in both unconstrained optimization problems and image restoration problems.

A family of spectral conjugate gradient methods with strong convergence and its applications in image restoration and machine learning

In this paper, we propose a family of spectral conjugate gradient methods for solving unconstrained optimization problems. Specifically, we provide two classes of bounded spectral parameters to be chosen, design a new truncation scheme of the non-negative conjugate parameter and set a restart procedure in our search direction. Independently of the specific spectral parameter, conjugate parameter and line search criterion, we prove that our proposed family satisfies the sufficient descent condition. We also prove its strong convergence under mild assumptions and the weak Wolfe line search. Numerical comparisons with other methods demonstrate the outstanding performances of our algorithm for solving medium–large-scale unconstrained optimization, image restoration and machine learning problems.