Wolfe Line Search Research Articles

Two modified conjugate gradient methods for unconstrained optimization

Conjugate gradient (CG) methods are a popular class of iterative methods for solving linear systems of equations and nonlinear optimization problems. Motivated by the construction of some modern CG methods, we propose two modified CG methods, named DHS* and DPRP*. Under the strong Wolfe line search (SWLS), the two presented methods are proven to be sufficient descent and globally convergent. Preliminary numerical results show that the DHS* and DPRP* methods are effective for the given test problems.

On the Dai–Liao conjugate gradient method for vector optimization

Several conjugate gradient (CG) parameters have led to promising methods for optimization problems. However, some parameters, such as the Dai-Liao (DL+) and modified Polak-Ribière-Polyak (EPRP) methods, have not yet been explored in the vector optimization setting. This paper addresses this gap by proposing the DL+ and EPRP+ methods to vector optimization. We start by developing a self-adjusting DL+ algorithm to find critical points of vector-valued functions within a closed, convex, pointed cone with a nonempty interior. The algorithm is designed to satisfy the sufficient descent condition (SDC) with a Wolfe line search; if this condition is not satisfied, the algorithm adjusts by redefining the DL+ to meet the SDC. We establish the global convergence of this algorithm under the assumption of SDC without requiring regular restarts or convexity assumption on the objective functions. Similarly, we developed the EPRP+ algorithm using the same approach as the DL+ algorithm. Furthermore, under exact line search, the DL+ and EPRP+ methods reduce to Hestenes-Stiefel (HS) and Polak-Ribière-Polyak (PRP) methods, respectively. We present numerical experiments on some selected test problems sourced from the multiobjective optimization literature that demonstrate the effectiveness and practical implementation of the proposed algorithms, highlighting their promising potential.

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

In this work, for unconstrained optimization, we introduce an improved Dai‐Liao‐style hybrid conjugate gradient method based on the hybridization‐based self‐adaptive technique, and the search direction generated fulfills the sufficient descent and trust region properties regardless of any line search. The global convergence is established under standard Wolfe line search and common assumptions. Then, combining the hyperplane projection technique and a new self‐adaptive line search, we extend the proposed conjugate gradient method and obtain an improved Dai‐Liao‐style hybrid conjugate gradient projection method to solve constrained nonlinear monotone equations. Under mild conditions, we obtain its global convergence without Lipschitz continuity. In addition, the convergence rates for the two proposed methods are analyzed, respectively. Finally, numerical experiments are conducted to demonstrate the effectiveness of the proposed methods.

Nonlinear conjugate gradient methods for unconstrained set optimization problems whose objective functions have finite cardinality

In this paper, we propose nonlinear conjugate gradient methods for unconstrained set optimization problems in which the objective function is given by a finite number of continuously differentiable vector-valued functions. First, we provide a general algorithm for the conjugate gradient method using Wolfe line search but without imposing an explicit restriction on the conjugate parameter. Later, we study two variants of the algorithm, namely Fletcher-Reeves and conjugate descent, using two different choices of the conjugate parameter but with the same line search rule. In the general algorithm, the direction of movement at each iterate is identified by finding out a descent direction of a vector optimization problem. This vector optimization problem is identified with the help of the concept of partition set at the current iterate. As this vector optimization problem is different at different iterations, the conventional conjugate gradient method for vector optimization cannot be straightly extended to solve the set optimization problem under consideration. The well-definedness of the methods is provided. Further, we prove the Zoutendijk-type condition, which assists in proving the global convergence of the methods even without a regular condition of the stationary points. No convexity assumption is assumed on the objective function to prove the convergence. Lastly, some numerical examples are illustrated to exhibit the performance of the proposed method. We compare the performance of the proposed conjugate gradient methods with the existing steepest descent method. It is found that the proposed method commonly outperforms the existing steepest descent method for set optimization.

An improved descent Perry‐type algorithm for large‐scale unconstrained nonconvex problems and applications to image restoration problems

AbstractConjugate gradient methods are much effective for large‐scale unconstrained optimization problems by their simple computations and low memory requirements. The Perry conjugate gradient method has been considered to be one of the most efficient methods in the context of unconstrained minimization. However, a globally convergent result for general functions has not been established yet. In this paper, an improved three‐term Perry‐type algorithm is proposed which automatically satisfies the sufficient descent property independent of the accuracy of line search strategy. Under the standard Wolfe line search technique and a modified secant condition, the proposed algorithm is globally convergent for general nonlinear functions without convexity assumption. Numerical results compared with the Perry method for stability, two modified Perry‐type conjugate gradient methods and two effective three‐term conjugate gradient methods for large‐scale problems up to 300,000 dimensions indicate that the proposed algorithm is more efficient and reliable than the other methods for the testing problems. Additionally, we also apply it to some image restoration problems.

A convex combination of improved Fletcher-Reeves and Rivaie-Mustafa-Ismail-Leong conjugate gradient methods for unconstrained optimization problems and applications

Conjugate gradient methods are probably the most used methods in solving large scale unconstrained optimization problems. They have become popular because of their simplicity and low memory requirements. In this paper, we propose a hybrid conjugate gradient method based on the improved Fletcher-Reeves (IFR) and Rivaie-Mustafa-Ismail-Leong+ (RMIL+) methods and establish its global convergence under the strong Wolfe line search conditions. The new conjugate gradient direction satisfies the sufficient descent condition. The method is then compared to other methods in the literature and numerical experiments show that it is competent when solving large scale unconstrained optimization problems. Furthermore, the method is applied to solve a problem in portfolio selection.

A novel hybrid conjugate gradient method for multiobjective optimization problems

In this paper, we propose a hybrid nonlinear conjugate gradient method with the strong Wolfe line search step-size rule to find the Pareto critical point of a multiobjective optimization problem. The novelty of our multiobjective conjugate gradient method (in short, MCGM) lies in that it uses a hybrid parameter β k CD − DY which is combined the CD parameter β k CD with the DY parameter β k DY introduced by Lucambio Pérez and Prudente. The global convergence of the proposed hybrid nonlinear MCGM is established under mild assumptions. Numerical experiments demonstrate the effectiveness of the proposed method.

On Spectral Dai-Yuan Structured Technique for Solving Nonlinear Least-Squares Optimisation

Handling nonlinear least squares (NLS) problems as unconstrained optimisation problems has led to the development and adaptation of numerous numerical approaches. Calculating the Hessian matrix expensively for each iteration is one of the problems with the current numerical approaches for solving NLS problems. Some methods designed to rely solely on first-derivative information while ignoring second-order details tend to underperform when dealing with problems involving non-zero residuals. This paper suggests a modification to the Dai-Yuan type (DY) formula by deriving a spectral parameter for the proposed search direction. The sufficient descent results of the new formula are established using the standard assumptions under the strong Wolfe line search. Furthermore, experiments are conducted on some benchmark functions to demonstrate the computational efficiency of the proposed technique. The results highlight the effectiveness of the algorithm compared to existing methods.Keywords: Conjugate gradient, convergence theory, Hessian matrix, optimisation, Quasi-Newton, spectral, structured secant method

An efficient hybrid conjugate gradient method with an adaptive strategy and applications in image restoration problems

In this study, we introduce a novel hybrid conjugate gradient method with an adaptive strategy called asHCG method. The asHCG method exhibits the following characteristics. (i) Its search direction guarantees sufficient descent property without dependence on any line search. (ii) It possesses strong convergence for the uniformly convex function using a weak Wolfe line search, and under the same line search, it achieves global convergence for the general function. (iii) Employing the Armijo line search, it provides an approximate guarantee for worst-case complexity for the uniformly convex function. The numerical results demonstrate promising and encouraging performances in both unconstrained optimization problems and image restoration problems.

A family of spectral conjugate gradient methods with strong convergence and its applications in image restoration and machine learning

In this paper, we propose a family of spectral conjugate gradient methods for solving unconstrained optimization problems. Specifically, we provide two classes of bounded spectral parameters to be chosen, design a new truncation scheme of the non-negative conjugate parameter and set a restart procedure in our search direction. Independently of the specific spectral parameter, conjugate parameter and line search criterion, we prove that our proposed family satisfies the sufficient descent condition. We also prove its strong convergence under mild assumptions and the weak Wolfe line search. Numerical comparisons with other methods demonstrate the outstanding performances of our algorithm for solving medium–large-scale unconstrained optimization, image restoration and machine learning problems.

Another hybrid conjugate gradient method as a convex combination of WYL and CD methods

Abstract Conjugate gradient (CG) methods are a popular class of iterative methods for solving linear systems of equations and nonlinear optimization problems. In this paper, a new hybrid conjugate gradient (CG) method is presented and analyzed for solving unconstrained optimization problems, where the parameter β k \beta_{k} is a convex combination of β k WYL \beta_{k}^{\mathrm{WYL}} and β k CD \beta_{k}^{\mathrm{CD}} . Under the strong Wolfe line search, the new method possesses the sufficient descent condition and the global convergence properties. The preliminary numerical results show the efficiency of our method in comparison with other CG methods. Furthermore, the proposed algorithm HWYLCD was extended to solve the problem of a mode function.

Two efficient spectral hybrid CG methods based on memoryless BFGS direction and Dai–Liao conjugacy condition

The spectral conjugate gradient (CG) method is one of the effective methods for solving unconstrained optimization problems. In this work, we introduce a composite hybrid CG parameter which is a convex combination of two new adaptive hybrid CG parameters. To derive the optimal choice of the combination coefficient in our proposed composite CG parameter, two approaches to calculating it are introduced. One is to minimize the distance between the hybrid CG direction and the self-scaling memoryless BFGS direction and the other is to apply the Dai–Liao conjugacy condition. Further, to make the search direction have better theoretical performance, two effective spectral hybrid CG methods are generated. Our proposed methods ensure the sufficient descent property regardless of the line search. And the global convergence results for general non-convex functions are established under some fundamental assumptions and Wolfe line search. Numerical experiments on solving unconstrained optimization problems illustrate the effectiveness of our proposed methods.

A sufficient descent hybrid conjugate gradient method without line search consideration and application

PurposePrevious RMIL versions of the conjugate gradient method proposed in literature exhibit sufficient descent with Wolfe line search conditions, yet their global convergence depends on certain restrictions. To alleviate these assumptions, a hybrid conjugate gradient method is proposed based on the conjugacy condition.Design/methodology/approachThe conjugate gradient (CG) method strategically alternates between RMIL and KMD CG methods by using a convex combination of the two schemes, mitigating their respective weaknesses. The theoretical analysis of the hybrid method, conducted without line search consideration, demonstrates its sufficient descent property. This theoretical understanding of sufficient descent enables the removal of restrictions previously imposed on versions of the RMIL CG method for global convergence result.FindingsNumerical experiments conducted using a hybrid strategy that combines the RMIL and KMD CG methods demonstrate superior performance compared to each method used individually and even outperform some recent versions of the RMIL method. Furthermore, when applied to solve an image reconstruction model, the method exhibits reliable results.Originality/valueThe strategy used to demonstrate the sufficient descent property and convergence result of RMIL CG without line search consideration through hybrid techniques has not been previously explored in literature. Additionally, the two CG schemes involved in the combination exhibit similar sufficient descent structures based on the assumption regarding the norm of the search direction.

An efficient Dai-Yuan projection-based method with application in signal recovery.

The Dai and Yuan conjugate gradient (CG) method is one of the classical CG algorithms using the numerator ‖gk+1‖2. When the usual Wolfe line search is used, the algorithm is shown to satisfy the descent condition and to converge globally when the Lipschitz condition is assumed. Despite these two advantages, the Dai-Yuan algorithm performs poorly numerically due to the jamming problem. This work will present an efficient variant of the Dai-Yuan CG algorithm that solves a nonlinear constrained monotone system (NCMS) and resolves the aforementioned problems. Our variant algorithm, like the unmodified version, converges globally when the Lipschitz condition and sufficient descent requirements are satisfied, regardless of the line search method used. Numerical computations utilizing algorithms from the literature show that this variant algorithm is numerically robust. Finally, the variant algorithm is used to reconstruct sparse signals in compressed sensing (CS) problems.

THE BEST SPECTRAL CORRECTION OF DMDY CONJUGATE GRADIENT METHOD

In this paper, we present an enhanced spectral correction for the DMDY conjugate gradient method. Our approach involves integrating a third term and determining its parameter through three different approaches. The primary objective is to ensure the sufficient descent condition. By applying the Wolfe line search conditions, we establish the global convergence property for all three proposed algorithms. Numerical tests conclusively demonstrate the superior efficiency of our algorithms, surpassing that of existing methods.

A three-term conjugate gradient descent method with some applications

The stationary point of optimization problems can be obtained via conjugate gradient (CG) methods without the second derivative. Many researchers have used this method to solve applications in various fields, such as neural networks and image restoration. In this study, we construct a three-term CG method that fulfills convergence analysis and a descent property. Next, in the second term, we employ a Hestenses-Stiefel CG formula with some restrictions to be positive. The third term includes a negative gradient used as a search direction multiplied by an accelerating expression. We also provide some numerical results collected using a strong Wolfe line search with different sigma values over 166 optimization functions from the CUTEr library. The result shows the proposed approach is far more efficient than alternative prevalent CG methods regarding central processing unit (CPU) time, number of iterations, number of function evaluations, and gradient evaluations. Moreover, we present some applications for the proposed three-term search direction in image restoration, and we compare the results with well-known CG methods with respect to the number of iterations, CPU time, as well as root-mean-square error (RMSE). Finally, we present three applications in regression analysis, image restoration, and electrical engineering.

Spectral-like conjugate gradient methods with sufficient descent property for vector optimization.

Several conjugate gradient (CG) parameters resulted in promising methods for optimization problems. However, it turns out that some of these parameters, for example, 'PRP,' 'HS,' and 'DL,' do not guarantee sufficient descent of the search direction. In this work, we introduce new spectral-like CG methods that achieve sufficient descent property independently of any line search (LSE) and for arbitrary nonnegative CG parameters. We establish the global convergence of these methods for four different parameters using Wolfe LSE. Our algorithm achieves this without regular restart and assumption of convexity regarding the objective functions. The sequences generated by our algorithm identify points that satisfy the first-order necessary condition for Pareto optimality. We conduct computational experiments to showcase the implementation and effectiveness of the proposed methods. The proposed spectral-like methods, namely nonnegative SPRP, SHZ, SDL, and SHS, exhibit superior performance based on their arrangement, outperforming HZ and SP methods in terms of the number of iterations, function evaluations, and gradient evaluations.

An accelerated hybrid Riemannian conjugate gradient method for unconstrained optimization

With the goal to deal with a series of optimization problems on general matrix manifolds with differentiable objective functions, we propose an accelerated hybrid Riemannian conjugate gradient technique. Specifically, the acceleration scheme of the proposed method using a modified stepsize which is multiplicatively determined by the Wolfe line search. The search direction of the proposed algorithm is determined by the hybrid conjugate parameter with computationally promising. We showed that the suggested approach converges globally to a stationary point. Our approach performs better than the state of art Riemannian conjugate gradient algorithms, as illustrated by computations on problems such as the orthogonal Procrustes problem and the Brockett-cost-function minimization problem.

Application of a globally convergent hybrid conjugate gradient method in portfolio optimization

Abstract In this paper, we propose a modification that improves efficiency, robustness and reliability of the famous HS conjugate gradient method. In particular, we propose a hybrid of the HS and DHS methods, where DHS is another recent modification of the HS method. Irrespective of the line search, the search direction of the proposed method is sufficiently descent. Moreover, the new approach guarantees global convergence for general functions under the strong Wolfe line search. Numerical results and performance profiles are reported, and indicate that the new approach outperforms three similar methods in the literature. We also give a practical application of the new approach in minimizing risk in portfolio selection.

An IoT‐based multiple teaching quality evaluation method for English translation with improved deep learning

AbstractTo enhance the teaching quality (TQ) of English translation (ET) courses, multivariate evaluation methods have garnered significant attention for their ability to identify deficiencies in the teaching process and subsequently improve instructional standards. Traditional evaluation techniques have shown limited effectiveness. Therefore, this paper proposes an ET in multiple teaching quality (MTQ) evaluation method with improved deep learning in internet of things (IoT). First, the paper establishes evaluation indicators for ET in MTQ, which mainly encompass ET teaching effectiveness, pedagogic competency in ET, teaching methods, content, as well as attitudes towards ET teaching. It then utilizes IoT technology to preliminarily collect data on these indicators, and using clustering algorithm based on weighted attributes and density (CABWAD) algorithm to mine data on English teaching MTQ evaluation indicators for ET. The extracted evaluation indicator data is then denoised using a probabilistic undirected graph model. Ultimately, the multilayer perceptron in deep learning is improved through Wolfe line search optimization, and this enhanced multilayer perceptron is employed to construct an ET in MTQ evaluation model. The denoised indicator data is inputted into the model, which then outputs precise MTQ evaluation results. The results show that the absolute value of the average Pearson correlation coefficient of this method is the highest, the Spearman correlation coefficient is the lowest, the mean average precision value is 0.965, and the positive category imbalance degree and the negative category imbalance degree are the lowest, indicating that the proposed method has outstanding performance in all aspects, and has certain application value in the field of TQ evaluation.