Wolfe Line Search Research Articles

A descent modified HS conjugate gradient method with an optimal property

In this paper, by minimizing the distance between the CG direction and the direction of the improved Perry conjugate gradient method [Yao et al., Numer. Algorithms 78 (2018) 1255–1269], we propose a descent modified HS conjugate gradient method. A remarkable property of the modified HS method is that it can produce sufficient descent property, which is independent of the line search used. Under suitable conditions, we prove that the modified HS method with the standard Armijo line search is globally convergent for uniformly convex functions and the modified HS+ method with standard Wolfe line search is globally convergent for general nonlinear functions. Extensive numerical experiments show that the proposed method is efficient.

A modified PRP-type conjugate gradient algorithm with complexity analysis and its application to image restoration problems

In this paper, a modified conjugate gradient method is proposed for nonconvex optimization. This method possesses the sufficient descent property independent of any line search. The global convergence property of the algorithm is established under the Wolfe line search strategy or the Armijo line search condition, respectively. Additionally, the complexity analysis of the proposed algorithm is investigated. To reach the point with the norm of the gradient below ɛ, the worst-case complexity bound matches that of the gradient method. Moreover, the obtained numerical results demonstrate that the modified method is effective for large-scale optimization problems and image restoration problems.

Two families of self-adjusting spectral hybrid DL conjugate gradient methods and applications in image denoising

The study in this paper introduces two families of self-adjusting spectral hybrid DL conjugate gradient methods to solve unconstrained optimization problems. The search directions in the proposed methods are improved by integrating the negative spectral gradient and the last search direction that has convex combination structure. One component of composite conjugate parameter in the search direction is a hybridization of DL-like conjugate parameter and another flexible conjugate parameter. It is proved that the search directions in the two families of newly proposed methods always automatically satisfy the sufficient descent property, which is independent of choosing the specific conjugate parameter and line search. With general nonconvex objective functions, the convergence results under the weak Wolfe line search are obtained. Numerical experiments verify the efficiency and applicability of our developed methods by comparison with some existing methods for solving unconstrained optimization and image restoration problems.

New Method for Solving the Inverse Thermal Conduction Problem (θ-Scheme Combined with CG Method under Strong Wolfe Line Search)

Most thermal researchers have solved thermal conduction problems (inverse or direct) using several different methods. These include the usual discretization methods, conventional and special estimation methods, in addition to simple synchronous gradient methods such as finite elements, including finite and special quantitative methods. Quantities found through the finite difference methods, i.e., explicit, implicit or Crank–Nicolson scheme method, have also been adopted. These methods offer many disadvantages, depending on the different cases; when the solutions converge, limited range stability conditions. Accordingly, in this paper, a new general outline of the thermal conduction phenomenon, called (θ-scheme), as well as a gradient conjugate method that includes strong Wolfe conditions has been used. This approach is the most useful, both because of its accuracy (16 decimal points of importance) and the speed of its solutions and convergence; by addressing unfavorable adverse problems and stability conditions, it can also have wide applications. In this paper, we applied two approaches for the control of the boundary conditions: constant and variable. The θ-scheme method has rarely been used in the thermal field, though it is unconditionally more stable for θ∈ [0.5, 1]. The simulation was carried out using Matlab software.

Using a new hybrid conjugate gradient method with descent property

The conjugate gradient method was an efficient technique for solving optimization problems. In this paper, we propose a new efficient (CG) coefficient bk, is computed as a convex combination of Salleh and Alhawarat algorithm and CG-Descent algorithm. We prove the sufficient descent condition and the global convergence of the proposed method. It is established that the ak satisfies the strong Wolfe line search conditions. The numerical results indicate that our method is robust and competitive.

An efficient spectral minimization of the Dai-Yuan method with application to image reconstruction

<abstract><p>In this paper, a spectral Dai and Yuan conjugate gradient (CG) method is proposed based on the generalized conjugacy condition for large-scale unconstrained optimization, in which the spectral parameter is motivated by some interesting theoretical features of quadratic convergence associated with the Newton method. Accordingly, utilizing the strong Wolfe line search to yield the step-length, the search direction of the proposed spectral method is sufficiently descending and converges globally. By applying some standard Euclidean optimization test functions, numerical results reports show the advantage of the method over some modified Dai and Yuan CG schemes in literature. In addition, the method also shows some reliable results, when applied to solve an image reconstruction model.</p></abstract>

A new three-term conjugate gradient algorithm with modified gradient-differences for solving unconstrained optimization problems

<abstract><p>Unconstrained optimization problems often arise from mining of big data and scientific computing. On the basis of a modified gradient-difference, this article aims to present a new three-term conjugate gradient algorithm to efficiently solve unconstrained optimization problems. Compared with the existing nonlinear conjugate gradient algorithms, the search directions in this algorithm are always sufficiently descent independent of any line search, as well as having conjugacy property. Using the standard Wolfe line search, global and local convergence of the proposed algorithm is proved under mild assumptions. Implementing the developed algorithm to solve 750 benchmark test problems available in the literature, it is shown that the numerical performance of this algorithm is remarkable, especially in comparison with that of the other similar efficient algorithms.</p></abstract>

A new family of hybrid three-term conjugate gradient method for unconstrained optimization with application to image restoration and portfolio selection

<abstract><p>The conjugate gradient (CG) method is an optimization method, which, in its application, has a fast convergence. Until now, many CG methods have been developed to improve computational performance and have been applied to real-world problems. In this paper, a new hybrid three-term CG method is proposed for solving unconstrained optimization problems. The search direction is a three-term hybrid form of the Hestenes-Stiefel (HS) and the Polak-Ribiére-Polyak (PRP) CG coefficients, and it satisfies the sufficient descent condition. In addition, the global convergence properties of the proposed method will also be proved under the weak Wolfe line search. By using several test functions, numerical results show that the proposed method is most efficient compared to some of the existing methods. In addition, the proposed method is used in practical application problems for image restoration and portfolio selection.</p></abstract>

A new conjugate gradient method with a restart direction and its application in image restoration

<abstract><p>We established a new conjugate gradient method with an efficient restart direction for solving large scale unconstrained optimization problems. The modified method was proposed under the Polak-Ribière-Polyak conjugate gradient method. Under the strong Wolfe line search, the search direction of the new method was sufficiently descent and its global convergence property could be proved. Compared with other methods having good numerical performances, numerical experiments and image restorations showed that the modified method was more effective.</p></abstract>

Two New Preconditioned Conjugate Gradient Methods for Minimization Problems

In application to general function, each of the conjugate gradient and Quasi-Newton methods has particular advantages and disadvantages. Conjugate gradient (CG) techniques are a class of unconstrained optimization algorithms with strong local and global convergence qualities and minimal memory needs. Quasi-Newton methods are reliable and efficient on a wide range of problems and they converge faster than the conjugate gradient method and require fewer function evaluations but they have the disadvantage of requiring substantially more storage and if the problem is ill-conditioned, they may take several iterations. A new class has been developed, termed preconditioned conjugate gradient (PCG) method. It is a method that combines two methods, conjugate gradient and Quasi-Newton. In this work, two new preconditioned conjugate gradient algorithms are proposed namely New PCG1 and New PCG2 to solve nonlinear unconstrained optimization problems. A new PCG1 combines conjugate gradient method Hestenes-Stiefel (HS) with new self-scaling symmetric Rank one (SR1), and a new PCG2 combines conjugate gradient method Hestenes-Stiefel (HS) with new self-scaling Davidon, Flecher and Powell (DFP). The algorithm uses the strong Wolfe line search condition. Numerical comparisons with standard preconditioned conjugate gradient algorithms show that for these new algorithms, computational scheme outperforms the preconditioned conjugate gradient.

A three-term conjugate gradient algorithm with restart procedure to solve image restoration problems

Based on the Liu–Storey conjugate gradient method, a three-term conjugate gradient method with restart procedure is proposed for unconstrained optimization. The designed restart condition is chosen according to the Liu–Storey conjugate parameter, and the restart direction consists of the first and third items in the non-restart direction, which can be picked flexibly. Without depending on any line search, the resulting direction satisfies the sufficient descent condition. Moreover, the proposed method possesses the global convergence under usual assumptions and the weak Wolfe line search. Finally, medium–large-scale numerical experiments are conducted for solving unconstrained optimization and image restoration problems, and the numerical results show that the proposed method is promising and practical, even compared with the state-of-the-art methods.

A class of improved conjugate gradient methods for nonconvex unconstrained optimization

AbstractIn this paper, based on a new class of conjugate gradient methods which are proposed by Rivaie, Dai and Omer et al. we propose a class of improved conjugate gradient methods for nonconvex unconstrained optimization. Different from the above methods, our methods possess the following properties: (i) the search direction always satisfies the sufficient descent condition independent of any line search; (ii) these approaches are globally convergent with the standard Wolfe line search or standard Armijo line search without any convexity assumption. Moreover, our numerical results also demonstrated the efficiencies of the proposed methods.

Hybrid conjugate gradient-BFGS methods based on Wolfe line search

"In this paper, we present some hybrid methods for solving unconstrained optimization problems. These methods are defined using proper combinations of the search directions and included parameters in conjugate gradient and quasi-Newton method of Broyden-Fletcher-Goldfarb-Shanno (CG-BFGS). Their global convergence under the Wolfe line search is analyzed for general objective functions. Numerical experiments show the superiority of the modified hybrid (CG-BFGS) method with respect to some existing methods."

Extra parameters for conjugate gradient method for removing impulse noise images

In most applications, denoising a photograph is essential for most image processing processes. In this study, a conjugate gradient method for impulse noise reduction is presented, which is based on a two-phase system. The noise candidates are found with an adaptive (center-weighted) median filter, and then restored with the CG method, which entails the minimization of an edge-preserving regularization functional. The fact that the search direction created at each iteration is descending is one of the most appealing characteristics of the suggested technique. Its global convergence result might be obtained if a strong Wolfe line search is used. The spectral conjugate gradient approach for impulse noise reduction is demonstrated through numerical experiments.

Modified spectral conjugate gradient iterative scheme for unconstrained optimization problems with application on COVID-19 model

In this work, a new class of spectral conjugate gradient (CG) method is proposed for solving unconstrained optimization models. The search direction of the new method uses the ZPRP and JYJLL CG coefficients. The search direction satisfies the descent condition independent of the line search. The global convergence properties of the proposed method under the strong Wolfe line search are proved with some certain assumptions. Based on some test functions, numerical experiments are presented to show the proposed method's efficiency compared with other existing methods. The application of the proposed method for solving regression models of COVID-19 is provided.Mathematics subject classification65K10, 90C52, 90C26.

Global convergence of a modified RMIL+ nonlinear conjugate gradient method with strong wolfe

<span lang="EN-US">Nonlinear conjugate gradient (CG) methods are extensively used as an important technique for addressing large-scale unconstrained optimization problems which are arise in many aspects of science, engineering, and economics. That is due to their simplicity, convergence properties, and low memory requirements. To generate a new approximation solution in each iteration, the CG methods usually implement under the strong Wolfe line search. For good performance, many studies have been carried out to modify well-known CG methods. In this paper, we did some modifications on one of CG method called RMIL+ in order to obtain a new CG method possesses the sufficient descent property and the global convergence under strong Wolfe line search. The numerical results demonstrate that the suggested method outperforms other CG methods.</span>

A New Class of Optimization Methods Based on Coefficient Conjugate Gradient

The coefficient conjugate serves as the foundation for many conjugate gradient methods. The quadratic model is used to derive a novel coefficient conjugate in this study. Its global convergence result might be produced under Wolfe line search circumstances. The conjugate gradient method’s performance for unconstrained optimization problems is demonstrated through numerical tests.

An efficient conjugate gradient-based algorithm for unconstrained optimization and its projection extension to large-scale constrained nonlinear equations with applications in signal recovery and image denoising problems

In this paper, a hybrid version of the Rivaie–Mustafa–Ismail–Leong, RMIL for short, conjugate gradient method is proposed for unconstrained optimization problem. The search direction fulfills the sufficient descent condition and trust region property at each iteration, independent of any line search. We establish the global convergence of the proposed algorithm under normal assumptions and Wolfe line search. To move forward, incorporating with derivative-free projection technique, the proposed conjugate coefficient is extended to deal with nonlinear monotone equations. Without Lipschitz continuity assumption, the global convergence is proved with mild conditions. The numerical results for both unconstrained optimization instances and large-scale nonlinear equations prove the efficiency of the proposed methods with less computation cost. In particular, we also explore the practical applications on signal recovery and image denoising problems.

Solving Unconstrained Optimization Problems with Some Three-term Conjugate Gradient Methods

In this paper, based on the efficient Conjugate Descent ({\tt CD}) method, two generalized {\tt CD}algorithms are proposed to solve the unconstrained optimization problems.These methods are three-term conjugate gradient methods which the generateddirections by using the conjugate gradient parameters and independent of the line searchsatisfy in the sufficient descent condition. Furthermore, under the strong Wolfe line search,the global convergence of the proposed methods are proved. Also, the preliminary numericalresults on the {\tt CUTEst} collection are presented to show effectiveness of our methods.

A criterion for the global convergence of conjugate gradient methods under strong Wolfe line search

From 1952 until now, the sufficient descent property and the global convergence of conjugate gradient (CG) methods have been studied extensively. However, the sufficient descent property and the global convergence of some CG methods such as the method of Polak, Ribière, and Polyak (PRP) and the method of Hestenes and Stiefel (HS) have not been established yet under the strong Wolfe line search. In this paper, based on Yousif (Yousif, 2020) we present a criterion that guarantees the generation of descent search directions property and the global convergence of CG methods when they are applied under the strong Wolfe line search. Moreover, the PRP and the HS methods are restricted in order to satisfy the presented criterion, so new modified versions of PRP and HS are proposed. Finally, to support the theoretical proofs, a numerical experiment is done.