Wolfe Line Search Research Articles

An efficient hybrid conjugate gradient method with descent properties under strong Wolfe line search

The hybrid conjugate gradient parameters are among the efficient variants of conjugate gradient (CG) methods for solving large-scale unconstrained optimization problems. This is due to their nice convergence properties and low memory requirements. In this paper, we present a new hybrid conjugate gradient method based on famous CG algorithms for large-scale unconstrained optimization. The proposed hybrid CG method can generate a descent search direction at each iteration provided the strong Wolfe line search is employed. Numerical results have been presented which show that the proposed method is efficient and promising.

A new hybrid conjugate gradient method of unconstrained optimization methods

In this paper, we present a new hybrid method to solve a nonlinear unconstrained optimization problem by using conjugate gradient, which is a convex combination of Liu–Storey (LS) conjugate gradient method and Hager–Zhang (HZ) conjugate gradient method. This method possesses the sufficient descent property with Strong Wolfe line search and the global convergence with the strong Wolfe line search. In the end of this paper, we illustrate our method by giving some numerical examples.

A new three-term spectral conjugate gradient algorithm with higher numerical performance for solving large scale optimization problems based on Quasi-Newton equation

A new spectral three-term conjugate gradient algorithm in virtue of the Quasi-Newton equation is developed for solving large-scale unconstrained optimization problems. It is proved that the search directions in this algorithm always satisfy a sufficiently descent condition independent of any line search. Global convergence is established for general objective functions if the strong Wolfe line search is used. Numerical experiments are employed to show its high numerical performance in solving large-scale optimization problems. Particularly, the developed algorithm is implemented to solve the 100 benchmark test problems from CUTE with different sizes from 1000 to 10,000, in comparison with some similar ones in the literature. The numerical results demonstrate that our algorithm outperforms the state-of-the-art ones in terms of less CPU time, less number of iteration or less number of function evaluation.

A new CG algorithm based on a scaled memoryless BFGS update with adaptive search strategy, and its application to large-scale unconstrained optimization problems

Conjugate gradient method is an effective method for solving large-scale unconstrained optimization problems. This paper proposes a new conjugate gradient algorithm based on the self-scaling memoryless BFGS update, which uses the Wolfe line search. The descent and global convergence of the method are given under mild conditions. Numerical experiments show that our new conjugate gradient method is effective.

Fast control parameterization optimal control with improved Polak–Ribière–Polyak conjugate gradient implementation for industrial dynamic processes

This paper proposes a fast control parameterization optimal control algorithm for industrial dynamic process with constraints. Derived from the frame of control variable parameterization (CVP) technique, the proposed method combines an efficient gradient computation strategy with an improved nonlinear optimization computation approach to overcome the challenge of computation efficiency caused by gradients and bounds in optimal control problems. Firstly, a fast gradient computation method based on the costate system of Hamiltonian function is developed to decrease the computational expense of gradients by employing approximate treatments and numerical integration strategy. Then, a trigonometric function transformation scheme is presented to tackle the boundary constraints so that the original optimal control problem is further converted into an unconstrained one. On this basis, an improved restricted Polak–Ribière–Polyak (PRP) conjugate gradient approach is introduced to solve the nonlinear optimization problem by using conjugate gradient iterations and strong Wolfe line search. Meanwhile, to enhance the convergence, a restricting condition is imposed in strong Wolfe line search to create iteration step-length. Finally, the proposed algorithm is implemented on three dynamic processes. The detailed comparison among the classical CVP method, literature results and the proposed method are carried out. Simulation studies show that the proposed fast approach averagely saves more than 90% computation time in contrast to the classical CVP method, demonstrating the effectiveness of the proposed fast optimal control approach.

The Global Convergence of a Modified BFGS Method under Inexact Line Search for Nonconvex Functions

Among the quasi-Newton algorithms, the BFGS method is often discussed by related scholars. However, in the case of inexact Wolfe line searches or even exact line search, the global convergence of the BFGS method for nonconvex functions is not still proven. Based on the aforementioned issues, we propose a new quasi-Newton algorithm to obtain a better convergence property; it is designed according to the following essentials: (1) a modified BFGS formula is designed to guarantee that B k + 1 inherits the positive definiteness of B k ; (2) a modified weak Wolfe–Powell line search is recommended; (3) a parabola, which is considered as the projection plane to avoid using the invalid direction, is proposed, and the next point x k + 1 is designed by a projection technique; (4) to obtain the global convergence of the proposed algorithm more easily, the projection point is used at all the next iteration points instead of the current modified BFGS update formula; and (5) the global convergence of the given algorithm is established under suitable conditions. Numerical results show that the proposed algorithm is efficient.

Some generalized three-term conjugate gradient methods based on CD approach for unconstrained optimization problems

Abstract In this paper, based on the efficient Conjugate Descent (CD) method, two generalized CD algorithms are proposed to solve the unconstrained optimization problems. These methods are three-term conjugate gradient methods which the generated directions by using the conjugate gradient parameters and independent of the line search satisfy in the sufficient descent condition. Furthermore, under the strong Wolfe line search, the global convergence of the proposed methods are proved. Also, the preliminary numerical results on the CUTEst collection are presented to show effectiveness of our methods.

A new hybrid conjugate gradient algorithm as a convex combination of MMWU and RMIL nonlinear problems

In our research, a new conjugate gradient (CG) method for solving nonlinear systems of equations is introduced and studied. The proposed method defined as a convex combination between MMWU and RMIL problems to derive the new scale parameter ϑ. The sufficient descent condition has been satisfied for the suggested method, when we apply strong Wolfe line search, global convergence has been satisfied. Our numerical results show that the suggested algorithm are also more efficient.

New hyrid conjugate gradient method as a convex combination of HZ and CD methods

In this paper, a new hybrid conjugate gradient algorithm is proposed for solving unconstrained optimization problems, the conjugate gradient parameter [Formula: see text] is computed as a convex combination of [Formula: see text] and [Formula: see text]. Under the wolfe line search, we prove the sufficient descent and the global convergence. Numerical results are reported to show the effectiveness of our procedure.

A Dai-Liao Hybrid Hestenes-Stiefel and Fletcher-Revees Methods for Unconstrained Optimization

Some problems have no analytical solution or too difficult to solve by scientists, engineers, and mathematicians, so the development of numerical methods to obtain approximate solutions became necessary. Gradient methods are more efficient when the function to be minimized continuously in its first derivative. Therefore, this article presents a new hybrid Conjugate Gradient (CG) method to solve unconstrained optimization problems. The method requires the first-order derivatives but overcomes the steepest descent method’s shortcoming of slow convergence and needs not to save or compute the second-order derivatives needed by the Newton method. The CG update parameter is suggested from the Dai-Liao conjugacy condition as a convex combination of Hestenes-Stiefel and Fletcher-Revees algorithms by employing an optimal modulating choice parameterto avoid matrix storage. Numerical computation adopts an inexact line search to obtain the step-size that generates a decent property, showing that the algorithm is robust and efficient. The scheme converges globally under Wolfe line search, and it’s like is suitable in compressive sensing problems and M-tensor systems.

A Modified Hybrid Conjugate Gradient Method for Unconstrained Optimization

The nonlinear conjugate gradient algorithms are a very effective way in solving large-scale unconstrained optimization problems. Based on some famous previous conjugate gradient methods, a modified hybrid conjugate gradient method was proposed. The proposed method can generate decent directions at every iteration independent of any line search. Under the Wolfe line search, the proposed method possesses global convergence. Numerical results show that the modified method is efficient and robust.

A Three-Term Gradient Descent Method with Subspace Techniques

We proposed a three-term gradient descent method that can be well applied to address the optimization problems in this article. The search direction of the obtained method is generated in a specific subspace. Specifically, a quadratic approximation model is applied in the process of generating the search direction. In order to reduce the amount of calculation and make the best use of the existing information, the subspace was made up of the gradient of the current and prior iteration point and the previous search direction. By using the subspace-based optimization technology, the global convergence result is established under Wolfe line search. The results of numerical experiments show that the new method is effective and robust.

A variant of the Levenberg-Marquardt method with adaptive parameters for systems of nonlinear equations

<abstract><p>The Levenberg-Marquardt method is one of the most important methods for solving systems of nonlinear equations and nonlinear least-squares problems. It enjoys a quadratic convergence rate under the local error bound condition. Recently, to solve nonzero-residue nonlinear least-squares problem, Behling et al. propose a modified Levenberg-Marquardt method with at least superlinearly convergence under a new error bound condtion <sup>[<xref ref-type="bibr" rid="b3">3</xref>]</sup>. To extend their results for systems of nonlinear equations, by choosing the LM parameters adaptively, we propose an efficient variant of the Levenberg-Marquardt method and prove its quadratic convergence under the new error bound condition. We also investigate its global convergence by using the Wolfe line search. The effectiveness of the new method is validated by some numerical experiments.</p></abstract>

A Spectral RMIL+ Conjugate Gradient Method for Unconstrained Optimization With Applications in Portfolio Selection and Motion Control

The Spectral conjugate gradient (SCG) methods are among the efficient variants of CG algorithms which are obtained by combining the spectral gradient parameter and CG parameter. The success of SCG methods relies on effective choices of the step-size $\alpha _{k}$ and the search direction $d_{k}$ . This paper presents an SCG method for unconstrained optimization models. The search directions generated by the new method possess sufficient descent property without the restart condition and independent of the line search procedure used. The global convergence of the new method is proved under the weak Wolfe line search. Preliminary numerical results are presented which show that the method is efficient and promising, particularly for large-scale problems. Also, the method was applied to solve the robotic motion control problem and portfolio selection problem.

The convergence properties of a new hybrid conjugate gradient parameter for unconstrained optimization models

The hybrid conjugate gradient (CG) algorithms are among the efficient modifications of the conjugate gradient methods. Some interesting features of the hybrid modifications include inherenting the nice convergence properties and efficient numerical performance of the existing CG methods. In this paper, we proposed a new hybrid CG algorithm that inherits the features of the Rivaie et al. (RMIL*) and Dai (RMIL+) conjugate gradient methods. The proposed algorithm generates a descent direction under the strong Wolfe line search conditions. Preliminary results on some benchmark problems reveal that the proposed method efficient and promising.

The Performance Analysis of a New Modification of Conjugate Gradient Parameter for Unconstrained Optimization Models

Conjugate Gradient (CG) method is the most prominent iterative mathematical technique that can be useful for the optimization of both linear and non-linear systems due to its simplicity, low memory requirement, computational cost, and global convergence properties. However, some of the classical CG methods have some drawbacks which include weak global convergence, poor numerical performance both in terms of number of iterations and the CPU time. To overcome these drawbacks, researchers proposed new variants of the CG parameters with efficient numerical results and nice convergence properties. Some of the variants of the CG method include the scale CG method, hybrid CG method, spectral CG method, three-term CG method, and many more. The hybrid conjugate gradient (CG) algorithm is among the efficient variant in the class of the conjugate gradient methods mentioned above. Some interesting features of the hybrid modifications include inherenting the nice convergence properties and efficient numerical performance of the existing CG methods. In this paper, we proposed a new hybrid CG algorithm that inherits the features of the Rivaie et al. (RMIL*) and Dai (RMIL+) conjugate gradient methods. The proposed algorithm generates a descent direction under the strong Wolfe line search conditions. Preliminary results on some benchmark problems show that the proposed method efficient and promising.

Global Convergence of Nonlinear Conjugate Gradient Coefficients with Inexact Line Search

Nonlinear conjugate gradient (CG) methods are significant for solving large-scale, unconstrained optimization problems, providing vital knowledge to determine the minimum point or optimize the objective functions. Many studies of modifications for nonlinear CG methods have been carried out to improve the performance of numerical computation and to establish global convergence properties. One of these studies is the modified CG method, which has been proposed by Rivaie et al. (2015). In this paper, we modify their work in such a way that one can obtain efficient numerical performance and global convergence properties. Due to the widespread use of the strong Wolfe line search in practice, our proposed modified method implemented its use. At the same time, to show the performance of the modified method in practice, a numerical experiment is performed. KEYWORDS Unconstrained optimization; conjugate gradient method; sufficient descent property; global convergence

An online conjugate gradient algorithm for large-scale data analysis in machine learning

In recent years, the amount of available data is growing exponentially, and large-scale data is becoming ubiquitous. Machine learning is a key to deriving insight from this deluge of data. In this paper, we focus on the large-scale data analysis, especially classification data, and propose an online conjugate gradient (CG) descent algorithm. Our algorithm draws from a recent improved Fletcher-Reeves (IFR) CG method proposed in Jiang and Jian[13] as well as a recent approach to reduce variance for stochastic gradient descent from Johnson and Zhang [15]. In theory, we prove that the proposed online algorithm achieves a linear convergence rate under strong Wolfe line search when the objective function is smooth and strongly convex. Comparison results on several benchmark classification datasets demonstrate that our approach is promising in solving large-scale machine learning problems, viewed from the points of area under curve (AUC) value and convergence behavior.

A new conjugate gradient method for acceleration of gradient descent algorithms

Abstract An accelerated of the steepest descent method for solving unconstrained optimization problems is presented. which propose a fundamentally different conjugate gradient method, in which the well-known parameter βk is computed by an new formula. Under common assumptions, by using a modified Wolfe line search, descent property and global convergence results were established for the new method. Experimental results provide evidence that our proposed method is in general superior to the classical steepest descent method and has a potential to significantly enhance the computational efficiency and robustness of the training process.

A New Family of Hybrid Conjugate Gradient Methods for Unconstrained Optimization

The conjugate gradient method is a very efficient iterative technique for solving large-scale unconstrainedoptimization problems. Motivated by recent modifications of some variants of the method and construction of hybrid methods, this study proposed four hybrid methods that are globally convergent as well as computationally efficient. The approach adopted for constructing the hybrid methods entails projecting ten recently modified conjugate gradient methods. Each of the hybrid methods is shown to satisfy the descent property independent of any line search technique and globally convergent under the influence of strong Wolfe line search. Results obtained from numerical implementation of these methods and performance profiling show that the methods are very competitive with well-known traditional methods.