Large-scale Unconstrained Optimization Research Articles

New scaled algorithm for non-linear conjugate gradients in unconstrained optimization

<p>A new scaled conjugate gradient (SCG) method is proposed throughout this paper, the SCG technique may be a special important generalization conjugate gradient (CG) method, and it is an efficient numerical method for solving nonlinear large scale unconstrained optimization. As a result, we proposed the new SCG method with a strong Wolfe condition (SWC) line search is proposed. The proposed technique's descent property, as well as its global convergence property, are satisfied without the use of any line searches under some suitable assumptions. The proposed technique's efficiency and feasibility are backed up by numerical experiments comparing them to traditional CG techniques.</p>

A Descent Four-Term of Liu and Storey Conjugate Gradient Method for Large Scale Unconstrained Optimization Problems

The conjugate gradient (CG) method is a useful tool for obtaining the optimum point for unconstrained optimization problems since it does not require a second derivative or its approximations. Moreover, the conjugate gradient method can be applied in many fields such as machine learning, deep learning, neural network, and many others. This paper constructs a four-term conjugate gradient method that satisfies the descent property and convergence properties to obtain the stationary point. The new modification was constructed based on Liu and Storey's conjugate gradient method, two-term conjugate gradient method, and three-term conjugate gradient method. To analyze the efficiency and robustness, we used more than 150 optimization functions from the CUTEst library with different dimensions and shapes. The numerical results show that the new modification outperforms the recent conjugate gradient methods such as CG-Descent, Dai and Liao, and others in terms of number of functions evaluations, number of gradient evaluations, number of iterations, and CPU time.

Global convergence of a descent PRP type conjugate gradient method for nonconvex optimization

Nonlinear conjugate gradient methods are often used to solve large-scale unconstrained optimization problems owing to their lower memory requirement and less computation cost. In this paper, we propose a new Polak-Ribiére-Polyak type conjugate gradient method, which satisfies the sufficient descent condition independent of any line search. A remarkable property about this method is its strong global convergence with the standard Wolfe line search conditions as well as the standard Armijo line search strategy without convexity assumption of the objective function. The numerical results demonstrating the efficiency of the proposed method are reported.

A Modified Perry's Conjugate Gradient Method Based on Powell's Equation for Solving Large-Scale Unconstrained Optimization

It is known that the conjugate gradient method is still a popular method for many researchers who are focused in solving the large-scale unconstrained optimization problems and nonlinear equations because the method avoids the computation and storage of some matrices so the memory's requirements of the method are very small. In this work, a modified Perry conjugate gradient method which fulfills a global convergence with standard assumptions is shown and analyzed. The idea of new method is based on Perry method by using the equation which is founded via Powell in 1978. The weak Wolfe–Powell search conditions are used to choose the optimal line search, under the line search and suitable conditions, we prove both descent and sufficient descent conditions. In particular, numerical results show that the new conjugate gradient method is more effective and competitive when compared to other of standard conjugate gradient methods including: - CG- Hestenes and Stiefel (H/S) method, CG-Perry method, CG- Dai and Yuan (D/Y) method. The comparison is completed under a group of standard test problems with various dimensions from the CUTEst test library and the comparative performances of the methods are evaluated by total the number of iterations and the total number of function evaluations.

On large-scale unconstrained optimization and arbitrary regularization

On large-scale unconstrained optimization and arbitrary regularization

Another Three-Term Conjugate Gradient Method Close to the Memoryless BFGS for Large-Scale Unconstrained Optimization Problems

Another Three-Term Conjugate Gradient Method Close to the Memoryless BFGS for Large-Scale Unconstrained Optimization Problems

A note on memory-less SR1 and memory-less BFGS methods for large-scale unconstrained optimization

A note on memory-less SR1 and memory-less BFGS methods for large-scale unconstrained optimization

A Modified Liu and Storey Conjugate Gradient Method for Large Scale Unconstrained Optimization Problems

The conjugate gradient method is one of the most popular methods to solve large-scale unconstrained optimization problems since it does not require the second derivative, such as Newton’s method or approximations. Moreover, the conjugate gradient method can be applied in many fields such as neural networks, image restoration, etc. Many complicated methods are proposed to solve these optimization functions in two or three terms. In this paper, we propose a simple, easy, efficient, and robust conjugate gradient method. The new method is constructed based on the Liu and Storey method to overcome the convergence problem and descent property. The new modified method satisfies the convergence properties and the sufficient descent condition under some assumptions. The numerical results show that the new method outperforms famous CG methods such as CG-Descent 5.3, Liu and Storey, and Dai and Liao. The numerical results include the number of iterations and CPU time.

An efficient hybrid conjugate gradient method with descent properties under strong Wolfe line search

The hybrid conjugate gradient parameters are among the efficient variants of conjugate gradient (CG) methods for solving large-scale unconstrained optimization problems. This is due to their nice convergence properties and low memory requirements. In this paper, we present a new hybrid conjugate gradient method based on famous CG algorithms for large-scale unconstrained optimization. The proposed hybrid CG method can generate a descent search direction at each iteration provided the strong Wolfe line search is employed. Numerical results have been presented which show that the proposed method is efficient and promising.

Two-versions of descent conjugate gradient methods for large-scale unconstrained optimization

<p>The conjugate gradient methods are noted to be exceedingly valuable for solving large-scale unconstrained optimization problems since it needn't the storage of matrices. Mostly the parameter conjugate is the focus for conjugate gradient methods. The current paper proposes new methods of parameter of conjugate gradient type to solve problems of large-scale unconstrained optimization. A Hessian approximation in a diagonal matrix form on the basis of second and third-order Taylor series expansion was employed in this study. The sufficient descent property for the proposed algorithm are proved. The new method was converged globally. This new algorithm is found to be competitive to the algorithm of fletcher-reeves (FR) in a number of numerical experiments.</p>

A new three-term spectral conjugate gradient algorithm with higher numerical performance for solving large scale optimization problems based on Quasi-Newton equation

A new spectral three-term conjugate gradient algorithm in virtue of the Quasi-Newton equation is developed for solving large-scale unconstrained optimization problems. It is proved that the search directions in this algorithm always satisfy a sufficiently descent condition independent of any line search. Global convergence is established for general objective functions if the strong Wolfe line search is used. Numerical experiments are employed to show its high numerical performance in solving large-scale optimization problems. Particularly, the developed algorithm is implemented to solve the 100 benchmark test problems from CUTE with different sizes from 1000 to 10,000, in comparison with some similar ones in the literature. The numerical results demonstrate that our algorithm outperforms the state-of-the-art ones in terms of less CPU time, less number of iteration or less number of function evaluation.

A new CG algorithm based on a scaled memoryless BFGS update with adaptive search strategy, and its application to large-scale unconstrained optimization problems

Conjugate gradient method is an effective method for solving large-scale unconstrained optimization problems. This paper proposes a new conjugate gradient algorithm based on the self-scaling memoryless BFGS update, which uses the Wolfe line search. The descent and global convergence of the method are given under mild conditions. Numerical experiments show that our new conjugate gradient method is effective.

A variant of Dai-Yuan conjugate gradient method for unconstrained optimization and its application in portfolio selection

The quasi-Newton (QN) method are among the efficient variants of conjugate gradient (CG) method for solving unconstrained optimization problems. The QN method utilizes the gradients of the function while ignoring the available value information at every iteration. In this paper, we extended the Dai-Yuan [39] coefficient in designing a new CG method for large-scale unconstrained optimization problems. An interesting feature of our method is that its algorithm not only uses the available gradient value, but also consider the function value information. The global convergence of the proposed method was established under some suitable Wolfe conditions. Extensive numerical computation have been carried out which show that the average performance of this new algorithm is efficient and promising. In addition, the proposed method was extended to solve practical application problem of portfolio selection.

An Efficient Modified AZPRP Conjugate Gradient Method for Large-Scale Unconstrained Optimization Problem

To find a solution of unconstrained optimization problems, we normally use a conjugate gradient (CG) method since it does not cost memory or storage of second derivative like Newton’s method or Broyden–Fletcher–Goldfarb–Shanno (BFGS) method. Recently, a new modification of Polak and Ribiere method was proposed with new restart condition to give a so-call AZPRP method. In this paper, we propose a new modification of AZPRP CG method to solve large-scale unconstrained optimization problems based on a modification of restart condition. The new parameter satisfies the descent property and the global convergence analysis with the strong Wolfe-Powell line search. The numerical results prove that the new CG method is strongly aggressive compared with CG_Descent method. The comparisons are made under a set of more than 140 standard functions from the CUTEst library. The comparison includes number of iterations and CPU time.

A New Conjugate Gradient Method with Sufficient Descent Property

In this paper, by linearly combining the numerator and denominator terms of the Dai-Liao (DL) and Bamigbola-Ali-Nwaeze (BAN) conjugate gradient methods (CGMs), a general form of DL-BAN method has been proposed. From this general form, a new hybrid CGM, which was found to possess a sufficient descent property is generated. Numerical experiment was carried out on the new CGM in comparison with four existing CGMs, using some set of large scale unconstrained optimization problems. The result showed a superior performance of new method over majority of the existing methods.

A Modified Hybrid Conjugate Gradient Method for Unconstrained Optimization

The nonlinear conjugate gradient algorithms are a very effective way in solving large-scale unconstrained optimization problems. Based on some famous previous conjugate gradient methods, a modified hybrid conjugate gradient method was proposed. The proposed method can generate decent directions at every iteration independent of any line search. Under the Wolfe line search, the proposed method possesses global convergence. Numerical results show that the modified method is efficient and robust.

A decent three term conjugate gradient method with global convergence properties for large scale unconstrained optimization problems

<abstract> <p>The conjugate gradient (CG) method is a method to solve unconstrained optimization problems. Moreover CG method can be applied in medical science, industry, neural network, and many others. In this paper a new three term CG method is proposed. The new CG formula is constructed based on DL and WYL CG formulas to be non-negative and inherits the properties of HS formula. The new modification satisfies the convergence properties and the sufficient descent property. The numerical results show that the new modification is more efficient than DL, WYL, and CG-Descent formulas. We use more than 200 functions from CUTEst library to compare the results between these methods in term of number of iterations, function evaluations, gradient evaluations, and CPU time.</p> </abstract>

An inexact Newton method for unconstrained total variation-based image denoising by approximate addition

The Inexact Newton method using the conjugate gradient has been widely used to solve large-scale unconstrained optimization problems, such as the total variation-based image denoising. To improve energy and latency, this paper initially proposes an additional step length parameter (), such that the required number of iterations (and therefore its energy dissipation) decreases. Then, a floating-point adder (32-bits) made of approximate or truncated cells is utilized to reduce the energy dissipation in each iteration. These two techniques are finally combined to reduce the total processing time and energy dissipation for image denoising. The results show that significantly reduces the number of iterations; the proposed technique is tested on a set of images taken from a public domain library and is found that when 1.39<<1.45, the number of iterations is the lowest. Moreover, the energy dissipation of the denoising algorithm decreases by applying an approximate adder at a very small loss of accuracy and quality of the output image; the number of iterations remains constant when the number of approximate or truncated cells in the least significant positions (given by so-called NAB) is below 10.

A HS-PRP-Type Hybrid Conjugate Gradient Method with Sufficient Descent Property.

In this paper, based on the HS method and a modified version of the PRP method, a hybrid conjugate gradient (CG) method is proposed for solving large-scale unconstrained optimization problems. The CG parameter generated by the method is always nonnegative. Moreover, the search direction possesses the sufficient descent property independent of line search. Utilizing the standard Wolfe–Powell line search rule to yield the stepsize, the global convergence of the proposed method is shown under the common assumptions. Finally, numerical results show that the proposed method is promising compared with two existing methods.

A variant of Dai-Yuan conjugate gradient method for unconstrained optimization and its application in portfolio selection

The quasi-Newton (QN) method are among the efficient variants of conjugate gradient (CG) method for solving unconstrained optimization problems. The QN method utilizes the gradients of the function while ignoring the available value information at every iteration. In this paper, we extended the Dai-Yuan [39] coefficient in designing a new CG method for large-scale unconstrained optimization problems. An interesting feature of our method is that its algorithm not only uses the available gradient value, but also consider the function value information. The global convergence of the proposed method was established under some suitable Wolfe conditions. Extensive numerical computation have been carried out which show that the average performance of this new algorithm is efficient and promising. In addition, the proposed method was extended to solve practical application problem of portfolio selection.