Methods For Large-scale Unconstrained Optimization Research Articles

A New Dai-Liao Conjugate Gradient Method based on Approximately Optimal Stepsize for Unconstrained Optimization

Conjugate gradient methods are a class of very effective iterative methods for large-scale unconstrained optimization. In this paper, a new Dai-Liao conjugate gradient method for solving large-scale unconstrained optimization problem is proposed. Based on the approximately optimal stepsize for the gradient method, we derive three new choices for the important parameters tk in Dai-Liao conjugate gradient method. The search direction satisfies the sufficient descent condition, and the global convergences of the proposed method for uniformly convex and general functions are proved under some mild conditions. Numerical experiments on a set of test problems from the CUTEst library show that the proposed method is superior to some well-known conjugate gradient methods.

A regularized limited memory BFGS method for large-scale unconstrained optimization and its efficient implementations

The limited memory BFGS (L-BFGS) method is one of the popular methods for solving large-scale unconstrained optimization. Since the standard L-BFGS method uses a line search to guarantee its global convergence, it sometimes requires a large number of function evaluations. To overcome the difficulty, we propose a new L-BFGS with a certain regularization technique. We show its global convergence under the usual assumptions. In order to make the method more robust and efficient, we also extend it with several techniques such as the nonmonotone technique and simultaneous use of the Wolfe line search. Finally, we present some numerical results for test problems in CUTEst, which show that the proposed method is robust in terms of solving more problems.

A note on memory-less SR1 and memory-less BFGS methods for large-scale unconstrained optimization

A note on memory-less SR1 and memory-less BFGS methods for large-scale unconstrained optimization

Two-versions of descent conjugate gradient methods for large-scale unconstrained optimization

<p>The conjugate gradient methods are noted to be exceedingly valuable for solving large-scale unconstrained optimization problems since it needn't the storage of matrices. Mostly the parameter conjugate is the focus for conjugate gradient methods. The current paper proposes new methods of parameter of conjugate gradient type to solve problems of large-scale unconstrained optimization. A Hessian approximation in a diagonal matrix form on the basis of second and third-order Taylor series expansion was employed in this study. The sufficient descent property for the proposed algorithm are proved. The new method was converged globally. This new algorithm is found to be competitive to the algorithm of fletcher-reeves (FR) in a number of numerical experiments.</p>

An improvement of adaptive cubic regularization method for unconstrained optimization problems

In this paper, we present two nonmonotone versions of adaptive cubic regularized (ARC) method for unconstrained optimization problems. The proposed methods are a combination of the ARC algorithm with the nonmonotone line search methods introduced by Zhang and Hager [A nonmonotone line search technique and its application to unconstrained optimization, SIAM J. Optim. 14 (2004), pp. 1043–1056] and Ahookhosh et al. [A nonmonotone trust-region line search method for large-scale unconstrained optimization, Appl. Math. Model. 36 (2012), pp. 478–487]. The global convergence analysis for these iterative algorithms is established under suitable conditions. Several numerical examples are given to illustrate the efficiency and robustness of the newly suggested methods. The obtained results show the satisfactory performance of the proposed algorithms when compared to the basic ARC algorithm.

A Spectral Conjugate Gradient Method with Descent Property

Spectral conjugate gradient method (SCGM) is an important generalization of the conjugate gradient method (CGM), and it is also one of the effective numerical methods for large-scale unconstrained optimization. The designing for the spectral parameter and the conjugate parameter in SCGM is a core work. And the aim of this paper is to propose a new and effective alternative method for these two parameters. First, motivated by the strong Wolfe line search requirement, we design a new spectral parameter. Second, we propose a hybrid conjugate parameter. Such a way for yielding the two parameters can ensure that the search directions always possess descent property without depending on any line search rule. As a result, a new SCGM with the standard Wolfe line search is proposed. Under usual assumptions, the global convergence of the proposed SCGM is proved. Finally, by testing 108 test instances from 2 to 1,000,000 dimensions in the CUTE library and other classic test collections, a large number of numerical experiments, comparing with both SCGMs and CGMs, for the presented SCGM are executed. The detail results and their corresponding performance profiles are reported, which show that the proposed SCGM is effective and promising.

A Class of Approximate Inverse Preconditioners Based on Krylov-Subspace Methods for Large-Scale Nonconvex Optimization

We introduce a class of positive definite preconditioners for the solution of large symmetric indefinite linear systems or sequences of such systems, in optimization frameworks. The preconditioners are iteratively constructed by collecting information on a reduced eigenspace of the indefinite matrix by means of a Krylov-subspace solver. A spectral analysis of the preconditioned matrix shows the clustering of some eigenvalues and possibly the nonexpansion of its spectrum. Extensive numerical experimentation is carried out on standard difficult linear systems and by embedding the class of preconditioners within truncated Newton methods for large-scale unconstrained optimization (the issue of major interest). Although the Krylov-based method may provide modest information on matrix eigenspaces, the results obtained show that the proposed preconditioners lead to substantial improvements in terms of efficiency and robustness, particularly on very large nonconvex problems.

A new accelerated conjugate gradient method for large-scale unconstrained optimization

In this paper, we present a new conjugate gradient method using an acceleration scheme for solving large-scale unconstrained optimization. The generated search direction satisfies both the sufficient descent condition and the Dai–Liao conjugacy condition independent of line search. Moreover, the value of the parameter contains more useful information without adding more computational cost and storage requirements, which can improve the numerical performance. Under proper assumptions, the global convergence result of the proposed method with a Wolfe line search is established. Numerical experiments show that the given method is competitive for unconstrained optimization problems, with a maximum dimension of 100,000.

A new modified scaled conjugate gradient method for large-scale unconstrained optimization with non-convex objective function

In this paper, according to the fifth-order Taylor expansion of the objective function and the modified secant equation suggested by Li and Fukushima, a new modified secant equation is presented. Also, a new modification of the scaled memoryless BFGS preconditioned conjugate gradient algorithm is suggested which is the idea to compute the scaling parameter based on a two-point approximation of our new modified secant equation. A remarkable feature of the proposed method is that it possesses a globally convergent even without convexity assumption on the objective function. Numerical results show that the proposed new modification of scaled conjugate gradient is efficient.

Properties of the block BFGS update and its application to the limited-memory block BNS method for unconstrained minimization

A block version of the Broyden–Fletcher–Goldfarb–Shanno (BFGS) variable metric update formula and its modifications are investigated. In spite of the fact that this formula satisfies the quasi-Newton conditions with all used difference vectors and that the improvement of convergence is the best one in some sense for quadratic objective functions, for general functions, it does not guarantee that the corresponding direction vectors are descent directions. To overcome this difficulty, but at the same time utilize the advantageous properties of the block BFGS update, a block version of the limited-memory variable metric BNS method for large-scale unconstrained optimization is proposed. The global convergence of the algorithm is established for convex sufficiently smooth functions. Numerical experiments demonstrate the efficiency of the new method.

A new spectral conjugate gradient method for large-scale unconstrained optimization

The spectral conjugate gradient methods, with simple construction and nice numerical performance, are a kind of effective methods for solving large-scale unconstrained optimization problems. In this paper, based on quasi-Newton direction and quasi-Newton condition, and motivated by the idea of spectral conjugate gradient method as well as Dai-Kou's selecting technique for conjugate parameter [SIAM J. Optim. 23 (2013), pp. 296–320], a new approach for generating spectral parameters is presented, where a new double-truncating technique, which can ensure both the sufficient descent property of the search directions and the bounded property of the sequence of spectral parameters, is introduced. Then a new associated spectral conjugate gradient method for large-scale unconstrained optimization is proposed. Under either the strong Wolfe line search or the generalized Wolfe line search, the proposed method is always globally convergent. Finally, a large number of comparison numerical experiments on large-scale instances from one thousand to two million variables are reported. The numerical results show that the proposed method is more promising.

Accelerated diagonal gradient-type method for large-scale unconstrained optimization

In this study, we propose an accelerated diagonal-updating scheme for solving large-scale optimization, where a scaled diagonal matrix is used to approximate the Hessian. We combine an accelerator with the diagonal-updating method to improve the efficiency of the algorithm. This accelerator is employed to ensure that the function value can be reduced significantly at each step. Moreover, the algorithm employs a suitable monotone strategy to guarantee the global convergence of the algorithm. Several numerical results are reported, which demonstrate that the proposed method is promising and more robust than other diagonal updating schemes.

A modified limited-memory BNS method for unconstrained minimization based on the conjugate directions idea

A modification of the limited-memory variable metric BNS method for large-scale unconstrained optimization is proposed, which consists in corrections (derived from the idea of conjugate directions) of the used difference vectors for better satisfaction of the previous quasi-Newton (QN) conditions. In comparison with [Vlček and Lukšan, A conjugate directions approach to improve the limited-memory BFGS method, Appl. Math. Comput. 219 (2012), pp. 800–809], where a similar approach is used, correction vectors from more previous iterations can be applied here. For quadratic objective functions, the improvement of convergence is the best one in some sense, all stored corrected difference vectors are conjugate and the QN conditions with these vectors are satisfied. Global convergence of the algorithm is established for convex sufficiently smooth functions. Numerical experiments demonstrate the efficiency of the new method.

A new edition of conjugate gradient methods for large-scale unconstrained optimization

A new edition of conjugate gradient methods for large-scale unconstrained optimization

A regularized limited memory BFGS method for nonconvex unconstrained minimization

The limited memory BFGS method (L-BFGS) is an adaptation of the BFGS method for large-scale unconstrained optimization. However, The L-BFGS method need not converge for nonconvex objective functions and it is inefficient on highly ill-conditioned problems. In this paper, we proposed a regularization strategy on the L-BFGS method, where the used regularization parameter may play a compensation role in some sense when the condition number of Hessian approximation tends to become ill-conditioned. Then we proposed a regularized L-BFGS method and established its global convergence even when the objective function is nonconvex. Numerical results show that the proposed method is efficient.

A Self-Adjusting Spectral Conjugate Gradient Method for Large-Scale Unconstrained Optimization

This paper presents a hybrid spectral conjugate gradient method for large-scale unconstrained optimization, which possesses a self-adjusting property. Under the standard Wolfe conditions, its global convergence result is established. Preliminary numerical results are reported on a set of large-scale problems in CUTEr to show the convergence and efficiency of the proposed method.

Recent Studies on Nonlinear Conjugate Gradient Methods for Large-scale Unconstrained Optimization

Recent Studies on Nonlinear Conjugate Gradient Methods for Large-scale Unconstrained Optimization

A nonmonotone trust-region line search method for large-scale unconstrained optimization

We consider an efficient trust-region framework which employs a new nonmonotone line search technique for unconstrained optimization problems. Unlike the traditional nonmonotone trust-region method, our proposed algorithm avoids resolving the subproblem whenever a trial step is rejected. Instead, it performs a nonmonotone Armijo-type line search in direction of the rejected trial step to construct a new point. Theoretical analysis indicates that the new approach preserves the global convergence to the first-order critical points under classical assumptions. Moreover, superlinear and quadratic convergence are established under suitable conditions. Numerical experiments show the efficiency and effectiveness of the proposed approach for solving unconstrained optimization problems.

A new two-step gradient-type method for large-scale unconstrained optimization

In this paper, we propose some improvements on a new gradient-type method for solving large-scale unconstrained optimization problems, in which we use data from two previous steps to revise the current approximate Hessian. The new method which we considered, resembles to that of Barzilai and Borwein (BB) method. The innovation features of this approach consist in using approximation of the Hessian in diagonal matrix form based on the modified weak secant equation rather than the multiple of the identity matrix in the BB method. Using this approach, we can obtain a higher order accuracy of Hessian approximation when compares to other existing BB-type method. By incorporating a simple monotone strategy, the global convergence of the new method is achieved. Practical insights into the effectiveness of the proposed method are given by numerical comparison with the BB method and its variant.

Comparison of advanced large-scale minimization algorithms for the solution of inverse ill-posed problems

We compare the performance of several robust large-scale minimization algorithms for the unconstrained minimization of an ill-posed inverse problem. The parabolized Navier–Stokes equation model was used for adjoint parameter estimation. The methods compared consist of three versions of nonlinear conjugate-gradient (CG) method, quasi-Newton Broyden–Fletcher–Goldfarb–Shanno (BFGS), the limited-memory quasi-Newton (L-BFGS) [D.C. Liu and J. Nocedal, On the limited memory BFGS method for large scale minimization, Math. Program. 45 (1989), pp. 503–528], truncated Newton (T-N) method [S.G. Nash, Preconditioning of truncated Newton methods, SIAM J. Sci. Stat. Comput. 6 (1985), pp. 599–616, S.G. Nash, Newton-type minimization via the Lanczos method, SIAM J. Numer. Anal. 21 (1984), pp. 770–788] and a new hybrid algorithm proposed by Morales and Nocedal [J.L. Morales and J. Nocedal, Enriched methods for large-scale unconstrained optimization, Comput. Optim. Appl. 21 (2002), pp. 143–154]. For all the methods employed and tested, the gradient of the cost function is obtained via an adjoint method. A detailed description of the algorithmic form of minimization algorithms employed in the minimization comparison is provided. For the inviscid case, the CG-descent method of Hager [W.W. Hager and H. Zhang, A new conjugate gradient method with guaranteed descent and efficient line search, SIAM J. Optim. 16 (1) (2005), pp. 170–192] performed the best followed closely by the hybrid method [J.L. Morales and J. Nocedal, Enriched methods for large-scale unconstrained optimization, Comput. Optim. Appl. 21 (2002), pp. 143–154], while in the viscous case, the hybrid method emerged as the best performed followed by CG [D.F. Shanno and K.H. Phua, Remark on algorithm 500. Minimization of unconstrained multivariate functions, ACM Trans. Math. Softw. 6 (1980), pp. 618–622] and CG-descent [W.W. Hager and H. Zhang, A new conjugate gradient method with guaranteed descent and efficient line search, SIAM J. Optim. 16 (1) (2005), pp. 170–192]. This required an adequate choice of parameters in the CG-descent method as well as controlling the number of L-BFGS and T-N iterations to be interlaced in the hybrid method.