quasi-Newton Step Research Articles

Full-low evaluation methods for derivative-free optimization

We propose a new class of rigorous methods for derivative-free optimization with the aim of delivering efficient and robust numerical performance for functions of all types, from smooth to non-smooth, and under different noise regimes. To this end, we have developed a class of methods, called Full-Low Evaluation methods, organized around two main types of iterations. The first iteration type (called Full-Eval) is expensive in function evaluations, but exhibits good performance in the smooth and non-noisy cases. For the theory, we consider a line search based on an approximate gradient, backtracking until a sufficient decrease condition is satisfied. In practice, the gradient was approximated via finite differences, and the direction was calculated by a quasi-Newton step (BFGS). The second iteration type (called Low-Eval) is cheap in function evaluations, yet more robust in the presence of noise or non-smoothness. For the theory, we consider direct search, and in practice we use probabilistic direct search with one random direction and its negative. A switch condition from Full-Eval to Low-Eval iterations was developed based on the values of the line-search and direct-search stepsizes. If enough Full-Eval steps are taken, we derive a complexity result of gradient-descent type. Under failure of Full-Eval, the Low-Eval iterations become the drivers of convergence yielding non-smooth convergence. Full-Low Evaluation methods are shown to be efficient and robust in practice across problems with different levels of smoothness and noise.

Learning to Optimize Molecular Geometries Using Reinforcement Learning.

Though quasi-Newton methods have been widely adopted in computational chemistry software for molecular geometry optimization, it is well known that these methods might not perform well for initial guess geometries far away from the local minima, where the quadratic approximation might be inaccurate. We propose a reinforcement learning approach to develop a model that produces a correction term for the quasi-Newton step calculated with the BFGS algorithm to improve the overall optimization performance. Our model is able to complete the optimization in about 30% fewer steps than pure BFGS for molecules starting from perturbed geometries. The new method has similar convergence to BFGS when complemented with a line search procedure, but it is much faster since it avoids the multiple gradient evaluations associated with line searches.

An active set quasi-Newton method with projection step for monotone nonlinear equations

In this paper, an active set quasi-Newton method for bound constrained nonlinear equation is proposed. By using this active set technique, we only need to solve a reduced dimension linear equation at each iteration to generate the search direction. The algorithm is a combination of the quasi-Newton method and projection method. Firstly we use the quasi-Newton step as the trial step and then use a projection technique to generate the next iteration point. Our key observation is that the algorithm generates a bounded iteration sequence automatically even if the bounds are equal to infinity and the global convergence is obtained in the sense that the whole sequence converges to the stationary point. The numerical tests show the efficiency of the algorithm.

Hybrid reconstruction method for multispectral bioluminescence tomography with log-sum regularization.

Bioluminescence tomography (BLT) has important applications in the in vivo visualization of a pathological process for preclinical studies. However, the reconstruction of BLT is severely ill-posed. To recover the bioluminescence source stably and efficiently, we use a log-sum regularization term in the objective function and utilize a hybrid optimization algorithm for solving the nonconvex regularized problems (HONOR). The hybrid optimization scheme of HONOR merges second-order information and first-order information to reconstruction by choosing either the quasi-Newton (QN) or gradient descent step at each iteration. The QN step uses the limited-memory Broyden-Fletcher-Goldfarb-Shanno algorithm (L-BFGS) to acquire second-order information. Simulations and in vivo experiments based on multispectral measurements demonstrated the remarkable performance of the proposed hybrid method in the sparse reconstruction of BLT.

An ℓ1-penalized adaptive normalized quasi-newton algorithm for sparsity-aware generalized eigen-subspace tracking

This paper presents an ℓ1-penalized extension of the adaptive normalized quasi-Newton algorithm (Nguyen and Yamada, 2013) which was established for online generalized eigenvalue problem. The proposed extension aims to exploit effectively the sparsity as a priori knowledge for efficient subspace tracking in signal processing and is also motivated by recent sparsity-aware eigenvector analysis in data sciences, e.g., sparse principal component analysis. For such an extension, we newly introduce ℓ1 penalty into a non-convex criterion which has been used to characterize, as its stationary point, the generalized eigen-pair. The proposed subspace tracking algorithm is derived by applying a quasi-Newton type step to the new criterion followed by a normalization step. A convergence analysis is given in the case for decaying weight of the penalty. We also discuss potential applications, e.g., online sparse principal component analysis, by controlling the weight sequence of the ℓ1 penalty. Numerical experiments demonstrate that the proposed algorithm (i) can improve the subspace tracking performance even for noisy observation of random vectors whose covariance matrix pencil has sparse principal generalized eigenvector and (ii) can promote the interpretability of the estimate of the principal generalized eigenvector.

Direct search nonsmooth constrained optimization via rounded ℓ1 penalty functions

ABSTRACT A class of direct search methods for locally minimizing a Lipschitz continuous black-box function f subject to locally Lipschitz constraints is presented. A sequence of smoothed penalty functions is used. Each smoothed penalty function is approximately minimized in turn. The smoothing is reduced after each minimization, exposing the exact penalty function in the limit. Convergence to a constrained Clarke stationary point is shown under appropriate regularity conditions. Convergence to one or more KKT points is shown under similar conditions when f and all active constraints are strictly differentiable at each limit point. An implementation of one method in this class is numerically tested and shown to be effective in practice. The implementation uses a discrete quasi-Newton step when possible. Otherwise a global direction search is used to locate a descent direction. Theoretical convergence properties are independent of the quasi-Newton step.

Quasi-Newton algorithm for optimal approximate linear regression design: Optimization in matrix space

Given a linear regression model and an experimental region for the independent variable, the problem of finding an optimal approximate design calls for minimizing a convex optimality criterion over a convex set of information matrices of feasible approximate designs. For numerical solution pure gradient methods are often used by design theorists, as vertex direction, vertex exchange, multiplicative algorithms, or combinations hereof. These methods have two major deficiencies: a slow convergence rate after a quick but rough approximation to the optimum, and often a large support of the obtained nearly optimal design. The latter feature is related to the fact that the methods optimize in the space of design measures which is usually of high or even infinite dimension, whereas the dimension of the information matrices is often small or moderate. For such situations a quasi-Newton method is revisited which was originally established by Gaffke & Heiligers (1996). In the present paper new possibilities of its application are demonstrated. The algorithm optimizes in matrix space. It shows a good global and an excellent local convergence behavior resulting in an accurate approximation of the optimum. A crucial subroutine solves convex quadratic minimization over the set of information matrices via repeated linear minimization over that set, providing thus the quasi-Newton step of the algorithm. This may also be of interest as a tool for computing an approximate design from a given information matrix and such that the support size of the design keeps Carathéodory’s bound. Illustrations are given for D- and I-optimality in particular multivariate random coefficient regression models and for T-optimal discriminating design in univariate polynomial models. Moreover, the behavior of the algorithm is tested for cases of larger dimensions: D- and I-optimal design for a third order polynomial model in several variables on a discretized cube.

Smoothed $$\ell _1$$ ℓ 1 -regularization-based line search for sparse signal recovery

$$\ell _1$$ -regularization-based sparse recovery has received a considerable attention over the last decade. In this paper, a solver called SQNSR is proposed to recover signals with high dynamic range. SQNSR utilizes linear search strategy and quasi-Newton step to the solve composite objective function for the sparse recovery problem. Since $$\ell _1$$ -norm-regularized item is nonsmooth, smoothing technique is introduced to obtain an approximate smoothed function. The sufficient and necessary condition is also derived for the feasible smoothed objective function. By limiting the step length in each iteration, the convergence of SQNSR is guaranteed to obtain the sparsest solution. Numerical simulations are implemented to test the performance of the proposed approach and verify the theoretical analysis.

Geometry Optimization with Multilayer Methods Using Least-Squares Minimization.

In this article, we introduce a least-squares minimization scheme for optimizing molecular structures with mixed quantum mechanics (QM) and molecular mechanics (MM) multilayer models. A mixed-coordinate optimization framework was developed. The QM and MM regions are modeled with redundant internal coordinates and Cartesian coordinates, respectively. Within this mixed-coordinate system, a least-squares minimization method using the quasi-Newton step as the evaluation of error is constructed. The couplings between layers are treated rigidly in accordance with the mechanical embedding approach, and the MM Hessian is approximated as a scalar constant of the root-mean-square QM Hessian eigenvalues. Both two-layer and three-layer models were tested. The performance of the method developed herein shows consistently stable and fast convergence.

A fast truncated Lagrange method for large-scale image restoration problems

In this work, we present a new method for the restoration of images degraded by noise and spatially invariant blur. In the proposed method, the original image restoration problem is replaced by an equality constrained minimization problem. A quasi-Newton method is applied to the first-order optimality conditions of the constrained problem. In each quasi-Newton iteration, the hessian of the Lagrangian is approximated by a circulant matrix and the Fast Fourier Transform is used to compute the quasi-Newton step. The quasi-Newton iteration is terminated according to the discrepancy principle. Results of numerical experiments are presented to illustrate the effectiveness and usefulness of the proposed method.

Numerical Experience with Multiple Update Quasi-Newton Methods for Unconstrained Optimization

The authors have derived what they termed quasi-Newton multi step methods in [2]. These methods have demonstrated substantial numerical improvements over the standard single step Secant-based BFGS. Such methods use a variant of the Secant equation that the updated Hessian (or its inverse) satisfies at each iteration. In this paper, new methods will be explored for which the updated Hessians satisfy multiple relations of the Secant-type. A rational model is employed in developing the new methods. The model hosts a free parameter which is exploited in enforcing symmetry on the updated Hessian approximation matrix thus obtained. The numerical performance of such techniques is then investigated and compared to other methods. Our results are encouraging and the improvements incurred supercede those obtained from other existing methods at minimal extra storage and computational overhead.

Density matrix search using direct inversion in the iterative subspace as a linear scaling alternative to diagonalization in electronic structure calculations

For electronic structure calculations on large systems, solving the self-consistent-field (SCF) equations is one of the key bottlenecks. Density matrix search methods provide an efficient linear scaling approach for circumventing the traditional O(N3) diagonalization procedure for solving the SCF equations. The conjugate gradient density matrix search (CG-DMS) method is a successful implementation of this approach. An alternative density matrix search method, QN–DMS, employs direct inversion in the iterative subspace using a quasi-Newton (QN) step to estimate the error vector. For linear polyglycine chains of 10–100 residues, the present approach scales linearly and is 30% faster than CG-DMS. For clusters of up to 300 water molecules, this method shows smoother and efficient convergence, and displays nearly linear scaling.

A Modified Trust Region Algorithm

In this article, we propose an algorithm which solves unconstrained optimization problems by combining the trust region method with the quasi-Newton line search method. During consecutive trust region steps, when the quasi-Newton matrix B k approximates the Hessian matrix of the objective function at x k well, the algorithm tries a full quasi-Newton step and, if this step is successful, continues trying quasi-Newton steps. On the other hand, when employing the line search method, if the quasi-Newton direction is nearly orthogonal to the gradient of the objective function or the step size becomes too small, the algorithm switches back to trust region steps. Convergence properties of the algorithm are proved, and numerical results are presented.

Proximal quasi-Newton methods for nondifferentiable convex optimization

. The method is based on Rockafellar’s proximal point algorithm and a cutting-plane technique. At each step, we use an approximate proximal point pa(xk) of xk to define a vk∈∂ekf(pa(xk)) with ek≤α∥vk∥, where α is a constant. The method monitors the reduction in the value of ∥vk∥ to identify when a line search on f should be used. The quasi-Newton step is used to reduce the value of ∥vk∥. Without the differentiability of f, the method converges globally and the rate of convergence is Q-linear. Superlinear convergence is also discussed to extend the characterization result of Dennis and More. Numerical results show the good performance of the method.

An extension of the theory of secant preconditioners

A theory of inexact Newton methods with secant preconditioners for solving large nonlinear systems of equations has been developed recently by Martínez ( Math. Comput., 1993). According to this theory, local and superlinear convergence with bounded work per iteration of the inexact Newton method is obtained if the first trial increment at each iteration is a suitable quasi-Newton step computed using least-change secant-update procedures. The Jacobian approximation is interpreted as a preconditioner of the iterative linear method. In this paper, we extend the theory in two ways. On the one hand, since in many iterative methods the true residual is not computed but the preconditioned residual is, we show how to stop the linear iteration using the preconditioned residual instead of the original one. On the other hand, we introduce damping parameters that modify the usual unitary secant step. Two natural damping parameters are introduced, one of them tries to reduce the true residual and the other one tries to reduce the preconditioned residual. We prove that the main results of the theory of secant preconditioners hold under these modifications.

SOR-Secant Methods

A family of SOR-secant methods for solving large-scale nonlinear systems of equations is introduced. The components and the variables of the system are divided into m blocks. At each cycle of the method, the groups of components are changed one at a time using a quasi-Newton (least-change secant) step. Proofs of local convergence at an ideal rate are given, which use the theory of fixed-point quasi-Newton methods [J.M. Martínez, SIAM J. Numer. Anal., 29 (1992), pp. 1413–1434]. Numerical experiments are presented.

A quasi-Newton algorithm for first-order saddle-point location

A new algorithm for the location of a transition-state structure on an energy hypersurface is proposed. The method is compared to three other quasi-Newton step calculations available in literature. Numerical results derived from several examples are compared to those obtained by the two algorithms implemented in the Gaussian package.

Issues in nonlinear equation solving in chemical engineering

Ten full-matrix problems and thirteen sparse-matrix problems, all based on chemical engineering problems, are used to perform a comprehensive test of iterative nonlinear equation solving techniques, in particular the techniques used for computing the correction step, and for updating sparse Jacobian matrices. Comparisons are made on the basis of both efficiency and reliability. Results indicate that Powell's dogleg correction step performs slightly better than the quasi-Newton step. Schubert's sparse Jacobian update performs very well, but the best overall is Bogle and Perkins's new least-relative-change update, at least when combined with the dogleg correction step.

A Pointwise Quasi-Newton Method for Integral Equations

For fully nonlinear integral equations, conventional quasi-Newton methods, such as Broyden’s method, may not possess good convergence properties. In this paper we present a new method that gives superlinear convergence for such integral equations and provides for savings over Newton’s method in the cost of solution of the linear equation for the quasi-Newton step.

The Conjugate Gradient Method and Trust Regions in Large Scale Optimization

Algorithms based on trust regions have been shown to be robust methods for unconstrained optimization problems. All existing methods, either based on the dogleg strategy or Hebden-More iterations, require solution of system of linear equations. In large scale optimization this may be prohibitively expensive. It is shown in this paper that an approximate solution of the trust region problem may be found by the preconditioned conjugate gradient method. This may be regarded as a generalized dogleg technique where we asymptotically take the inexact quasi-Newton step. We also show that we have the same convergence properties as existing methods based on the dogleg strategy using an approximate Hessian.