Hessian Approximation Research Articles

New Two-step Conjugate Gradient Method for Unconstrained Optimization

Two-step methods are secant-like techniques of the quasi-Newton type that, unlike the classical methods, construct nonlinear alternatives to the quantities used in the so-called Secant equation. Two-step methods instead incorporate data available from the two most recent iterations and thus create an alternative to the Secant equation with the intention of creating better Hessian approximations that induce faster convergence to the minimizer of the objective function. Such methods, based on reported numerical results published in several research papers related to the subject, have introduced substantial savings in both iteration and function evaluation counts. Encouraged by the successful performance of the methods, we explore in this paper employing them in developing a new Conjugate Gradient (CG) algorithm. CG methods gain popularity on big problems and in situations when memory resources are scarce. The numerical experimentations on the new methods are encouraging and open venue for further investigation of such techniques to explore their merits in a multitude of applications.

Inexact Generalized Gauss–Newton for Scaling the Canonical Polyadic Decomposition With Non-Least-Squares Cost Functions

The canonical polyadic decomposition (CPD) allows one to extract compact and interpretable representations of tensors. Several optimization-based methods exist to fit the CPD of a tensor for the standard least-squares (LS) cost function. Extensions have been proposed for more general cost functions such as β-divergences as well. For these non-LS cost functions, a generalized Gauss-Newton (GGN) method has been developed. This is a second-order method that uses an approximation of the Hessian of the cost function to determine the next iterate and with this algorithm, fast convergence can be achieved close to the solution. While it is possible to construct the full Hessian approximation for small tensors, the exact GGN approach becomes too expensive for tensors with larger dimensions, as found in typical applications. In this paper, we therefore propose to use an inexact GGN method and provide several strategies to make this method scalable to large tensors. First, the approximation of the Hessian is only used implicitly and its multilinear structure is exploited during Hessian-vector products, which greatly improves the scalability of the method. Next, we show that by using a compressed instance of the GGN Hessian approximation, the computation time of the inexact GGN method can be lowered even more, with only limited influence on the convergence speed. We also propose dedicated preconditioners for the problem. Further, the maximum likelihood estimator for Rician distributed data is examined in detail as an example of an alternative cost function. This cost function is useful for the analysis of the moduli of complex data, as in functional magnetic resonance imaging, for instance. We compare the proposed method to the existing CPD methods and demonstrate the method's speed and effectiveness on synthetic and simulated real-life data. Finally, we show that the method can scale by using randomized block sampling.

On iteratively regularized predictor–corrector algorithm for parameter identification * *Supported by NSF awards 1818886 and 2011622 (DMS Computational Mathematics) and Russian Science Foundation project 20-11-20085.

We study a constrained optimization problem of stable parameter estimation given some noisy (and possibly incomplete) measurements of the state observation operator. In order to find a solution to this problem, we introduce a hybrid regularized predictor–corrector scheme that builds upon both, all-at-once formulation, recently developed by B. Kaltenbacher and her co-authors, and the so-called traditional route, pioneered by A. Bakushinsky. Similar to all-at-once approach, our proposed algorithm does not require solving the constraint equation numerically at every step of the iterative process. At the same time, the predictor–corrector framework of the new method avoids the difficulty of dealing with large solution spaces resulting from all-at-once make-up, which inevitably leads to oversized Jacobian and Hessian approximations. Therefore our predictor–corrector algorithm (PCA) has the potential to save time and storage, which is critical when multiple runs of the iterative scheme are carried out for uncertainty quantification. To assess numerical efficiency of novel PCA, two parameter estimation inverse problems in epidemiology are considered. All experiments are carried out with real data on COVID-19 pandemic in Netherlands and Spain.

Compressed Gradient Methods With Hessian-Aided Error Compensation

The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. The performance bottleneck, which used to be computations, is now often communications. Several gradient compression techniques have been proposed to reduce the communication load at the price of a loss in solution accuracy. Recently, it has been shown how compression errors can be compensated for in the optimization algorithm to improve the solution accuracy. Even though convergence guarantees for error-compensated algorithms have been established, there is very limited theoretical support for quantifying the observed improvements in solution accuracy. In this paper, we show that Hessian-aided error compensation, unlike other existing schemes, avoids accumulation of compression errors on quadratic problems. We also present strong convergence guarantees of Hessian-based error compensation for stochastic gradient descent. Our numerical experiments highlight the benefits of Hessian-based error compensation, and demonstrate that similar convergence improvements are attained when only a diagonal Hessian approximation is used.

Adjoint-assisted Pareto front tracing in aerodynamic and conjugate heat transfer shape optimization

In this paper, a prediction-correction algorithm, built on the method proposed in [1], uses the adjoint method to trace the Pareto front. The method is initialized by a point on the Pareto front obtained by considering one of the objectives only. During the prediction and correction steps, different systems of equations are derived and solved by treating the Karush-Kuhn-Tucker (KKT) optimality conditions in two different ways. The computation of second derivatives of the objective functions (Hessian matrix) which appear in the equations solved to update the design variables is avoided. Instead, two approaches are used: (a) the computation of Hessian-vector products driving a Krylov subspace solver and (b) the approximations of the Hessian via Quasi-Newton methods. Three different variants of the prediction-correction method are developed, applied to 2D aerodynamic shape optimization problems with geometrical constraints and compared in terms of computational cost. It is shown that the inclusion of the prediction step in the algorithm and the use of Quasi-Newton methods with Hessian approximations in both steps has the lowest computational cost. This method is, then, used to compute the Pareto front in a 3D conjugate heat transfer shape optimization problem, with the total pressure losses and max. solid temperature as the two contradicting objectives.

A Gauss–Newton-Like Hessian Approximation for Economic NMPC

Economic Model Predictive Control (EMPC) has recently become popular because of its ability to control constrained nonlinear systems while explicitly optimizing a prescribed performance criterion. Large performance gains have been reported for many applications and closed-loop stability has been recently investigated. However, computational performance still remains an open issue and only few contributions have proposed real-time algorithms tailored to EMPC. We perform a step towards computationally cheap algorithms for EMPC by proposing a new positive-definite Hessian approximation which does not hinder fast convergence and is suitable for being used within the real-time iteration (RTI) scheme. We provide two simulation examples to demonstrate the effectiveness of RTI-based EMPC relying on the proposed Hessian approximation.

A variation of Broyden class methods using Householder adaptive transforms

In this work we introduce and study novel Quasi Newton minimization methods based on a Hessian approximation Broyden Class-\textit{type} updating scheme, where a suitable matrix $\tilde{B}_k$ is updated instead of the current Hessian approximation $B_k$. We identify conditions which imply the convergence of the algorithm and, if exact line search is chosen, its quadratic termination. By a remarkable connection between the projection operation and Krylov spaces, such conditions can be ensured using low complexity matrices $\tilde{B}_k$ obtained projecting $B_k$ onto algebras of matrices diagonalized by products of two or three Householder matrices adaptively chosen step by step. Extended experimental tests show that the introduction of the adaptive criterion, which theoretically guarantees the convergence, considerably improves the robustness of the minimization schemes when compared with a non-adaptive choice; moreover, they show that the proposed methods could be particularly suitable to solve large scale problems where $L$-$BFGS$ performs poorly.

Symmetrical Rank-Three Vectorized Loading Scores Quasi-Newton for Identification of Hydrogeological Parameters and Spatiotemporal Recharges

In a multi-layered groundwater model, achieving accurate spatiotemporal identification and solving the ill-posed problem is the vital topic for model calibration. This study proposes a symmetry rank three vectorized loading scores (SR3 VLS) quasi-Newton algorithm by modifying the Levenberg–Marquardt algorithm and importing a rank three structure from Broyden–Fletcher–Goldfarb–Shanno algorithm for identification of hydrogeological parameters and spatiotemporal recharge simultaneously. To accelerate directional convergence and approach a global optimum, this study uses a vectorized limited switchable step size in the transmissive groundwater inverse problem. The Hessian approximation rank three uses high and low-rank factor loading scores analyzed from simulated storage fluctuation between adjacent iterations for calculation and matrix correction. Two numerical experiments were designed to validate the proposing algorithm, showing the SR3 VLS quasi-Newton reduced the error percentages of the identified parameters by 1.63% and 9.65% compared to the Jacobian quasi-Newton. The proposing method is applied to the Chou-Shui River alluvial fan groundwater system in Taiwan. Results show that the simulated storage error decreased rapidly in six iterations, and has good head convergence as small as 0.11% with a root-mean-square-error (RMSE) of 0.134 m, indicating that the proposing algorithm reduces the computational cost to converge to the true solution.

The spectral conjugate gradient method in variational adjoint assimilation for model terrain correction I: Theoretical frame

A spectral conjugate gradient (SCG) method is proposed within the mathematical framework of a variational adjoint assimilation system to correct the bottom terrain of a shallow-water equations model. The formulation of this method is described from the mathematical point of view with determination of the descent direction by using Andrei’s limited-memory form and the step length by solving the tangent linear model. It is proved to benefit from (i) the iterative regularization strategy, (ii) the inverse Hessian approximation involving the second-order information from Broyden-Fletcher-Goldfarb-Shanno class (BFGS), and (iii) the optimal step length. The regularization term introduced to the cost function will allow the incorporation of the known information about the desired bottom terrain and guarantee the uniqueness of the optimal solution.

Scaled projected-directions methods with application to transmission tomography

Statistical image reconstruction in X-ray computed tomography yields large-scale regularized linear least-squares problems with nonnegativity bounds, where the memory footprint of the operator is a concern. Discretizing images in cylindrical coordinates results in significant memory savings, and allows parallel operator–vector products without on-the-fly computation of the operator, without necessarily decreasing image quality. However, it deteriorates the conditioning of the operator. We improve the Hessian conditioning by way of a block-circulant scaling operator and we propose a strategy to handle nondiagonal scaling in the context of projected-directions methods for bound-constrained problems. We describe our implementation of the scaling strategy using two algorithms: TRON, a trust-region method with exact second derivatives, and L-BFGS-B, a linesearch method with a limited-memory quasi-Newton Hessian approximation. We compare our approach with one where a change of variable is made in the problem. On two reconstruction problems, our approach converges faster than the change of variable approach, and achieves much tighter accuracy in terms of optimality residual than a first-order method.

Least Conservative Linearized Constraint Formulation for Real-Time Motion Generation

Abstract Today robotics has shown many successful strategies to solve several navigation problems. However, moving into a dynamic environment is still a challenging task. This paper presents a novel method for motion generation in dynamic environments based on real-time nonlinear model predictive control (NMPC). At the core of our approach is a least conservative linearized constraint formulation built upon the real-time iteration (RTI) scheme with Gauss-Newton Hessian approximation. We demonstrate that the proposed constraint formulation is less conservative for planners based on Newton-type method than for those based on a fully converged NMPC method. Additionally, we show the performance of our approach in simulation, in a scenario where the Crazyflie nanoquadcopter avoids balls and reaches its desired goal in spite of the uncertainty about when the balls will be thrown. The numerical results validate our theoretical findings and illustrate the computational efficiency of the proposed scheme.

A Scaled Conjugate Gradient Method Based on New BFGS Secant Equation with Modified Nonmonotone Line Search

In this paper, we provide and analyze a new scaled conjugate gradient method and its performance, based on the modified secant equation of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) method and on a new modified nonmonotone line search technique. The method incorporates the modified BFGS secant equation in an effort to include the second order information of the objective function. The new secant equation has both gradient and function value information, and its update formula inherits the positive definiteness of Hessian approximation for general convex function. In order to improve the likelihood of finding a global optimal solution, we introduce a new modified nonmonotone line search technique. It is shown that, for nonsmooth convex problems, the proposed algorithm is globally convergent. Numerical results show that this new scaled conjugate gradient algorithm is promising and efficient for solving not only convex but also some large scale nonsmooth nonconvex problems in the sense of the Dolan-Moré performance profiles.

Image domain least-squares migration with a Hessian matrix estimated by non-stationary matching filters

AbstractAssuming that an accurate background velocity is obtained, least-squares migration (LSM) can be used to estimate underground reflectivity. LSM can be implemented in either the data domain or image domain. The data domain LSM (DDLSM) is not very practical because of its huge computational cost and slow convergence rate. The image domain LSM (IDLSM) might be a flexible alternative if estimating the Hessian matrix using a cheap and accurate approach. It has practical potential to analyse convenient Hessian approximation methods because the Hessian matrix is too huge to compute and save. In this paper, the Hessian matrix is approximated with non-stationary matching filters. The filters are calculated to match the conventional migration image to the demigration/remigration image. The two images are linked by the Hessian matrix. An image deblurring problem is solved with the estimated filters for the IDLSM result. The combined sparse and total variation regularisations are used to produce accurate and reasonable inversion results. The numerical experiments based on part of Sigsbee model, Marmousi model and a 2D field data set illustrate that the non-stationary matching filters can give a good approximation for the Hessian matrix, and the results of the image deblurring problem with combined regularisations can provide high-resolution and true-amplitude reflectivity estimations.

A subspace version of the Wang–Yuan Augmented Lagrangian-Trust Region method for equality constrained optimization

Subspace properties are presented for the trust-region subproblem that appears in the Augmented Lagrangian-Trust-Region method recently proposed by Wang and Yuan (2015). Specifically, when the approximate Lagrangian Hessians are updated by suitable quasi-Newton formulas, we show that any solution of the corresponding kth subproblem belongs to the subspace spanned by all gradient vectors of the objective and of the constraints computed up to iteration k. From this result, a subspace version of the referred method is proposed for large-scale equality constrained optimization problems. The subspace method is suitable to problems in which the number of constraints is much lower than the number of variables.

Mesh adaptive direct search with simplicial Hessian update

Recently a second directional derivative-based Hessian updating formula was used for Hessian approximation in mesh adaptive direct search (MADS). The approach combined with a quadratic program solver significantly improves the performance of MADS. Unfortunately it imposes some strict requirements on the position of points and the order in which they are evaluated. The subject of this paper is the introduction of a Hessian update formula that utilizes the points from the neighborhood of the incumbent solution without imposing such strict restrictions. The obtained approximate Hessian can then be used for constructing a quadratic model of the objective and the constraints. The proposed algorithm was compared to the reference implementation of MADS (NOMAD) on four sets of test problems. On all but one of them it outperformed NOMAD. The biggest performance difference was observed on constrained problems. To validate the algorithm further the approach was tested on several real-world optimization problems arising from yield approximation and worst case analysis in integrated circuit design. On all tested problems the proposed approach outperformed NOMAD.

A robust multi-batch L-BFGS method for machine learning

ABSTRACTThis paper describes an implementation of the L-BFGS method designed to deal with two adversarial situations. The first occurs in distributed computing environments where some of the computational nodes devoted to the evaluation of the function and gradient are unable to return results on time. A similar challenge occurs in a multi-batch approach in which the data points used to compute function and gradients are purposely changed at each iteration to accelerate the learning process. Difficulties arise because L-BFGS employs gradient differences to update the Hessian approximations, and when these gradients are computed using different data points the updating process can be unstable. This paper shows how to perform stable quasi-Newton updating in the multi-batch setting, studies the convergence properties for both convex and non-convex functions, and illustrates the behaviour of the algorithm in a distributed computing platform on binary classification logistic regression and neural network training problems that arise in machine learning.

An improved weight-constrained neural network training algorithm

In this work, we propose an improved weight-constrained neural network training algorithm, named iWCNN. The proposed algorithm exploits the numerical efficiency of the L-BFGS matrices together with a gradient-projection strategy for handling the bounds on the weights. Additionally, an attractive property of iWCNN is that it utilizes a new scaling factor for defining the initial Hessian approximation used in the L-BFGS formula. Since the L-BFGS Hessian approximation is defined utilizing a small number of correction vector pairs, our motivation is to further exploit them in order to increase the efficiency of the training algorithm and the convergence rate of the minimization process. The preliminary numerical experiments provide empirical evidence that the proposed training algorithm accelerates the training process.

Subsampled inexact Newton methods for minimizing large sums of convex functions

Abstract This paper deals with the minimization of a large sum of convex functions by inexact Newton (IN) methods employing subsampled functions, gradients and Hessian approximations. The conjugate gradient method is used to compute the IN step and global convergence is enforced by a nonmonotone line-search procedure. The aim is to obtain methods with affordable costs and fast convergence. Assuming strongly convex functions, R-linear convergence and worst-case iteration complexity of the procedure are investigated when functions and gradients are approximated with increasing accuracy. A set of rules for the forcing parameters and subsample Hessian sizes are derived that ensure local q-linear/q-superlinear convergence of the proposed method. The random choice of the Hessian subsample is also considered and convergence in the mean square, both for finite and infinite sums of functions, is proved. Finally, the analysis of global convergence with asymptotic R-linear rate is extended to the case of the sum of convex functions and strongly convex objective function. Numerical results on well-known binary classification problems are also given. Adaptive strategies for selecting forcing terms and Hessian subsample size, streaming out of the theoretical analysis, are employed and the numerical results show that they yield effective IN methods.

New Dual Parameter Quasi-Newton Methods for Unconstrained Nonlinear Programs

A framework model of multi-step quasi-Newton methods developed which utilizes values of the objective function. The model is constructed using iteration genereted data from the m+1 most recent iterates/gradient evaluations. It hosts double free parameters which introduce a certain degree of flexibility. This permits the interpolating polynomials to exploit available computed function values which are otherwise discarded and left unused. Two new algorithms are derived for those function values incorporated in the update of the inverse Hessian approximation at each iteration to accelerate convergence. The idea of incorporating function values configure quasi-Newton methods, but the presentation constitutes a new approach for such algorithms. Several earlier works only include function values data in the update of the Hessian approximation numerically to improve the convergence of Secant-like methods. The methods are a useful tool for solving nonlinear problems arising in engineering, physics, machine learning, decision science, approximations techniques to Bayesian Regressors and a variety of numerical analysis applications.

Trust-region algorithms for training responses: machine learning methods using indefinite Hessian approximations

ABSTRACTMachine learning (ML) problems are often posed as highly nonlinear and nonconvex unconstrained optimization problems. Methods for solving ML problems based on stochastic gradient descent are easily scaled for very large problems but may involve fine-tuning many hyper-parameters. Quasi-Newton approaches based on the limited-memory Broyden-Fletcher-Goldfarb-Shanno (BFGS) update typically do not require manually tuning hyper-parameters but suffer from approximating a potentially indefinite Hessian with a positive-definite matrix. Hessian-free methods leverage the ability to perform Hessian-vector multiplication without needing the entire Hessian matrix, but each iteration's complexity is significantly greater than quasi-Newton methods. In this paper we propose an alternative approach for solving ML problems based on a quasi-Newton trust-region framework for solving large-scale optimization problems that allow for indefinite Hessian approximations. Numerical experiments on a standard testing data set show that with a fixed computational time budget, the proposed methods achieve better results than the traditional limited-memory BFGS and the Hessian-free methods.