Abstract
The limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) optimization method performs very efficiently for large-scale problems. A trust region search method generally performs more efficiently and robustly than a line search method, especially when the gradient of the objective function cannot be accurately evaluated. The computational cost of an L-BFGS trust region subproblem (TRS) solver depend mainly on the number of unknown variables (n) and the number of variable shift vectors and gradient change vectors (m) used for Hessian updating, withm << nfor large-scale problems. In this paper, we analyze the performances of different methods to solve the L-BFGS TRS. The first method is the direct method using the Newton-Raphson (DNR) method and Cholesky factorization of a densen×nmatrix, the second one is the direct method based on an inverse quadratic (DIQ) interpolation, and the third one is a new method that combines the matrix inversion lemma (MIL) with an approach to update associated matrices and vectors. The MIL approach is applied to reduce the dimension of the original problem withnvariables to a new problem withmvariables. Instead of directly using expensive matrix-matrix and matrix-vector multiplications to solve the L-BFGS TRS, a more efficient approach is employed to update matrices and vectors iteratively. The L-BFGS TRS solver using the MIL method performs more efficiently than using the DNR method or DIQ method. Testing on a representative suite of problems indicates that the new method can converge to optimal solutions comparable to those obtained using the DNR or DIQ method. Its computational cost represents only a modest overhead over the well-known L-BFGS line-search method but delivers improved stability in the presence of inaccurate gradients. When compared to the solver using the DNR or DIQ method, the new TRS solver can reduce computational cost by a factor proportional ton2/mfor large-scale problems.
Highlights
Decision-making tools based on optimization procedures have been successfully applied to solve practical problems in a wide range of areas
We present three different methods to solve the limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) trust region subproblem (TRS): the direct method using the Newton-Raphson (DNR) method, the direct method using inverse quadratic (DIQ) interpolation approach proposed by Gao et al [32], and the technique using matrix inversion lemma (MIL) together with an efficient matrix updating algorithm
The Algorithm to Solve the L-BFGS TRS Using the Matrix Inversion Lemma Given the trust region size Δ Δ(k) > 0, threshold of convergence δcr > 0, maximum number of iterations allowed NTRS,max > 0, the scaling factor α > αcr > 0, m ≥ 0, n × m matrix V [S(k), Z(k)], m × m matrices P P(k) and WI [W(k)]− 1, m-dimensional vector u u(k) and n-dimensional gradient vector g g(k) evaluated at the current best solution x(k), both the Lagrange multiplier λ*and the trust region search step s* s(λ*) can be solved from the TRS defined in Eq 8 using the matrix inversion lemma (MIL) as summarized in Algorithm-5
Summary
Decision-making tools based on optimization procedures have been successfully applied to solve practical problems in a wide range of areas. Because multiple search points are generated in each iteration, multiple optimal solutions can be found in parallel These distributed optimization methods are referred to as multiplethread optimization methods. (the optimization thread index) in the following discussions Both DGN and DQN optimization methods approximate the objective function by a quadratic model, and they are designed for problems with smooth objective function and continuous variables. Their convergence is not guaranteed for problems with integer type variables (e.g., well location optimization), if those integers can be treated as truncated continuous variables, our numerical tests indicate that these distributed optimization methods can improve the objective function significantly and locate multiple suboptimal solutions for problems with integer type variables in only a few iterations. The Hessian H(k+1) updated using Eq 2 is guaranteed positive definite if the condition ε(k) [z(k)]T s(k) > c3z(k)s(k) is satisfied where c3>0 is a small positive number
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
