Limited Memory BFGS Method Research Articles

Parallel-in-time multiple shooting for optimal control problems governed by the Navier–Stokes equations

In the past decades, multiple shooting methods have proven to be a promising direction to speed up the optimization process, especially in the context of ODE-based optimization. Very recently, Fang et al. (Journal of Computational Physics, vol. 452, 110926, 2022) proposed a multiple shooting algorithm for large-scale PDE-constrained optimization. The current paper continues this line of work and explores the potential of multiple shooting methods for optimal control problems governed by the three-dimensional Navier–Stokes equations. Similar to Fang et al., the augmented Lagrangian (AL) method is used to solve the resulting equality-constrained optimization problem, and we employ the classical limited-memory BFGS method for the unconstrained subproblems inside the AL loop. In the current work, we exploit the multiple shooting paradigm in full by processing the shooting windows parallel-in-time, allowing for significant parallel speed-ups compared to single shooting. The proposed method is validated on a velocity tracking case, using up to 100 windows. Our analysis shows that the multiple shooting algorithm allows for considerable algorithmic and parallel speed-ups. While algorithmic speed-up depends on the exact tracking case and initialization of the shooting windows, the multiple shooting algorithm always outperforms single shooting in terms of computational time (if the number of windows is sufficiently high) due to the parallel-in-time implementation. For a given amount of resources, we also show that the proposed parallel-in-time strategy can outperform spatial parallelization alone, especially when the spatial parallelization is saturated.

Three-Dimensional Limited-Memory BFGS Inversion of Magnetic Data Based on a Multiplicative Objective Function

At present, the traditional magnetic three-dimensional inversion method has been fully developed and is widely used. Magnetic exploration is a kind of geophysical exploration method that uses the magnetic field changes (magnetic anomalies) caused by the magnetic differences between various rocks in the crust to find useful mineral resources and study the underground structure. Traditional magnetic three-dimensional inversion is relatively inefficient. Moreover, the traditional additive objective function (data fitting difference term plus regularization term and logarithmic obstacle term), which causes the regularization factor selection to be more complicated, is implemented in this method. Therefore, it is necessary to establish a new efficient three-dimensional magnetic inversion algorithm and optimize the selection of regularization factors. In this paper, based on the limited-memory BFGS (L-BFGS) method, the three-dimensional magnetic inversion of a multiplicative objective function is realized. The inversion test is conducted in this paper using both theoretical synthesis data and measured data. The results demonstrate that the limited-memory BFGS method significantly enhances the inversion efficiency and yields superior inversion outcomes compared to traditional magnetic three-dimensional inversion methods. Additionally, the multiplicative objective function-based three-dimensional magnetic inversion method simplifies the process of selecting weight factors for regularization terms.

An improved inversion method for shale elastic parameters measurement based on ultrasonic resonance spectroscopy

The accurate measurement of elastic parameters of shale cores has important guidance for the exploration and development of shale oil and gas. The traditional shale acoustic experiment requires multiple samples from different angles and its preparation process is complex. Resonant Ultrasound Spectroscopy has the advantages of repeatable measurement, high accuracy, and the ability to obtain all elastic parameters from a single small sample. However, the influence of clamping force and measuring angle needs comprehensive analysis to ensure that formant can be measured as many and accurately as possible. This research first conducts sensitivity analysis on the elastic parameters of different shale to guide the experiment. In addition, due to the more complex inversion objective function of the anisotropic shale compared to isotropic samples, the traditional Levenberg–Marquardt method has a strong dependence on the initial value and is prone to falling into local optimal solution. The L-BFGS (limited memory BFGS) method is introduced into the inversion of anisotropic elastic parameters of shale, which can significantly improve the inversion efficiency. Finally, the robustness and reliability of the inversion method are validated through theoretical and experimental measurement results. [Work supported by the National Natural Science Foundation of China (42004117 and 42127807).]

Three-Dimensional Joint Inversion of the Resistivity Method and Time-Domain-Induced Polarization Based on the Cross-Gradient Constraints

The resistivity method and time-domain-induced polarization (TDIP) are two branches of electric exploration that are used to solve problems in mineral exploration, hydrogeology and engineering geology. In recent years, integrating different physical parameters for joint inversion to improve the accuracy of inversion results has been extensively examined; however, three-dimensional joint inversion of the two methods above has not been realized. To further address this issue, in this research, we used the limited-memory BFGS (L-BFGS) method to develop a three-dimensional joint inversion algorithm of the resistivity method and TDIP based on the cross-gradient constraints. In the new algorithm, the resistivity method and TDIP inversion were iteratively updated alternately to ensure that the inversion results can simultaneously meet the two conditions of obtaining minimum data misfits and finding structural similarity. The three-dimensional synthetic dataset inversion results showed that the models obtained by joint inversion are more accurate in the recovery of both the boundaries and the values of the anomalies. Especially in the background of high noise, joint inversion has higher resolution for the target body. The joint inversion algorithm was also successfully applied to a groundwater detection practice in Beijing, China, in which the practicability of the algorithm was confirmed.

Classification of CO Environmental Parameter for Air Pollution Monitoring with Grammatical Evolution

Air pollution is a pressing concern in urban areas, necessitating the critical monitoring of air quality to understand its implications for public health. Internet of Things (IoT) devices are widely utilized in air pollution monitoring due to their sensor capabilities and seamless data transmission over the Internet. Artificial intelligence (AI) and machine learning techniques play a crucial role in classifying patterns derived from sensor data. Environmental stations offer a multitude of parameters that can be obtained to uncover hidden patterns showcasing the impact of pollution on the surrounding environment. This paper focuses on utilizing the CO parameter as an indicator of pollution in two datasets collected from wireless environmental monitoring devices in the greater Port area and the Town Hall of Igoumenitsa City in Greece. The datasets are normalized to facilitate their utilization in classification algorithms. The k-means algorithm is applied, and the elbow method is used to determine the optimal number of clusters. Subsequently, the datasets are introduced to the grammatical evolution algorithm to calculate the percentage fault. This method constructs classification programs in a human-readable format, making it suitable for analysis. Finally, the proposed method is compared against four state-of-the-art models: the Adam optimizer for optimizing artificial neural network parameters, a genetic algorithm for training an artificial neural network, the Bayes model, and the limited-memory BFGS method applied to a neural network. The comparison reveals that the GenClass method outperforms the other approaches in terms of classification error.

Full waveform inversion based on hybrid conjugate gradient with BFGS direction

Full waveform inversion (FWI) has the problem of low computational efficiency. The main reasons are low convergence rate and unstable convergence. We mainly use optimization methods to solve computationally inefficient problems. Among the commonly used optimization methods, the L-BFGS (Limited memory Broyden-Fletcher-Goldfarb-Shanno) method converges fast but not stably, while the HCG (Hybrid Conjugate gradient) method converges stably but slowly. Neither of these methods can maximize the computational efficiency of FWI. By using Gauss-Newton direction to approximate the search direction (QN-HCG) of HCG method, the second-order convergence rate of Gauss-Newton method and the strong convergence stability of HCG method can be used at the same time. However, QN-HCG requires the Hessian matrix to be positive definite, which is complicated to compute and occupies a large memory. The memoryless BFGS method is developed from the L-BFGS method. Since the Hessian matrix is not required to be positive definite, the method is simple to compute and occupies less memory, and is therefore more suitable to approximate the search direction of the HCG method. First, in order to improve the convergence speed of FWI and ensure the stability of convergence, we use BFGS without memory variable to approximate the search direction of HCG method (BFGS-HCG), and introduce this method into FWI. Second, the computational efficiency of the proposed method is verified in model tests of acoustic and elastic wave FWI. Model tests show that this approach can improve the computational efficiency of full waveform inversion.

Studies on modified limited-memory BFGS method in full waveform inversion

Full waveform inversion (FWI) is a non-linear optimization problem based on full-wavefield modeling to obtain quantitative information of subsurface structure by minimizing the difference between the observed seismic data and the predicted wavefield. The limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) method is an effective quasi-Newton method in FWI due to its high inversion efficiency with low calculation and storage requirements. Like other conventional quasi-Newton methods, the approximation of the Hessian matrix in the L-BFGS method satisfies the quasi-Newton equation, which only exploits the gradient and model information while the available objective function value is neglected. The modified quasi-Newton equation considers the gradient, model, and objective function information together. Theoretical analysis reveals that the modified quasi-Newton equation is superior to the conventional quasi-Newton equation as it achieves higher-order accuracy in approximating the Hessian matrix. The modified L-BFGS method can be obtained by using the modified quasi-Newton equation to modify the L-BFGS method. This modification improves the accuracy of the Hessian matrix approximation with little increase of calculation for each iteration. We incorporate the modified L-BFGS method into FWI, numerical results show that the modified L-BFGS method has a higher convergence rate, achieves better inversion results, and has stronger anti-noise ability than the conventional L-BFGS method.

An inexact projected gradient method with rounding and lifting by nonlinear programming for solving rank-one semidefinite relaxation of polynomial optimization

We consider solving high-order and tight semidefinite programming (SDP) relaxations of nonconvex polynomial optimization problems (POPs) that often admit degenerate rank-one optimal solutions. Instead of solving the SDP alone, we propose a new algorithmic framework that blends local search using the nonconvex POP into global descent using the convex SDP. In particular, we first design a globally convergent inexact projected gradient method (iPGM) for solving the SDP that serves as the backbone of our framework. We then accelerate iPGM by taking long, but safeguarded, rank-one steps generated by fast nonlinear programming algorithms. We prove that the new framework is still globally convergent for solving the SDP. To solve the iPGM subproblem of projecting a given point onto the feasible set of the SDP, we design a two-phase algorithm with phase one using a symmetric Gauss–Seidel based accelerated proximal gradient method (sGS-APG) to generate a good initial point, and phase two using a modified limited-memory BFGS (L-BFGS) method to obtain an accurate solution. We analyze the convergence for both phases and establish a novel global convergence result for the modified L-BFGS that does not require the objective function to be twice continuously differentiable. We conduct numerical experiments for solving second-order SDP relaxations arising from a diverse set of POPs. Our framework demonstrates state-of-the-art efficiency, scalability, and robustness in solving degenerate SDPs to high accuracy, even in the presence of millions of equality constraints.

Polyphase waveform design for ISRJ suppression based on L-BFGS method

In order to eliminate the false-target peaks caused by the interrupted sampling repeater jamming (ISRJ) and increase the true target detection ability in pulse compression, a polyphase radar waveform consisting of an effective time discontinuous waveform (TDW) and protection pulse (PP) is proposed in this paper. In the waveform design procedure, the ISRJ parameters, including the sampling pulse duration, sampling period and retransmitting delay, are estimated using the sliding-truncation matched filter algorithm. Based on the estimation results, the orthogonal PP and TDW are designed to fill the ISRJ sampling and retransmitting durations according to the jamming operator matrix. The phase code of radar transmitted waveform is calculated iteratively to maximize the orthogonality between the PP and TDW using the limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) algorithm. A mismatched filter for the TDW waveform is optimized to further suppress the false-target peak caused by the ISRJ. Numerical simulations with different jamming parameters are given to demonstrate the efficiency of the proposed method, the designed waveform exhibits a moderate target signal integration loss and a large ISRJ suppression, thereby achieves a high anti-ISRJ performance. The false-target peaks of ISRJ are eliminated and the target signal is retained simultaneously.

Fast-Converging and Low-Complexity Linear Massive MIMO Detection With L-BFGS Method

In massive multiple-input multiple-output (MIMO) uplink, linear minimum mean square error (MMSE) detection achieves near-optimal performance but suffers from undesirable computational burden due to the high-dimensional matrix inversion involved. To address this issue, the limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) quasi-Newton method is introduced in this paper to realize free-matrix-inversion MMSE detection in an iterative way with significant reduction in computational complexity from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(K^{3})$</tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(L K^{2})$</tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$K$</tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L$</tex-math></inline-formula> denote the number of user antennas and iterations, respectively. For the better performance and complexity trade-off, three low-complexity L-BFGS based linear MMSE detection algorithms (namely E-LBFGS, I-LBFGS and S-LBFGS) are proposed successively. Furthermore, an efficient initialization strategy for the L-BFGS method based MMSE detection is devised, which helps to accelerate convergence and improve performance with acceptable complexity overhead added compared to the conventional L-BFGS based detection algorithms. Simulation results finally validate that the proposed detection scheme based on the L-BFGS method can closely approach to the MMSE accuracy with a small number of iterations even in poor propagation environments and provide attractive trade-offs between performance and complexity.

Limited Memory BFGS Method for Least Squares Semidefinite Programming with Banded Structure

Limited Memory BFGS Method for Least Squares Semidefinite Programming with Banded Structure

Nonmonotone diagonally scaled limited-memory BFGS methods with application to compressive sensing based on a penalty model

According to a minimization problem founded upon the Byrd–Nocedal measure function coupled with a penalty term of the secant equation, a diagonal quasi–Newton updating formula is given. Then, the proposed update is applied as the initial (inverse) Hessian estimation of the limited-memory BFGS (Broyden–Fletcher–Goldfarb–Shanno) method. An improved version of the method is also given based on the Li–Fukushima modified secant equation. Convergence analysis for the general functions is conducted based on a nonmonotone Armijo line search. Eventually, practical advantages of the methods are computationally appraised through some CUTEr functions and also, on the compressive sensing problem.

QNG: A Quasi-Natural Gradient Method for Large-Scale Statistical Learning

Natural gradient method provides a powerful paradigm for training statistical models and offers several appealing theoretic benefits. It constructs the Fisher information matrix to correct the ordinary gradients, and thus, the cost may become prohibitively expensive in the large-scale setting. This paper proposes a quasi-natural gradient method to address this computational issue. It employs a rank-one model to capture the second-order information from the underlying statistical manifold and to iteratively update the Fisher approximations. A three-loop procedure is designed to implement the updating formulas. This procedure resembles the classical two-loop procedure in the limited-memory BFGS method but saves half of the memory usage while it can be made faster. The resulting method retains the convergence rate advantages of existing stochastic optimization methods and has inherent ability in handling nonconvexity. Numerical studies conducted on several machine learning tasks demonstrate the reduction in convergence time and the robustness in tackling nonconvexity relative to stochastic gradient descent and the online limited-memory BFGS method.

A regularized limited memory BFGS method for large-scale unconstrained optimization and its efficient implementations

The limited memory BFGS (L-BFGS) method is one of the popular methods for solving large-scale unconstrained optimization. Since the standard L-BFGS method uses a line search to guarantee its global convergence, it sometimes requires a large number of function evaluations. To overcome the difficulty, we propose a new L-BFGS with a certain regularization technique. We show its global convergence under the usual assumptions. In order to make the method more robust and efficient, we also extend it with several techniques such as the nonmonotone technique and simultaneous use of the Wolfe line search. Finally, we present some numerical results for test problems in CUTEst, which show that the proposed method is robust in terms of solving more problems.

Diagonal BFGS updates and applications to the limited memory BFGS method

Diagonal BFGS updates and applications to the limited memory BFGS method

An Efficient Linear Detection Scheme Based on L-BFGS Method for Massive MIMO Systems

For massive multiple-input multiple-output (MIMO) systems, minimum mean square error (MMSE) detection is near-optimal, but requires high-complexity matrix inversion. To avoid matrix inversion, we formulate MMSE detection as a strictly convex quadratic optimization problem, which can be solved iteratively by the recognized most efficient Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method. According to special properties of massive MIMO systems, we propose a novel limited-memory BFGS (L-BFGS) scheme for MMSE detection with one correction search, unit step length, and simplified initialization, which can greatly reduce the storage and computation cost compared to BFGS method. Simulation results finally verify the effectiveness of the proposed scheme.

Pre-conditioned BFGS-based uncertainty quantification in elastic full-waveform inversion

SUMMARYFull-waveform inversion has become an essential technique for mapping geophysical subsurface structures. However, proper uncertainty quantification is often lacking in current applications. In theory, uncertainty quantification is related to the inverse Hessian (or the posterior covariance matrix). Even for common geophysical inverse problems its calculation is beyond the computational and storage capacities of the largest high-performance computing systems. In this study, we amend the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm to perform uncertainty quantification for large-scale applications. For seismic inverse problems, the limited-memory BFGS (L-BFGS) method prevails as the most efficient quasi-Newton method. We aim to augment it further to obtain an approximate inverse Hessian for uncertainty quantification in FWI. To facilitate retrieval of the inverse Hessian, we combine BFGS (essentially a full-history L-BFGS) with randomized singular value decomposition to determine a low-rank approximation of the inverse Hessian. Setting the rank number equal to the number of iterations makes this solution efficient and memory-affordable even for large-scale problems. Furthermore, based on the Gauss–Newton method, we formulate different initial, diagonal Hessian matrices as pre-conditioners for the inverse scheme and compare their performances in elastic FWI applications. We highlight our approach with the elastic Marmousi benchmark model, demonstrating the applicability of pre-conditioned BFGS for large-scale FWI and uncertainty quantification.

Limited-memory BFGS with displacement aggregation

A displacement aggregation strategy is proposed for the curvature pairs stored in a limited-memory BFGS (a.k.a. L-BFGS) method such that the resulting (inverse) Hessian approximations are equal to those that would be derived from a full-memory BFGS method. This means that, if a sufficiently large number of pairs are stored, then an optimization algorithm employing the limited-memory method can achieve the same theoretical convergence properties as when full-memory (inverse) Hessian approximations are stored and employed, such as a local superlinear rate of convergence under assumptions that are common for attaining such guarantees. To the best of our knowledge, this is the first work in which a local superlinear convergence rate guarantee is offered by a quasi-Newton scheme that does not either store all curvature pairs throughout the entire run of the optimization algorithm or store an explicit (inverse) Hessian approximation. Numerical results are presented to show that displacement aggregation within an adaptive L-BFGS scheme can lead to better performance than standard L-BFGS.

Nonlinear Conjugate Gradient Methods for PDE Constrained Shape Optimization Based on Steklov--Poincaré-Type Metrics

Shape optimization based on shape calculus has received a lot of attention in recent years, particularly regarding the development, analysis, and modification of efficient optimization algorithms. In this paper we propose and investigate nonlinear conjugate gradient methods based on Steklov-Poincar\'e-type metrics for the solution of shape optimization problems constrained by partial differential equations. We embed these methods into a general algorithmic framework for gradient-based shape optimization methods and discuss the numerical discretization of the algorithms. We numerically compare the proposed nonlinear conjugate gradient methods to the already established gradient descent and limited memory BFGS methods for shape optimization on several benchmark problems. The results show that the proposed nonlinear conjugate gradient methods perform well in practice and that they are an efficient and attractive addition to already established gradient-based shape optimization algorithms.

A Stochastic Quasi-Newton Method for Large-Scale Nonconvex Optimization With Applications.

Ensuring the positive definiteness and avoiding ill conditioning of the Hessian update in the stochastic Broyden-Fletcher-Goldfarb-Shanno (BFGS) method are significant in solving nonconvex problems. This article proposes a novel stochastic version of a damped and regularized BFGS method for addressing the above problems. While the proposed regularized strategy helps to prevent the BFGS matrix from being close to singularity, the new damped parameter further ensures the positivity of the product of correction pairs. To alleviate the computational cost of the stochastic limited memory BFGS (LBFGS) updates and to improve its robustness, the curvature information is updated using the averaged iterate at spaced intervals. The effectiveness of the proposed method is evaluated through the logistic regression and Bayesian logistic regression problems in machine learning. Numerical experiments are conducted by using both synthetic data set and several real data sets. The results show that the proposed method generally outperforms the stochastic damped LBFGS (SdLBFGS) method. In particular, for problems with small sample sizes, our method has shown superior performance and is capable of mitigating ill-conditioned problems. Furthermore, our method is more robust to the variations of the batch size and memory size than the SdLBFGS method.