Limited Memory quasi-Newton Method Research Articles

Three-Dimensional Separate and Joint Inversions of Multi-Component Frequency-Domain Airborne Electromagnetic Data: Synthetic Model Studies.

Airborne electromagnetic (AEM) surveys using airborne mobile platforms enable rapid and efficient exploration of areas where groundwork is difficult. They have been widely used in fields such as shallow resource exploration and environmental engineering. Three-dimensional AEM inversion is the main technique used in fine structural interpretation. However, most current methods focus on separate component data inversions, which limit the kinds of structures that can be recovered in the inversion results. To address this issue, a method for the robust 3D joint inversion of multicomponent frequency-domain AEM data was developed in this study. First, a finite element method based on unstructured tetrahedral grids was used to solve the forward problem of frequency-domain AEM data for both isotropic and anisotropic media. During inversion, a limited-memory quasi-Newton (L-BFGS) method was used to reduce the memory requirements and enable the joint inversion of large-scale multicomponent AEM data. The effectiveness of our algorithm was demonstrated using synthetic models for both isotropic and anisotropic cases, with 5% Gaussian noise added to the modeling data to simulate the measured data for separate and joint inversions. The results of the synthetic models show that joint inversion has advantages over separate inversion in that it enables the recovery of finer underground structures and provides a novel approach for the fine interpretation of frequency-domain AEM data.

Joint inversion of two-dimensional magnetotelluric and ambient noise data constrained with cross gradient

We present an alternating joint two-dimensional inversion algorithm for imaging integrated magnetotelluric (MT) and ambient noise dispersion data sets. This approach is based on the limited-memory quasi-Newton method and enforced with cross-gradient constraints that seeks the geometrically similar structural features from multi-geophysical data. A classic synthetic model with same geometric boundaries has been used to test the stability and effectiveness of the joint inversion algorithm. Compared with individual MT and ambient noise tomography (ANT) inversion results, the resistivity and velocity models are captured with higher resolution, especially the electric structures below conductive units are recovered more correctly. The algorithm is also performed on another synthetic example with partial inconsistent boundaries in the resistivity and velocity models more analogous to the realistic scenario. The result demonstrates that, at the inconsistent boundaries, the resistivity model is still imaged correctly but the velocity model is constructed with some artificial structures. We suggest that, in the application of the actual case, the resistivity model obtained from joint inversion is superior and the velocity model should be evaluated comprehensively from individual and joint inversion results.

Cold and Wet Mantle Transition Zone Beneath the Mediterranean Sea Revealed by the Electrical Image

A three-dimensional electrical conductivity model of the mantle beneath North Africa and Southern Europe is obtained by geomagnetic depth sounding. C-responses are estimated from geomagnetic data observed at observatories in and around the region and converted to the electrical structure of the mantle transition zone. The limited-memory quasi-Newton method is chosen to minimize the nonlinear objective function of inversion, while the forward modeling relies on a staggered-grid finite difference method in the spherical coordinate system. The data misfit term of the inversion function is measured using the L1-norm in order to suppress the response instability caused by the significant noise contained in the observed data. In order to adequately correct the ocean effect in observatories near the coast, a fixed shell comprised of ocean and land is incorporated in inversion. A banded zone with reduced conductivity is present in the three-dimensional model, primarily seen in the lower mantle transition zone and lower mantle beneath the Mediterranean Sea. Combining laboratory-measured conductivity models, we propose that subducted slabs causing reduced temperature and a water reservoir in the mantle transition zone should be responsible for the observed electrical model.

Wavelet-Based Three-Dimensional Inversion for Geomagnetic Depth Sounding

The complexity of Earth’s structure poses a challenge to the multiscale detection capability of geophysics. In this paper, we present a new wavelet-based three-dimensional inversion method for geomagnetic depth sounding. This method is based on wavelet functions to transfer model parameters in the space domain into the wavelet domain. The model is represented by wavelet coefficients containing both large- and fine-scale information, enabling wavelet-based inversion to describe multiscale anomalies. L1-norm measurement is applied to measure the model roughness to accomplish the sparsity constraint in the wavelet domain. Meanwhile, a staggered-grid finite difference method in a spherical coordinate system is used to calculate the forward responses, and the limited-memory quasi-Newton method is applied to seek the solution of the inversion objective function. Inversion tests of synthetic data for multiscale models show that wavelet-based inversion is stable and has multiresolution. Although higher-order wavelets can lead to finer results, our tests present that a db6 wavelet is suitable for geomagnetic depth sounding inversion. The db6 inversion results of responses at 129 geomagnetic observatories around the world reveal a higher-resolution image of the mantle.

Two-dimensional anisotropic magnetotelluric inversion using a limited-memory quasi-Newton method

We have developed a 2D anisotropic magnetotelluric (MT) inversion algorithm that uses a limited-memory quasi-Newton (QN) method for bounds-constrained optimization. This algorithm solves the inverse problem, which is nonlinear, by iterative minimization of linearized approximations of the classic Tikhonov regularized objective function. The QN approximation for the Hessian matrix is only implemented for the data-misfit term of the objective function; the part of the Hessian matrix for the regularization is explicitly computed. This adjustment results in a better approximation for the data-misfit term in particular. The inversion algorithm considers arbitrary anisotropy, and it is extended for special cases including azimuthal and vertical anisotropy. The algorithm is shown to be stable and converges rapidly for several simple anisotropic models. These synthetic tests also confirm that the anisotropic inversion produces a correct anomaly with different but equivalent anisotropic parameters. We also consider a complex 2D anisotropic model; the successful results for this model further confirm that the inversion algorithm presented here, which uses the novel modified limited-memory QN approach, is capable of solving the 2D anisotropic MT inverse problem. Finally, we have evaluated a practical application on MT data collected in northern Tibet to demonstrate the effectiveness and stability of our algorithm.

Three-Dimensional Inversion of Controlled-Source Electromagnetic Data in Fanchang Region, Anhui Province

Fanchang area of Anhui Province is an important link in the middle and lower reaches of the Yangtze River. To study the distribution of rock mass near the surface of the area and the contact relationship between the strata, and to determine the possible ore body space, we carry out a 15km controlled-source electromagnetic(CSEM) survey. The observed data is the electric field amplitude data simulating wide-area electromagnetic. For the single-section electric field amplitude data of controlled source, because of the lack of magnetic field data, the impedance data needed for audio geomagnetic inversion cannot be formed. This paper uses the limited memory quasi-Newton(L-BFGS) method to perform a three-dimensional inversion with the source directly. The theoretical model test results show that the inversion method can recover the underground resistivity structure and obtain better anomalous body resolution. The three-dimensional inversion of the CSEM data in Fanchang area shows that we could get the best inversion results when the grid size is about 1/3~1/2 of the dot pitch, the background resistivity is 100 Ω·m, and the error level is 5%. The inversion result of the full-section data, including two sources, shows that the south and north end of the profile is high-resistivity, and the intermediate resistivity is low. However, in the middle of the profile, the resistivity is significantly higher than the two sides. Combined with geological data, this change reflects the distribution of strata/rock masses on the surface of the work area and the contact relationship between them. This consistency verifies the feasibility and necessity of three-dimensional source inversion for the CSEM field amplitude |Ex| data of multi-source and single profile. The underground three-dimensional resistivity structure can be effectively obtained by using the L-BFGS method with the controlled source.

A Matrix-Free Likelihood Method for Exploratory Factor Analysis of High-Dimensional Gaussian Data

ABSTRACTThis technical note proposes a novel profile likelihood method for estimating the covariance parameters in exploratory factor analysis of high-dimensional Gaussian datasets with fewer observations than number of variables. An implicitly restarted Lanczos algorithm and a limited-memory quasi-Newton method are implemented to develop a matrix-free framework for likelihood maximization. Simulation results show that our method is substantially faster than the expectation-maximization solution without sacrificing accuracy. Our method is applied to fit factor models on data from suicide attempters, suicide ideators, and a control group. Supplementary materials for this article are available online.

A Registration Method for Profile Error Inspection of Complex Surface Under Minimum Zone Criterion

Although registration of the measured model and the design model has been widely studied, it remains an open problem in complex surface profile error inspection. False rejection of some qualified parts is prone to occur due to the improper registration between the measured points and the nominal model. To solve this problem, we define a surface profile error metric in terms of a new range norm under minimum zone criterion. An optimal framework based on the range norm is proposed to find the registration pose of the measurement data set, in which the surface profile error is minimized. To deal with the computational intractability, the non-differentiable merit function is uniformly approximated by a smooth aggregation function, which can be effectively solved with the limited-memory Quasi-Newton method. Distinct numerical stability control and the active set selection mechanism are proposed to guarantee the accuracy and efficiency of the method. Experiments on a simulated and a real part are included to verify the superiority of the proposed method.

3D magnetotelluric inversions with unstructured finite-element and limited-memory quasi-Newton methods

Traditional 3D Magnetotelluric (MT) forward modeling and inversions are mostly based on structured meshes that have limited accuracy when modeling undulating surfaces and arbitrary structures. By contrast, unstructured-grid-based methods can model complex underground structures with high accuracy and overcome the defects of traditional methods, such as the high computational cost for improving model accuracy and the difficulty of inverting with topography. In this paper, we used the limited-memory quasi-Newton (L-BFGS) method with an unstructured finite-element grid to perform 3D MT inversions. This method avoids explicitly calculating Hessian matrices, which greatly reduces the memory requirements. After the first iteration, the approximate inverse Hessian matrix well approximates the true one, and the Newton step (set to 1) can meet the sufficient descent condition. Only one calculation of the objective function and its gradient are needed for each iteration, which greatly improves its computational efficiency. This approach is well-suited for large-scale 3D MT inversions. We have tested our algorithm on data with and without topography, and the results matched the real models well. We can recommend performing inversions based on an unstructured finite-element method and the L-BFGS method for situations with topography and complex underground structures.

A Diagonally Updated Limited-Memory Quasi-Newton Method for the Weighted Density Approximation

We propose a limited-memory quasi-Newton method using the bad Broyden update and apply it to the nonlinear equations that must be solved to determine the effective Fermi momentum in the weighted density approximation for the exchange energy density functional. This algorithm has advantages for nonlinear systems of equations with diagonally dominant Jacobians, because it is easy to generalize the method to allow for periodic updates of the diagonal of the Jacobian. Systematic tests of the method for atoms show that one can determine the effective Fermi momentum at thousands of points in less than fifteen iterations.

Efficient construction of sparse radial basis function neural networks using [formula omitted]-regularization

This paper investigates the construction of sparse radial basis function neural networks (RBFNNs) for classification problems. An efficient two-phase construction algorithm (which is abbreviated as TPCLR1 for simplicity) is proposed by using L1 regularization. In the first phase, an improved maximum data coverage (IMDC) algorithm is presented for the initialization of RBF centers and widths. Then a specialized Orthant-Wise Limited-memory Quasi-Newton (sOWL-QN) method is employed to perform simultaneous network pruning and parameter optimization in the second phase. The advantages of TPCLR1 lie in that better generalization performance is guaranteed with higher model sparsity, and the required storage space and testing time are much reduced. Besides these, only the regularization parameter and the maximum number of function evaluations are required to be prescribed, then the entire construction procedure becomes automatic. The learning algorithm is verified by several classification benchmarks with different levels of complexity. The experimental results show that an appropriate value of the regularization parameter is easy to find without using costly cross validation, and the proposed TPCLR1 offers an efficient procedure to construct sparse RBFNN classifiers with good generalization performance.

ℓ1-Regularized full-waveform inversion with prior model information based on orthant-wise limited memory quasi-Newton method

Full-waveform inversion (FWI) is an ill-posed optimization problem which is sensitive to noise and initial model. To alleviate the ill-posedness of the problem, regularization techniques are usually adopted. The ℓ1-norm penalty is a robust regularization method that preserves contrasts and edges. The Orthant-Wise Limited-Memory Quasi-Newton (OWL-QN) method extends the widely-used limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) method to the ℓ1-regularized optimization problems and inherits the efficiency of L-BFGS. To take advantage of the ℓ1-regularized method and the prior model information obtained from sonic logs and geological information, we implement OWL-QN algorithm in ℓ1-regularized FWI with prior model information in this paper. Numerical experiments show that this method not only improve the inversion results but also has a strong anti-noise ability.

A limited memory quasi-Newton trust-region method for box constrained optimization

By means of Wolfe conditions strategy, we propose a quasi-Newton trust-region method to solve box constrained optimization problems. This method is an adequate combination of the compact limited memory BFGS and the trust-region direction while the generated point satisfies the Wolfe conditions and therefore maintains a positive-definite approximation to the Hessian of the objective function. The global convergence and the quadratic convergence of this method are established under suitable conditions. Finally, we compare our algorithms (IWTRAL and IBWTRAL) with an active set trust-region algorithm (ASTRAL) Xu and Burke (2007) on the CUTEst box constrained test problems presented by Gould et al. (2015). Numerical results show that the presented method is competitive and totally interesting for solving box constrained optimization.

An improved OWL-QN method for solving sparse logistic regression problems

Sparse logistic regression, is a class of important problems in machine learning, which has wide applications in cybernetics, management sciences and internet, etc. This paper proposes an Improved Orthant-Wise Limited-memory Quasi-Newton (IOWL-QN) method for sparse logistic regression problems. The proposed method adopts the well-known BB stepsize strategy to approximate the Hessian of the goal objective function, and constructs the quasi-Newton vectors based on the gradient of the goal objective function. Under mild conditions, global convergence of the proposed method is established. Numerical results are reported to illustrate that the proposed method is feasible and efficient.

Dynamic scaling on the limited memory BFGS method

Abstract This paper describes a limited-memory quasi-Newton method in which the initial inverse Hessian approximation is constructed based on the concept of equilibration of the inverse Hessian matrix. Curvature information about the objective function is stored in the form of a diagonal matrix, and plays the dual role of providing an initial matrix and of equilibrating for limited memory BFGS (LBFGS) iterations. An extensive numerical testing has been performed showing that the diagonal scaling strategy proposed is very effective.

An accelerated subspace minimization three-term conjugate gradient algorithm for unconstrained optimization

A three-term conjugate gradient algorithm for large-scale unconstrained optimization using subspace minimizing technique is presented. In this algorithm the search directions are computed by minimizing the quadratic approximation of the objective function in a subspace spanned by the vectors: −g k+1, s k and y k . The search direction is considered as: d k+1 = −g k+1 + a k s k + b k y k , where the scalars a k and b k are determined by minimization the affine quadratic approximate of the objective function. The step-lengths are determined by the Wolfe line search conditions. We prove that the search directions are descent and satisfy the Dai-Liao conjugacy condition. The suggested algorithm is of three-term conjugate gradient type, for which both the descent and the conjugacy conditions are guaranteed. It is shown that, for uniformly convex functions, the directions generated by the algorithm are bounded above, i.e. the algorithm is convergent. The numerical experiments, for a set of 750 unconstrained optimization test problems, show that this new algorithm substantially outperforms the known Hestenes and Stiefel, Dai and Liao, Dai and Yuan and Polak, Ribiere and Poliak conjugate gradient algorithms, as well as the limited memory quasi-Newton method L-BFGS and the discrete truncated-Newton method TN.

A New Deformable Image Registration Method Based on B-Spline for Clinical 4D CT

Four-dimensional computed tomography (4D CT) which clearly includes the temporal changes in anatomy during the diagnosis, planning, and delivery of radiotherapy has great promise. Deformable image registration has the potential to reduce the geometrical uncertainty of the target, and makes it possible to signally improve the treatment accuracy by optimizing treatment in response to anatomical uncertainty. In this paper, we used Scale Invariant Feature Transform (SIFT) algorithm to extract landmark points, and we proposed a registration method based on B-Spline model, then used a limited memory quasi-Newton method to optimize the system, also calls the limited memory BFGS (L-BFGS) method. The deformable registration model B-Spline model can derive the images at all intermediate phases from sets of 3D images acquired at a few known phase points. Because 4D CT can track the location of region of interest (ROI) and tumors over several respiratory cycles, so 4D CT can make the apparent size of the tumor which is caused by breathing motion more accurate. The method is evaluated on 10 4D-CT data sets of patients in a breathing cycle.

Global 3-D imaging of mantle electrical conductivity based on inversion of observatory C-responses-I. An approach and its verification

SUMMARY We present a novel frequency-domain inverse solution to recover the 3-D electrical conductivity distribution in the mantle. The solution is based on analysis of local C-responses. It exploits an iterative gradient-type method—limited-memory quasi-Newton method—for minimizing the penalty function consisting of data misfit and regularization terms. The integral equation code is used as a forward engine to calculate responses and data misfit gradients during inversion. An adjoint approach is implemented to compute misfit gradients efficiently. Further improvements in computational load come from parallelizing the scheme with respect to frequencies, and from setting the most time-consuming part of the forward calculations—calculation of Green’s tensors—apart from the inversion loop. Convergence, performance, and accuracy of our 3-D inverse solution are demonstrated with a synthetic numerical example. A companion paper applies the strategy set forth here to real data.

Fitting very large sparse Gaussian graphical models

In this paper we consider some methods for the maximum likelihood estimation of sparse Gaussian graphical (covariance selection) models when the number of variables is very large (tens of thousands or more). We present a procedure for determining the pattern of zeros in the model and we discuss the use of limited memory quasi-Newton algorithms and truncated Newton algorithms to fit the model by maximum likelihood. We present efficient ways of computing the gradients and likelihood function values for such models suitable for a desktop computer. For the truncated Newton method we also present an efficient way of computing the action of the Hessian matrix on an arbitrary vector which does not require the computation and storage of the Hessian matrix. The methods are illustrated and compared on simulated data and applied to a real microarray data set.The limited memory quasi-Newton method is recommended for practical use.

On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning

This paper describes how to incorporate sampled curvature information in a Newton-CG method and in a limited memory quasi-Newton method for statistical learning. The motivation for this work stems from supervised machine learning applications involving a very large number of training points. We follow a batch approach, also known in the stochastic optimization literature as a sample average approximation approach. Curvature information is incorporated in two subsampled Hessian algorithms, one based on a matrix-free inexact Newton iteration and one on a preconditioned limited memory BFGS iteration. A crucial feature of our technique is that Hessian-vector multiplications are carried out with a significantly smaller sample size than is used for the function and gradient. The efficiency of the proposed methods is illustrated using a machine learning application involving speech recognition.