Truncated Newton Method Research Articles

Truncated Newton full waveform inversion method for the human brain imaging

Abstract It has been shown that full-waveform inversion (FWI) method can be a competitive alternative for medical imaging problems. It offers high-resolution results while delivering the advantages of being fast, safe, portable, and affordable, compared to magnetic resonance imaging (MRI) and x-ray computed tomography. However, to the best of our knowledge, FWI applications in medical imaging only use the first-order derivative information. In this case, the high parameter contrasts between different tissues of human body and multi-scatterings problems may lead FWI to local minima. Thus, we present a competitive truncated Newton method for high-resolution imaging of the human brain. This truncated Newton method, based on the efficient linear solver MINRES-QLP, can make full use of multiple scattering wavefield information. Compared with the truncated Newton method based on conjugate gradient, the MINRES-QLP iterative method presents various advantages when solving linear systems, such as the capacity to handle both non-singular and singular systems, less computational cost, and efficiency even for ill-conditioned systems. Numerical experiments for imaging the brain with and without the skull are conducted. Numerical results indicate that, compared with the truncated Newton method based on conjugate gradient, the truncated Newton method based on MINRES-QLP exhibits computational efficiency while maintaining the same level of accuracy.

Investigating Hessian-based inversion velocity analysis

Inversion velocity analysis (IVA) is an image-domain method built upon the spatial scale separation of the model. Accordingly, the IVA method is performed with an iterative process composed of two minimization steps consisting of migration (inner loop) and tomography (outer loop), respectively, with each step accounting for its Hessian or not. The migration part provides the common-image gathers (CIGs) with extension in the horizontal subsurface offset. Then, the differential semblance optimization (DSO) misfit measures the focusing of the events in the CIGs, which indicates the quality of the velocity model. Commonly, the velocity updates are obtained from the DSO gradient. IVA is a modified version in which the approximate inverse replaces the adjoint of the inner loop process; in that case, the migration Hessian is approximately diagonal in the high-frequency regime. In this work, we report on implementation of the tomographic Hessian (i.e., the second derivative of the DSO misfit with respect to the background model) for the estimation of the background velocity model. We apply the second-order adjoint-state method to obtain the application of the tomographic Hessian on a vector. Then, we use the truncated-Newton (TN) method to obtain the update directions by computing approximately the application of the inverse of the tomographic Hessian on the descent direction. We also make a theoretical comparison between tomography in the IVA and full-waveform inversion contexts. Two numerical examples are used to compare, in terms of geophysical results and computational costs, the TN method with different gradient-based optimization methods applied to the IVA. A small model allows us to evaluate the eigenvalues of the tomographic Hessian, which explains the large damping needed in the TN case.

METRIC BINARY CLASSIFIER WITH SELECTION OF FEATURE WEIGHTS

The family of metric algorithms based on determining the distance from one observation to another has a number of advantages, such as their suitability for many types of problems and results have a clear interpretation. Therefore, metric algorithms are widely used in credit risk modeling, non-destructive quality control of products, medical diagnostics, geology, and many other practical areas. The most common metric algorithm in practice is the k-nearest neighbors method. At the same time, one of the key problems of metric algorithms is the problem of dimension, since the decision is made on the basis of all observations of the training sample. In addition, usually all variables have the same weight when calculating the distance, which leads to a drop in the quality of the algorithm with an increase in the number of features. The article discusses a new machine learning method for solving classification problems – a metric classifier with the selection of feature weights, which allows to solve these problems to a large extent. Nine algorithms were used to optimize the function. Classification quality based on them is checked on 3 problems from the UCI repository. As a result of the comparison, the truncated Newton method was chosen to build a new metric classifier. The quality of the new classifier was tested on 8 datasets from the same repository and compared with the quality of the classical nearest neighbor method. This classifier has a higher quality for problems with a large number of features in comparison to the classical approach. Data set characteristics and calculation results are presented in the corresponding tables.

A Study on Truncated Newton Methods for Linear Classification.

Truncated Newton (TN) methods have been a useful technique for large-scale optimization. Instead of obtaining the full Newton direction, a truncated method approximately solves the Newton equation with an inner conjugate gradient (CG) procedure (TNCG for the whole method). These methods have been employed to efficiently solve linear classification problems. However, even in this deeply studied field, various theoretical and numerical aspects were not completely explored. The first contribution of this work is to comprehensively study the global and local convergence when TNCG is applied to linear classification. Because of the lack of twice differentiability under some losses, many past works cannot be applied here. We prove various missing pieces of theory from scratch and clarify many proper references. The second contribution is to study the termination of the CG method. For the first time when TNCG is applied to linear classification, we show that the inner stopping condition strongly affects the convergence speed. We propose using a quadratic stopping criterion to achieve both robustness and efficiency. The third contribution is that of combining the study on inner stopping criteria with that of preconditioning. We discuss how convergence theory is affected by preconditioning and finally propose an effective preconditioned TNCG.

Parsimonious truncated Newton method for time-domain full-waveform inversion based on the Fourier-domain full-scattered-field approximation

To exploit Hessian information in full-waveform inversion (FWI), the matrix-free truncated Newton method can be used. In such a method, Hessian-vector product computation is one of the major concerns due to the huge memory requirements and demanding computational cost. Using the adjoint-state method, the Hessian-vector product can be estimated by zero-lag crosscorrelation of the first-/second-order incident wavefields and the second-/first-order adjoint wavefields. Different from the implementation in frequency-domain FWI, Hessian-vector product construction in the time domain becomes much more challenging because it is not affordable to store all of the time-dependent wavefields. The widely used wavefield recomputation strategy leads to computationally intensive tasks. We have developed an efficient alternative approach to computing the Hessian-vector product for time-domain FWI. In our method, discrete Fourier transform is applied to extract frequency-domain components of involved wavefields, which are used to compute wavefield crosscorrelation in the frequency domain. This makes it possible to avoid reconstructing the first- and second-order incident wavefields. In addition, a full-scattered-field approximation is proposed to efficiently simplify the second-order incident and adjoint wavefield computation, which enables us to refrain from repeatedly solving the first-order incident and adjoint equations for the second-order incident and adjoint wavefields (re)computation. With our method, the computational time can be reduced by 70% and 80% in viscous media for Gauss-Newton and full-Newton Hessian-vector product construction, respectively. The effectiveness of our method is also verified in the frame of a 2D multiparameter inversion, in which our method almost reaches the same iterative convergence of the conventional time-domain implementation.

Introduction of the Hessian in joint migration inversion and improved recovery of structural information using image-based regularization

Joint migration inversion (JMI) is a method based on one-way wave equations that aims at fitting seismic reflection data to estimate an image and a background velocity. The depth-migrated image describes the high spatial-frequency content of the subsurface and, in principle, is true amplitude. The background velocity model accounts mainly for the large spatial-scale kinematic effects of the wave propagation. Looking for a deeper understanding of the method, we briefly review the continuous equations that compose the forward-modeling engine of JMI for acoustic media and angle-independent scattering. Then, we use these equations together with the first-order adjoint-state method to arrive at a new formulation of the model gradients. To estimate the image, we combine the second-order adjoint-state method with the truncated-Newton method to obtain the image updates. For the model (velocity) estimation, in comparison to the image update, we reduce the computational cost by adopting a diagonal preconditioner for the corresponding gradient in combination with an image-based regularizing function. Based on this formulation, we build our implementation of the JMI algorithm. Our image-based regularization of the model estimate allows us to carry over structural information from the estimated image to the jointly estimated background model. As demonstrated by our numerical experiments, this procedure can help to improve the resolution of the estimated model and make it more consistent with the image.

Multiparameter viscoelastic full-waveform inversion of shallow-seismic surface waves with a pre-conditioned truncated Newton method

SUMMARY 2-D full-waveform inversion (FWI) of shallow-seismic Rayleigh waves has become a powerful method for reconstructing viscoelastic multiparameter models of shallow subsurface with high resolution. The multiparameter reconstruction in FWI is challenging due to the potential presence of cross-talk between different parameters and the unbalanced sensitivity of Rayleigh-wave data with respect to different parameter classes. Accounting for the inverse Hessian using truncated Newton methods based on second-order adjoint methods provides an effective tool to mitigate cross-talk caused by the coupling between different parameters. In this study, we apply a pre-conditioned truncated Newton (PTN) method to shallow-seismic FWI to simultaneously invert for multiparameter near-surface models (P- and S-wave velocities, attenuation of P and S waves, and density). We first investigate scattered wavefields caused by these parameters to evaluate the coupling between them. Then we investigate the performance of the PTN method on shallow-seismic FWI of Rayleigh wave for reconstructing all five parameters simultaneously. The application to spatially correlated and uncorrelated models demonstrates that the PTN method helps to mitigate the cross-talk and improves the resolution of the multiparameter reconstructions, especially for the weak parameters with small sensitivity such as attenuation and density parameters. The attenuation of Pwaves cannot be inverted reliably due to its negligible sensitivity on the Rayleigh wave. The comparison with the classical pre-conditioned conjugate gradient method highlights the improved performance of the PTN method and thus the benefit of accounting for the information included in the Hessian.

Modeling Hessian-vector products in nonlinear optimization: new Hessian-free methods

Abstract In this paper we suggest two ways of calculating interpolation models for unconstrained smooth nonlinear optimization when Hessian-vector products are available. The main idea is to interpolate the objective function using a quadratic on a set of points around the current one, and concurrently using the curvature information from products of the Hessian times appropriate vectors, possibly defined by the interpolating points. These enriched interpolating conditions then form an affine space of model Hessians or model Newton directions, from which a particular one can be computed once an equilibrium or least secant principle is defined. A first approach consists of recovering the Hessian matrix satisfying the enriched interpolating conditions, from which then a Newton direction model can be computed. In a second approach we pose the recovery problem directly in the Newton direction. These techniques can lead to a significant reduction in the overall number of Hessian-vector products when compared to the inexact or truncated Newton method, although simple implementations may pay a cost in the number of function evaluations and the dense linear algebra involved poses a scalability challenge.

Elastic Full Waveform Inversion Based on the Trust Region Strategy

In this paper, we investigate the elastic wave full-waveform inversion (FWI) based on the trust region method. The FWI is an optimization problem of minimizing the misfit between the observed data and simulated data. Usually, the line search method is used to update the model parameters iteratively. The line search method generates a search direction first and then finds a suitable step length along the direction. In the trust region method, it defines a trial step length within a certain neighborhood of the current iterate point and then solves a trust region subproblem. The theoretical methods for the trust region FWI with the Newton type method are described. The algorithms for the truncated Newton method with the line search strategy and for the Gauss-Newton method with the trust region strategy are presented. Numerical computations of FWI for the Marmousi model by the L-BFGS method, the Gauss-Newton method and the truncated Newton method are completed. The comparisons between the line search strategy and the trust region strategy are given and show that the trust region method is more efficient than the line search method and both the Gauss-Newton and truncated Newton methods are more accurate than the L-BFGS method.

Issues on the use of a modified Bunch and Kaufman decomposition for large scale Newton\u2019s equation

In this work, we deal with Truncated Newton methods for solving large scale (possibly nonconvex) unconstrained optimization problems. In particular, we consider the use of a modified Bunch and Kaufman factorization for solving the Newton equation, at each (outer) iteration of the method. The Bunch and Kaufman factorization of a tridiagonal matrix is an effective and stable matrix decomposition, which is well exploited in the widely adopted SYMMBK (Bunch and Kaufman in Math Comput 31:163–179, 1977; Chandra in Conjugate gradient methods for partial differential equations, vol 129, 1978; Conn et al. in Trust-region methods. MPS-SIAM series on optimization, Society for Industrial Mathematics, Philadelphia, 2000; HSL, A collection of Fortran codes for large scale scientific computation, http://www.hsl.rl.ac.uk/; Marcia in Appl Numer Math 58:449–458, 2008) routine. It can be used to provide conjugate directions, both in the case of 1times 1 and 2times 2 pivoting steps. The main drawback is that the resulting solution of Newton’s equation might not be gradient–related, in the case the objective function is nonconvex. Here we first focus on some theoretical properties, in order to ensure that at each iteration of the Truncated Newton method, the search direction obtained by using an adapted Bunch and Kaufman factorization is gradient–related. This allows to perform a standard Armijo-type linesearch procedure, using a bounded descent direction. Furthermore, the results of an extended numerical experience using large scale CUTEst problems is reported, showing the reliability and the efficiency of the proposed approach, both on convex and nonconvex problems.

Use of prismatic waves in full-waveform inversion with the exact Hessian

The truncated Newton method uses information contained in the exact Hessian in full-waveform inversion (FWI). The exact Hessian physically contains information regarding doubly scattered waves, especially prismatic events. These waves are mainly caused by the scattering at steeply dipping structures, such as salt flanks and vertical or nearly vertical faults. We have systematically investigated the properties and applications of the exact Hessian. We begin by giving the formulas for computing each term in the exact Hessian and numerically analyzing their characteristics. We show that the second term in the exact Hessian may be comparable in magnitude to the first term. In particular, when there are apparent doubly scattered waves in the observed data, the influence of the second term may be dominant in the exact Hessian and the second term cannot be neglected. Next, we adopt a migration/demigration approach to compute the Gauss-Newton-descent direction and the Newton-descent direction using the approximate Hessian and the exact Hessian, respectively. In addition, we determine from the forward and the inverse perspectives that the second term in the exact Hessian not only contributes to the use of doubly scattered waves, but it also compensates for the use of single-scattering waves in FWI. Finally, we use three numerical examples to prove that by considering the second term in the exact Hessian, the role of prismatic waves in the observed data can be effectively revealed and steeply dipping structures can be reconstructed with higher accuracy.

A Class of Approximate Inverse Preconditioners Based on Krylov-Subspace Methods for Large-Scale Nonconvex Optimization

We introduce a class of positive definite preconditioners for the solution of large symmetric indefinite linear systems or sequences of such systems, in optimization frameworks. The preconditioners are iteratively constructed by collecting information on a reduced eigenspace of the indefinite matrix by means of a Krylov-subspace solver. A spectral analysis of the preconditioned matrix shows the clustering of some eigenvalues and possibly the nonexpansion of its spectrum. Extensive numerical experimentation is carried out on standard difficult linear systems and by embedding the class of preconditioners within truncated Newton methods for large-scale unconstrained optimization (the issue of major interest). Although the Krylov-based method may provide modest information on matrix eigenspaces, the results obtained show that the proposed preconditioners lead to substantial improvements in terms of efficiency and robustness, particularly on very large nonconvex problems.

Improving Truncated Newton Method for the Logit-Based Stochastic User Equilibrium Problem

This study proposes an improved truncated Newton (ITN) method for the logit-based stochastic user equilibrium problem. The ITN method incorporates a preprocessing procedure to the traditional truncated Newton method so that a good initial point is generated, on the basis of which a useful principle is developed for the choice of the basic variables. We discuss the rationale of both improvements from a theoretical point of view and demonstrate that they can enhance the computational efficiency in the early and late iteration stages, respectively, when solving the logit-based stochastic user equilibrium problem. The ITN method is compared with other related methods in the literature. Numerical results show that the ITN method performs favorably over these methods.

An adaptive nonmonotone truncated Newton method for optimal control of a class of parabolic distributed parameter systems

A black-box method using the finite elements, the Crank–Nicolson and a nonmonotone truncated Newton (TN) method is presented for solving optimal control problems (OCPs) governed by partial differential equations (PDEs). The proposed method finds the optimal control of a class of linear and nonlinear parabolic distributed parameter systems with a quadratic cost functional. To this end, the piecewise linear finite elements method and the well-known Crank–Nicolson method are used for discretizing in space and in time, respectively. Afterwards, regarding the implicit function theorem (IFT), the optimal control problem is transformed into an unconstrained nonlinear optimization problem. Considering that in a gradient-based method for solving optimal control problems, the evaluations of gradients and Hessians of the cost functional is important, hence, an adjoint technique is used to evaluate them effectively. In addition, to make a globalization strategy, we first introduce an adaptive nonmonotone strategy which properly controls the degree of nonmonotonicity and then incorporate it into an inexact Armijo-type line search approach to construct a more relaxed line search procedure. Finally, the obtained unconstrained nonlinear optimization problem is solved by utilizing the proposed nonmonotone truncated Newton method. Results gained from the new offered method compared with existing methods show that the new method is promising.

Inverts permittivity and conductivity with structural constraint in GPR FWI based on truncated Newton method

Full waveform inversion (FWI) of ground penetrating radar (GPR) is a promising technique to quantitatively evaluate the permittivity and conductivity of near subsurface. However, these two parameters are simultaneously inverted in the GPR FWI, increasing the difficulty to obtain accurate inversion results for both parameters. In this study, I present a structural constrained GPR FWI procedure to jointly invert the two parameters, aiming to force a structural relationship between permittivity and conductivity in the process of model reconstruction. The structural constraint is enforced by a cross-gradient function. In this procedure, the permittivity and conductivity models are inverted alternately at each iteration and updated with hierarchical frequency components in the frequency domain. The joint inverse problem is solved by the truncated Newton method which considering the effect of Hessian operator and using the approximated solution of Newton equation to be the perturbation model in the updating process. The joint inversion procedure is tested by three synthetic examples. The results show that jointly inverting permittivity and conductivity in GPR FWI effectively increases the structural similarities between the two parameters, corrects the structures of parameter models, and significantly improves the accuracy of conductivity model, resulting in a better inversion result than the individual inversion.

An adaptive truncation criterion, for linesearch-based truncated Newton methods in large scale nonconvex optimization

Starting from the paper by Nash and Sofer (1990), we propose a heuristic adaptive truncation criterion for the inner iterations within linesearch-based truncated Newton methods. Our aim is to possibly avoid “over-solving” of the Newton equation, based on a comparison between the predicted reduction of the objective function and the actual reduction obtained. A numerical experience on unconstrained optimization problems highlights a satisfactory effectiveness and robustness of the adaptive criterion proposed, when a residual-based truncation criterion is selected.

Logistic regression in large rare events and imbalanced data: A performance comparison of prior correction and weighting methods

AbstractThe purpose of this study is to use the truncated Newton method in prior correction logistic regression (LR). A regularization term is added to prior correction LR to improve its performance, which results in the truncated‐regularized prior correction algorithm. The performance of this algorithm is compared with that of weighted LR and the regular LR methods for large imbalanced binary class data sets. The results, based on the KDD99 intrusion detection data set, and 6 other data sets at both the prior correction and the weighted LRs have the same computational efficiency when the truncated Newton method is used in both of them. A higher discriminative performance, however, resulted from weighting, which exceeded both the prior correction and the regular LR on nearly all the data sets. From this study, we conclude that weighting outperforms both the regular and prior correction LR models in most data sets and it is the method of choice when LR is used to evaluate imbalanced and rare event data.

Full Waveform Inversion and the Truncated Newton Method

Full waveform inversion (FWI) is a powerful method for reconstructing subsurface parameters from local measurements of the seismic wavefield. This method consists in minimizing the distance between predicted and recorded data. The predicted data are computed as the solution of a wave-propagation problem. Conventional numerical methods for the solution of FWI problems are gradient-based methods, such as the preconditioned steepest descent, the nonlinear conjugate gradient, or more recently the $l$-BFGS quasi-Newton algorithm. In this study, we investigate the desirability of applying a truncated Newton method to FWI. The inverse Hessian operator plays a crucial role in the parameter reconstruction, as it should help to mitigate finite-frequency effects and to better remove artifacts arising from multiscattered waves. For multiparameter reconstruction, the inverse Hessian operator also offers the possibility of better removing trade-offs due to coupling effects between parameter classes. The truncated Newto...

Ranking Support Vector Machine with Kernel Approximation.

Learning to rank algorithm has become important in recent years due to its successful application in information retrieval, recommender system, and computational biology, and so forth. Ranking support vector machine (RankSVM) is one of the state-of-art ranking models and has been favorably used. Nonlinear RankSVM (RankSVM with nonlinear kernels) can give higher accuracy than linear RankSVM (RankSVM with a linear kernel) for complex nonlinear ranking problem. However, the learning methods for nonlinear RankSVM are still time-consuming because of the calculation of kernel matrix. In this paper, we propose a fast ranking algorithm based on kernel approximation to avoid computing the kernel matrix. We explore two types of kernel approximation methods, namely, the Nyström method and random Fourier features. Primal truncated Newton method is used to optimize the pairwise L2-loss (squared Hinge-loss) objective function of the ranking model after the nonlinear kernel approximation. Experimental results demonstrate that our proposed method gets a much faster training speed than kernel RankSVM and achieves comparable or better performance over state-of-the-art ranking algorithms.

An Approach for Analyzing the Global Rate of Convergence of Quasi-Newton and Truncated-Newton Methods

Quasi-Newton and truncated-Newton methods are popular methods in optimization and are traditionally seen as useful alternatives to the gradient and Newton methods. Throughout the literature, results are found that link quasi-Newton methods to certain first-order methods under various assumptions. We offer a simple proof to show that a range of quasi-Newton methods are first-order methods in the definition of Nesterov. Further, we define a class of generalized first-order methods and show that the truncated-Newton method is a generalized first-order method and that first-order methods and generalized first-order methods share the same worst-case convergence rates. Further, we extend the complexity analysis for smooth strongly convex problems to finite dimensions. An implication of these results is that in a worst-case scenario, the local superlinear or faster convergence rates of quasi-Newton and truncated-Newton methods cannot be effective unless the number of iterations exceeds half the size of the problem dimension.