Inexact Newton Iteration Research Articles

On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning

This paper describes how to incorporate sampled curvature information in a Newton-CG method and in a limited memory quasi-Newton method for statistical learning. The motivation for this work stems from supervised machine learning applications involving a very large number of training points. We follow a batch approach, also known in the stochastic optimization literature as a sample average approximation approach. Curvature information is incorporated in two subsampled Hessian algorithms, one based on a matrix-free inexact Newton iteration and one on a preconditioned limited memory BFGS iteration. A crucial feature of our technique is that Hessian-vector multiplications are carried out with a significantly smaller sample size than is used for the function and gradient. The efficiency of the proposed methods is illustrated using a machine learning application involving speech recognition.

On inexact Newton methods based on doubling iteration scheme for non-symmetric algebraic Riccati equations

Newton iteration method can be used to find the minimal non-negative solution of a certain class of non-symmetric algebraic Riccati equations. However, a serious bottleneck exists in efficiency and storage for the implementation of the Newton iteration method, which comes from the use of some direct methods in exactly solving the involved Sylvester equations. In this paper, instead of direct methods, we apply a fast doubling iteration scheme to inexactly solve the Sylvester equations. Hence, a class of inexact Newton iteration methods that uses the Newton iteration method as the outer iteration and the doubling iteration scheme as the inner iteration is obtained. The corresponding procedure is precisely described and two practical methods of monotone convergence are algorithmically presented. In addition, the convergence property of these new methods is studied and numerical results are given to show their feasibility and effectiveness for solving the non-symmetric algebraic Riccati equations. Copyright © 2010 John Wiley & Sons, Ltd.

A multilevel hybrid Newton–Krylov–Schwarz method for the Bidomain model of electrocardiology

A multilevel hybrid Newton–Krylov–Schwarz (NKS) method is constructed and studied numerically for implicit time discretizations of the Bidomain reaction–diffusion system in three dimensions. This model describes the bioelectrical activity of the heart by coupling two degenerate parabolic equations with a stiff system of ordinary differential equations. The NKS Bidomain solver employs an outer inexact Newton iteration to solve the nonlinear finite element system originating at each time step of the implicit discretization. The Jacobian update during the Newton iteration is solved by a Krylov method employing a multilevel hybrid overlapping Schwarz preconditioner, additive within the levels and multiplicative among the levels. Several parallel tests on Linux clusters are performed, showing that the convergence of the method is independent of the number of subdomains (scalability), the discretization parameters and the number of levels (optimality).

A Scalable Newton–Krylov–Schwarz Method for the Bidomain Reaction-Diffusion System

A novel two-level Newton-Krylov-Schwarz (NKS) solver is constructed and analyzed for implicit time discretizations of the bidomain reaction-diffusion system in three dimensions. This multiscale system describes the bioelectrical activity of the heart by coupling two degenerate parabolic equations with several ordinary differential equations at each point in space. Together with a finite element discretization in space, the proposed NKS Bidomain solver employs an outer inexact Newton iteration to solve the nonlinear finite element system originating at each time step of the decoupled implicit discretization. The Jacobian update during the Newton iteration is solved by a Krylov method employing a two-level overlapping Schwarz preconditioner. A convergence rate estimate is proved for the resulting preconditioned operator, showing that its condition number is independent of the number of subdomains (scalability) and bounded by the ratio of the subdomains characteristic size and the overlap size. This theoretical result is confirmed by several parallel simulations employing up to more than 2000 processors for scaled and standard speedup tests in three dimensions. The results show the scalability of the proposed NKS Bidomain solver in terms of both nonlinear and linear iterations, in both Cartesian slabs and ellipsoidal cardiac domains.

Inexact Kleinman–Newton Method for Riccati Equations

In this paper we consider the numerical solution of the algebraic Riccati equation using Newton's method. We propose an inexact variant which allows one control the number of the inner iterates used in an iterative solver for each Newton step. Conditions are given under which the monotonicity and global convergence result of Kleinman also hold for the inexact Newton iterates. Numerical results illustrate the efficiency of this method.

On the semilocal convergence of inexact Newton methods in Banach spaces

We provide two types of semilocal convergence theorems for approximating a solution of an equation in a Banach space setting using an inexact Newton method [I.K. Argyros, Relation between forcing sequences and inexact Newton iterates in Banach spaces, Computing 63 (2) (1999) 134–144; I.K. Argyros, A new convergence theorem for the inexact Newton method based on assumptions involving the second Fréchet-derivative, Comput. Appl. Math. 37 (7) (1999) 109–115; I.K. Argyros, Forcing sequences and inexact Newton iterates in Banach space, Appl. Math. Lett. 13 (1) (2000) 77–80; I.K. Argyros, Local convergence of inexact Newton-like iterative methods and applications, Comput. Math. Appl. 39 (2000) 69–75; I.K. Argyros, Computational Theory of Iterative Methods, in: C.K. Chui, L. Wuytack (Eds.), in: Studies in Computational Mathematics, vol. 15, Elsevier Publ. Co., New York, USA, 2007; X. Guo, On semilocal convergence of inexact Newton methods, J. Comput. Math. 25 (2) (2007) 231–242]. By using more precise majorizing sequences than before [X. Guo, On semilocal convergence of inexact Newton methods, J. Comput. Math. 25 (2) (2007) 231–242; Z.D. Huang, On the convergence of inexact Newton method, J. Zheijiang University, Nat. Sci. Ed. 30 (4) (2003) 393–396; L.V. Kantorovich, G.P. Akilov, Functional Analysis, Pergamon Press, Oxford, 1982; X.H. Wang, Convergence on the iteration of Halley family in weak condition, Chinese Sci. Bull. 42 (7) (1997) 552–555; T.J. Ypma, Local convergence of inexact Newton methods, SIAM J. Numer. Anal. 21 (3) (1984) 583–590], we provide (under the same computational cost) under the same or weaker hypotheses: finer error bounds on the distances involved; an at least as precise information on the location of the solution. Moreover if the splitting method is used, we show that a smaller number of inner/outer iterations can be obtained.

Newton regularizations for impedance tomography: a numerical study**Financial support by the Institut für Wissenschaftliches Rechnen und Mathematische Modellbildung, Universität Karlsruhe (TH), and the Mathematical Department of the University of Delaware, USA, is gratefully acknowledged.

The inexact Newton iteration REGINN for regularizing nonlinear ill-posed problems consists of two components: the (outer) Newton iteration, stopped by a discrepancy principle, and the inner iteration, which computes the Newton update by solving approximately a linearized system. The second author proved the convergence of REGINN furnished with the conjugate gradients method as inner iteration (Rieder 2005 SIAM J. Numer. Anal. 43 604–22). Amongst others the following feature distinguishes REGINN from other Newton-like regularization schemes: the regularization level for the locally linearized systems can be adapted dynamically incorporating information on the local degree of ill-posedness gained during the iteration. Of course, the potential of this feature can be fully explored only by meaningful numerical experiments in a realistic setting. Therefore, we apply REGINN to the 2D-electrical impedance tomography problem with the complete electrode model. This inverse problem is known to be severely ill-posed. The achieved reconstructions are compared qualitatively and quantitatively with reconstructions from a one-step method which is closely related to the NOSER algorithm (Cheney et al 1990 Int. J. Imag. Syst. Technol. 2 66–75), an often used solver in impedance tomography. Our detailed numerical comparison reveals REGINN to be a competitive solver outperforming the one-step method when noise corrupts the data and/or a moderately large number of electrodes is used.

Pseudo-Transient Continuation for Nonsmooth Nonlinear Equations

Pseudo-transient continuation is a Newton-like iterative method for computing steady-state solutions of differential equations in cases where the initial data are far from a steady state. The iteration mimics a temporal integration scheme, with the time step being increased as steady state is approached. The iteration is an inexact Newton iteration in the terminal phase. In this paper we show how steady-state solutions to certain ordinary and differential algebraic equations with nonsmooth dynamics can be computed with the method of pseudo-transient continuation. An example of such a case is a discretized PDE with a Lipschitz continuous, but nondifferentiable, constitutive relation as part of the nonlinearity. In this case we can approximate a generalized derivative with a difference quotient. The existing theory for pseudo-transient continuation requires Lipschitz continuity of the Jacobian. Newton-like methods for nonsmooth equations have been globalized by trust-region methods, smooth approximations, and splitting methods in the past, but these approaches are not designed to find steady-state solutions of time-dependent problems. The method in this paper synthesizes the ideas from nonsmooth calculus and the method of pseudo-transient continuation.

Inexact Newton Regularization Using Conjugate Gradients as Inner Iteration

In our papers [Inverse Problems, 15 (1999), pp. 309--327] and [Numer. Math., 88 (2001), pp. 347--365] we proposed algorithm {\tt REGINN}, an inexact Newton iteration for the stable solution of nonlinear ill-posed problems. {\tt REGINN} consists of two components: the outer iteration, which is a Newton iteration stopped by the discrepancy principle, and an inner iteration, which computes the Newton correction by solving the linearized system. The convergence analysis presented in both papers covers virtually any linear regularization method as inner iteration, especially Landweber iteration, $\nu$-methods, and Tikhonov--Phillips regularization. In the present paper we prove convergence rates for {\tt REGINN} when the conjugate gradient method, which is nonlinear, serves as inner iteration. Thereby we add to a convergence analysis of {Hanke}, who had previously investigated {\tt REGINN} furnished with the conjugate gradient method [Numer. Funct. Anal. Optim., 18 (1997), pp. 971--993]. By numerical experiments we illustrate that the conjugate gradient method outperforms the $\nu$-method as inner iteration.

Affine invariant conditions for the inexact perturbed Newton method

The high convergence orders of the inexact Newton iterates were characterized by Ypma in terms of some affine invariant conditions. Using these results, we obtain affine invariant characterizations for the convergence orders of the inexact perturbed Newton iterates.

A note on the quadratic convergence of the inexact Newton methods

We show that a new sufficient condition for the convergence with \(q\)-order two of the inexact Newton iterates may be obtained by considering the normwise backward error of the approximate steps and a result on perturbed Newton methods.This condition is in fact equivalent to the characterization given by Dembo, Eisenstat and Steihaug.

Forcing sequences and inexact Newton iterates in Banach space

We use inexact Newton iterates to approximate a solution of a nonlinear equation in a Banach space. Solving a nonlinear equation using Newton iterates at each stage is very expensive in general. That is why we consider inexact Newton methods, where the Newton equations are solved only approximately and in some unspecified manner. In the elegant paper [1], natural assumptions under which the forcing sequence is uniformly less than one were given based on the first Fréchet-derivative of the operator involved. Here, we use assumption on the second Fréchet-derivative. This way, we essentially reproduce all results found earlier. However, our upper error bounds on the distances involved are smaller.

A parallelizable preconditioner for the iterative solution of implicit Runge–Kutta-type methods

The main difficulty in the implementation of most standard implicit Runge–Kutta (IRK) methods applied to (stiff) ordinary differential equations (ODEs) is to efficiently solve the nonlinear system of equations. In this article we propose the use of a preconditioner whose decomposition cost for a parallel implementation is equivalent to the cost for the implicit Euler method. The preconditioner is based on the W-transformation of the RK coefficient matrices discovered by Hairer and Wanner. For stiff ODEs the preconditioner is by construction asymptotically exact for methods with an invertible RK coefficient matrix. The methodology is particularly useful when applied to super partitioned additive Runge–Kutta (SPARK) methods. The nonlinear system can be solved by inexact simplified Newton iterations: at each simplified Newton step the linear system can be approximately solved by an iterative method applied to the preconditioned linear system.

Relations Between Forcing Sequences and Inexact Newton Iterates in Banach Space

We use inexact Newton iterates to approximate a solution of a nonlinear equation in a Banach space. Solving a nonlinear equation using Newton iterates at each stage is very expensive in general. That is why we consider inexact Newton methods, where the Newton equations are solved only approximately and in some unspecified manner. In the elegant paper [6] natural assumptions under which the forcing sequence is uniformly less than one were given based on the first-Frechet derivative of the operator involved. Here, we use assumptions on the second Frechet-derivative. This way, we essentially reproduce all results found earlier. However, our upper error bounds on the distances involved are smaller.

Time Step Size Selection for Radiation Diffusion Calculations

This work presents efficient solution techniques for radiative transfer in the smoothed particle hydrodynamics discretization. Two choices that impact efficiency are how the material and radiation energy are coupled, which determines the number of iterations needed to converge the emission source, and how the radiation diffusion equation is solved, which must be done in each iteration. The coupled material and radiation energy equations are solved using an inexact Newton iteration scheme based on nonlinear elimination, which reduces the number of Newton iterations needed to converge within each time step. During each Newton iteration, the radiation diffusion equation is solved using Krylov iterative methods with a multigrid preconditioner, which abstracts and optimizes much of the communication when running in parallel. The code is verified for an infinite medium problem, a one-dimensional Marshak wave, and a two and three-dimensional manufactured problem, and exhibits first-order convergence in time and second-order convergence in space. For these problems, the number of iterations needed to converge the inexact Newton scheme and the diffusion equation is independent of the number of spatial points and the number of processors.

Equilibrium Analysis of Finite Width Tension Dominated Foil Bearings

A foil bearing is formed when a flexible material passes over a stationary rigid surface and entrains a thin layer of air that lubricates the relative motion. Some applications include the transport of magnetic tape, and the manufacturing of paper and metal sheets. In each case, the transverse displacement of the moving medium couples with the air film’s pressure, requiring simultaneous solution of the governing elastic and lubrication equations. Such simulations can become computationally intensive, particularly when the model incorporates the finite width effects of side flow and cross-web deformation. Of primary focus in this paper is the development of an efficient numerical algorithm for such simulations. Two improvements in that regard are discussed: inexact Newton iteration and adaptive nodal point placement. In two case studies adapted from the literature, implementation of these techniques offered roughly five-fold improvements in computational effort when the equilibrium film thickness, air pressure, and web displacement were calculated. The key advantages of the approach are a more rapid convergence rate, and the opportunity to use fewer degrees of freedom in describing the pressure and film thickness since the nodes are assigned automatically during iteration to regions where the solution is most rapidly changing.

On the regularization of nonlinear ill-posed problems via inexact Newton iterations

Inexact Newton methods for the stable solution of nonlinear ill-posed problems are considered. The corresponding inner scheme can be chosen to be any linear regularization with a sufficient modulus of convergence. The regularization property of these Newton-type algorithms is verified, that is, the iterates converge to a solution of the nonlinear problem with exact data when the noise level tends to zero. Moreover, convergence rates are given. Finally, implementation issues are discussed and the algorithm is applied to a parameter identification problem for an elliptic PDE. The numerical results reproduce nicely theoretical predictions and show the efficiency of the proposed method.

Relations between forcing sequences and inexact newton-like iterates in banach space

We use inexact Newton-like iterates to approximate a solution of a nonlinear equation in a Banach space. Solving a nonlinear equation using Newton-like iterates at each stage is very expensive in general. That is why we consider inexact Newton-like methods, where the Newton- like equations are solved only approximately and in some unspecified manner. In the elegant paper [6] natural assumptions under which the forcing sequence is uniformly less than one were given based on the first-Fréchet derivative of the operator involved in the special case of inexact Newton iterates. Here, we use assumptions on the first and second Fréchet-derivative. This way, we essentially reproduce all results found earlier but for inexact Newton-like iterates. However, our upper error bounds on the distances involved are smaller