Regularized Newton Method Research Articles

On the convergence of a modified regularized Newton method for convex optimization with singular solutions

In this paper we propose a modified regularized Newton method for convex minimization problems whose Hessian matrices may be singular. The proposed method is proved to converge globally if the gradient and Hessian of the objective function are Lipschitz continuous. Under the local error bound condition, we first show that the method converges quadratically, which implies that ‖xk−x∗‖ is equivalent to dist(xk,X), where X is the solution set and xk→x∗∈X. Then we in turn prove the cubic convergence of the proposed method under the same local error bound condition, which is weaker than nonsingularity.

A regularized Newton method for the efficient approximation of tensors represented in the canonical tensor format

In the present survey, we consider a rank approximation algorithm for tensors represented in the canonical format in arbitrary pre-Hilbert tensor product spaces. It is shown that the original approximation problem is equivalent to a finite dimensional l 2 minimization problem. The l 2 minimization problem is solved by a regularized Newton method which requires the computation and evaluation of the first and second derivative of the objective function. A systematic choice of the initial guess for the iterative scheme is introduced. The effectiveness of the approach is demonstrated in numerical experiments.

A regularized Newton method for locating thin tubular conductivity inhomogeneities

We consider the inverse problem of determining the position and shape of a thin tubular object, such as for instance a wire, a thin channel or a curve-like crack, embedded in some three-dimensional homogeneous body from a single measurement of electrostatic currents and potentials on the boundary of the body. Using an asymptotic model describing perturbations of electrostatic potentials caused by such thin objects, we reformulate the inverse problem as a nonlinear operator equation. We establish Fréchet differentiability of the corresponding operator, compute its Fréchet derivative and set up a regularized Newton scheme to solve the inverse problem numerically. We discuss our implementation of this method and present numerical results for simulated forward data.

A regularized Newton method for degenerate unconstrained optimization problems

A regularized Newton method is presented in this paper to solve unconstrained nonconvex minimization problems without the nonsingularity assumption of solutions. The modified Cholesky factorization method proposed by Cheng and Higham is employed to calculate the regularized Newton step. Under suitable conditions, the global convergence of the regularized Newton method is established, and the fast local convergence result is achieved under the local error bound conditions. Some preliminary numerical results are also reported.

Parallel regularized Newton method for nonlinear ill-posed equations

We introduce a regularized Newton method coupled with the parallel splitting-up technique for solving nonlinear ill-posed equations with smooth monotone operators. We analyze the convergence of the proposed method and carry out numerical experiments for nonlinear integral equations.

A modified smoothing and regularized Newton method for monotone second-order cone complementarity problems

In this paper, we propose a globally and quadratically convergent Newton-type algorithm for solving monotone second-order cone complementarity problems (denoted by SOCCPs). This algorithm is based on smoothing and regularization techniques by incorporating smoothing Newton’s method. Many Newton-type methods with smoothing and regularization techniques have been studied for solving nonlinear complementarity problems (NCPs) and box constrained variational inequalities (BVIs). Our algorithm is regarded as an extension of those methods to SOCCP. However, it is different from the existing methods, because we solve SOCCP by treating both the smoothing parameter μ and the regularization parameter ε as independent variables. In addition, numerical experiments indicate that the proposed method is quite effective.

A NEWTON-IMPLICIT ITERATIVE METHOD FOR NONLINEAR INVERSE PROBLEMS

A regularized Newton method for nonlinear ill-posed problems is considered. In each Newton step an implicit iterative method with an appropriate stopping rule is proposed and analyzed. Under certain assumptions on the nonlinear operator, the convergence of the algorithm is proved and the algorithm is stable if the discrepancy principle is used to terminate the outer iteration. Numerical experiment shows the effectiveness of the method.

Acceleration techniques for regularized Newton methods applied to electromagnetic inverse medium scattering problems

We study the construction and updating of spectral preconditioners for regularized Newton methods and their application to electromagnetic inverse medium scattering problems. Moreover, we show how a Lepski-type stopping rule can be implemented efficiently for these methods. In numerical examples, the proposed method compares favorably with other iterative regularization method in terms of work-precision diagrams for exact data. For data perturbed by random noise, the Lepski-type stopping rule performs considerably better than the commonly used discrepancy principle.

Investigation of Preconditioning Techniques for the Iteratively Regularized Gauss–Newton Method for Exponentially Ill-Posed Problems

We examine preconditioning techniques for exponentially ill-posed problems which significantly reduce the total complexity of regularized Newton methods. The construction of the preconditioners exploits the close connection of the CG method, which is applied to solve the linear systems, and Lanczos' method, which is used to determine spectral data to construct and update spectral preconditioners. In examples from acoustic and electromagnetic scattering problems we show the superiority of our preconditioned Newton methods when compared with other kinds of Newton methods.

A Subspace Minimization Method for the Trust-Region Step

We consider methods for large-scale unconstrained minimization based on finding an approximate minimizer of a quadratic function subject to a two-norm trust-region constraint. The Steihaug-Toint method uses the conjugate-gradient method to minimize the quadratic over a sequence of expanding subspaces until the iterates either converge to an interior point or cross the constraint boundary. However, if the conjugate-gradient method is used with a preconditioner, the Steihaug-Toint method requires that the trust-region norm be defined in terms of the preconditioning matrix. If a different preconditioner is used for each subproblem, the shape of the trust-region can change substantially from one subproblem to the next, which invalidates many of the assumptions on which standard methods for adjusting the trust-region radius are based. In this paper we propose a method that allows the trust-region norm to be defined independently of the preconditioner. The method solves the inequality constrained trust-region subproblem over a sequence of evolving low-dimensional subspaces. Each subspace includes an accelerator direction defined by a regularized Newton method for satisfying the optimality conditions of a primal-dual interior method. A crucial property of this direction is that it can be computed by applying the preconditioned conjugate-gradient method to a positive-definite system in both the primal and dual variables of the trust-region subproblem. Numerical experiments on problems from the CUTEr test collection indicate that the method can require significantly fewer function evaluations than other methods. In addition, experiments with general-purpose preconditioners show that it is possible to significantly reduce the number of matrix-vector products relative to those required without preconditioning.

Convergence Properties of the Regularized Newton Method for the Unconstrained Nonconvex Optimization

The regularized Newton method (RNM) is one of the efficient solution methods for the unconstrained convex optimization. It is well-known that the RNM has good convergence properties as compared to the steepest descent method and the pure Newton’s method. For example, Li, Fukushima, Qi and Yamashita showed that the RNM has a quadratic rate of convergence under the local error bound condition. Recently, Polyak showed that the global complexity bound of the RNM, which is the first iteration k such that ‖∇f(xk)‖≤e, is O(e−4), where f is the objective function and e is a given positive constant. In this paper, we consider a RNM extended to the unconstrained “nonconvex” optimization. We show that the extended RNM (E-RNM) has the following properties. (a) The E-RNM has a global convergence property under appropriate conditions. (b) The global complexity bound of the E-RNM is O(e−2) if ∇2f is Lipschitz continuous on a certain compact set. (c) The E-RNM has a superlinear rate of convergence under the local error bound condition.

A new smoothing and regularization Newton method for P 0-NCP

The nonlinear complementarity problem (denoted by NCP(F)) can be reformulated as the solution of a nonsmooth system of equations. In this paper, we propose a new smoothing and regularization Newton method for solving nonlinear complementarity problem with P 0-function (P 0-NCP). Without requiring strict complementarity assumption at the P 0-NCP solution, the proposed algorithm is proved to be convergent globally and superlinearly under suitable assumptions. Furthermore, the algorithm has local quadratic convergence under mild conditions. Numerical experiments indicate that the proposed method is quite effective. In addition, in this paper, the regularization parameter ? in our algorithm is viewed as an independent variable, hence, our algorithm seems to be simpler and more easily implemented compared to many previous methods.

A Newton method for reconstructing non star-shaped domains in electrical impedance tomography

We study the reconstruction of the shape of a perfectly conducting inclusionin three dimensional electrical impedance tomography (EIT) using a regularizedNewton method. This method involves a least squares penalty in theform of an additional nonlinear operator to cope with the non-uniquenessof general parametrizations of the unknown boundary.We provide a convergence result for thismethod in the general framework of nonlinear ill-posed operator equations.Moreover, we discuss the evaluation of the forward operator in EIT,its derivative, and the adjoint of the derivative using a wavelet basedboundary element method.Numerical examples illustrate the performance of our method.

Iterative Methods for Finding a Trust-region Step

We consider methods for large-scale unconstrained minimization based on finding an approximate minimizer of a quadratic function subject to a two-norm trust-region inequality constraint. The Steihaug-Toint method uses the conjugate-gradient algorithm to minimize the quadratic over a sequence of expanding subspaces until the iterates either converge to an interior point or cross the constraint boundary. The benefit of this approach is that an approximate solution may be obtained with minimal work and storage. However, the method does not allow the accuracy of a constrained solution to be specified. We propose an extension of the Steihaug-Toint method that allows a solution to be calculated to any prescribed accuracy. If the Steihaug-Toint point lies on the boundary, the constrained problem is solved on a sequence of evolving low-dimensional subspaces. Each subspace includes an accelerator direction obtained from a regularized Newton method applied to the constrained problem. A crucial property of this direction is that it can be computed by applying the conjugate-gradient method to a positive-definite system in both the primal and dual variables of the constrained problem. The method includes a parameter that allows the user to take advantage of the tradeoff between the overall number of function evaluations and matrix-vector products associated with the underlying trust-region method. At one extreme, a low-accuracy solution is obtained that is comparable to the Steihaug-Toint point. At the other extreme, a high-accuracy solution can be specified that minimizes the overall number of function evaluations at the expense of more matrix-vector products.

Iteratively Regularized Gauss–Newton Method for Nonlinear Inverse Problems with Random Noise

We study the convergence of regularized Newton methods applied to nonlinear operator equations in Hilbert spaces if the data are perturbed by random noise. It is shown that the expected square error is bounded by a constant times the minimax rates of the corresponding linearized problem if the stopping index is chosen using a priori knowledge of the smoothness of the solution. For unknown smoothness the stopping index can be chosen adaptively based on Lepski's balancing principle. For this stopping rule we establish an oracle inequality, which implies order optimal rates for deterministic errors, and optimal rates up to a logarithmic factor for random noise. The performance and the statistical properties of the proposed method are illustrated by Monte Carlo simulations.

Existence of solution to an evolution equation and a justification of the DSM for equations with monotone operators

An evolution equation, arising in the study of the Dynamical Systems Method (DSM) for solving equations with monotone operators, is studied in this paper. The evolution equation is a continuous analog of the regularized Newton method for solving ill-posed problems with monotone nonlinear operators F . Local and global existence of the unique solution to this evolution equation are proved, apparently for the firs time, under the only assumption that F (u) exists and is continuous with respect to u. The earlier published results required more smoothness of F . The Dynamical Systems method (DSM) for solving equations F (u) = 0 with monotone Frechet differentiable operator F is justified under the above assumption apparently for the first time.

Numerical simulation for the shape reconstruction of a cavity

AbstractWe consider the inverse problem of reconstructing the interior boundary curve of a cavity from the knowledge of the measurements on the exterior boundary. The domain derivative of the corresponding operator is presented, and this allows the investigation of the regularized Newton method for the solution of the ill‐posed and nonlinear problem. Numerical examples indicate the feasibility of our method. © 2008 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq, 2009

Newton methods for nonlinear inverse problems with random noise

AbstractWe study the convergence of regularized Newton methods applied to nonlinear operator equations in Hilbert spaces if the data are perturbed by random noise. We show that under certain conditions it is possible to achieve the minimax rates of the corresponding linearized problem if the smoothness of the solution is known. If the smoothness is unknown and the stopping index is determined by Lepskij's balancing principle, we show that the rates remain the same up to a logarithmic factor due to adaptation. (© 2008 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)

Shape reconstruction for the time‐dependent Navier‐Stokes flow

AbstractThis article deals with the shape reconstruction of a bounded domain with a viscous incompressible fluid driven by the time‐dependent Navier‐Stokes equations. For the approximate solution of the ill‐posed and nonlinear problem we propose a regularized Newton method. A theoretical foundation for the Newton method is given by establishing the differentiability of the initial boundary value problem with respect to the interior boundary curve in the sense of the domain derivative. Numerical examples indicate the feasibility of our method. © 2007 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq, 2008

Truncated regularized Newton method for convex minimizations

Recently, Li et al. (Comput. Optim. Appl. 26:131---147, 2004) proposed a regularized Newton method for convex minimization problems. The method retains local quadratic convergence property without requirement of the singularity of the Hessian. In this paper, we develop a truncated regularized Newton method and show its global convergence. We also establish a local quadratic convergence theorem for the truncated method under the same conditions as those in Li et al. (Comput. Optim. Appl. 26:131---147, 2004). At last, we test the proposed method through numerical experiments and compare its performance with the regularized Newton method. The results show that the truncated method outperforms the regularized Newton method.