Regularized Newton Method Research Articles

Accelerated Regularized Newton Methods for Minimizing Composite Convex Functions

In this paper, we study accelerated regularized Newton methods for minimizing objectives formed as a sum of two functions: one is convex and twice differentiable with Hölder-continuous Hessian, and the other is a simple closed convex function. For the case in which the Hölder parameter $\nu\in [0,1]$ is known, we propose methods that take at most $\mathcal{O}\big({1 \over \epsilon^{1/(2+\nu)}}\big)$ iterations to reduce the functional residual below a given precision $\epsilon > 0$. For the general case, in which the $\nu$ is not known, we propose a universal method that ensures the same precision in at most $\mathcal{O}\big({1 \over \epsilon^{2/[3(1+\nu)]}}\big)$ iterations without using $\nu$ explicitly in the scheme.

A class of homotopy with regularization for nonlinear ill-posed problems in Hilbert space

Abstract Nonlinear ill-posed problems arise in many inverse problems in Hilbert space. We investigate the homotopy method, which can obtain global convergence to solve the problems. The “homotopy with Tikhonov regularization” and “homotopy without derivative” are developed in this paper. The existence of the homotopy curve is proved. Several numerical schemes for tracing the homotopy curve are given, including adaptive tracing skills. Compared to the regularized Newton method, the numerical examples show that our proposed methods are stable and effective.

A new modified trust region algorithm for solving unconstrained optimization problems

Iterative methods for optimization can be classified into two categories: line search methods and trust region methods. In this paper, we propose a modified regularized Newton method without line search for minimizing nonconvex functions whose Hessian matrix may be singular. The proposed method is proved to converge globally if the Gradient and Hessian of the objective function are Lipschitz continuous. Moreover, we report numerical results that show that the proposed algorithm is competitive with the existing methods.

An inexact regularized Newton framework with a worst-case iteration complexity of $ {\mathscr O}(\varepsilon^{-3/2}) $ for nonconvex optimization

Abstract An algorithm for solving smooth nonconvex optimization problems is proposed that, in the worst-case, takes ${\mathscr O}(\varepsilon ^{-3/2})$ iterations to drive the norm of the gradient of the objective function below a prescribed positive real number $\varepsilon $ and can take ${\mathscr O}(\varepsilon ^{-3})$ iterations to drive the leftmost eigenvalue of the Hessian of the objective above $-\varepsilon $. The proposed algorithm is a general framework that covers a wide range of techniques including quadratically and cubically regularized Newton methods, such as the Adaptive Regularization using Cubics (arc) method and the recently proposed Trust-Region Algorithm with Contractions and Expansions (trace). The generality of our method is achieved through the introduction of generic conditions that each trial step is required to satisfy, which in particular allows for inexact regularized Newton steps to be used. These conditions center around a new subproblem that can be approximately solved to obtain trial steps that satisfy the conditions. A new instance of the framework, distinct from arc and trace, is described that may be viewed as a hybrid between quadratically and cubically regularized Newton methods. Numerical results demonstrate that our hybrid algorithm outperforms a cubically regularized Newton method.

A new regularized quasi-Newton method for unconstrained optimization

In this paper, we propose a new regularized quasi-Newton method for unconstrained optimization. At each iteration, a regularized quasi-Newton equation is solved to obtain the search direction. The step size is determined by a non-monotone Armijo backtracking line search. An adaptive regularized parameter, which is updated according to the step size of the line search, is employed to compute the next search direction. The presented method is proved to be globally convergent. Numerical experiments show that the proposed method is effective for unconstrained optimizations and outperforms the existing regularized Newton method.

Adaptive Quadratically Regularized Newton Method for Riemannian Optimization

Optimization on Riemannian manifolds widely arises in eigenvalue computation, density functional theory, Bose--Einstein condensates, low rank nearest correlation, image registration, signal process...

A Regularized Newton Method for Computing Ground States of Bose–Einstein Condensates

In this paper, we propose a regularized Newton method for computing ground states of Bose-Einstein condensates (BECs), which can be formulated as an energy minimization problem with a spherical constraint. The energy functional and constraint are discretized by either the finite difference, or sine or Fourier pseudospectral discretization schemes and thus the original infinite dimensional nonconvex minimization problem is approximated by a finite dimensional constrained nonconvex minimization problem. Then an initial solution is first constructed by using a feasible gradient type method, which is an explicit scheme and maintains the spherical constraint automatically. To accelerate the convergence of the gradient type method, we approximate the energy functional by its second-order Taylor expansion with a regularized term at each Newton iteration and adopt a cascadic multigrid technique for selecting initial data. It leads to a standard trust-region subproblem and we solve it again by the feasible gradient type method. The convergence of the regularized Newton method is established by adjusting the regularization parameter as the standard trust-region strategy. Extensive numerical experiments on challenging examples, including a BEC in three dimensions with an optical lattice potential and rotating BECs in two dimensions with rapid rotation and strongly repulsive interaction, show that our method is efficient, accurate and robust.

Regularized Newton Methods for Minimizing Functions with Hölder Continuous Hessians

In this paper, we study the regularized second-order methods for unconstrained minimization of a twice-differentiable (convex or nonconvex) objective function. For the current function, these metho...

Erratum to: A regularized Newton method without line search for unconstrained optimization

Erratum to: A regularized Newton method without line search for unconstrained optimization

Backward–forward algorithms for structured monotone inclusions in Hilbert spaces

In this paper, we study the backward–forward algorithm as a splitting method to solve structured monotone inclusions, and convex minimization problems in Hilbert spaces. It has a natural link with the forward–backward algorithm and has the same computational complexity, since it involves the same basic blocks, but organized differently. Surprisingly enough, this kind of iteration arises when studying the time discretization of the regularized Newton method for maximally monotone operators. First, we show that these two methods enjoy remarkable involutive relations, which go far beyond the evident inversion of the order in which the forward and backward steps are applied. Next, we establish several convergence properties for both methods, some of which were unknown even for the forward–backward algorithm. This brings further insight into this well-known scheme. Finally, we specialize our results to structured convex minimization problems, the gradient-projection algorithms, and give a numerical illustration of theoretical interest.

A new smoothing and regularization Newton method for the symmetric cone complementarity problem

Based on a regularized Chen–Harker–Kanzow–Smale (CHKS) smoothing function, we propose a new smoothing and regularization Newton method for solving the symmetric cone complementarity problem. By using the theory of Euclidean Jordan algebras, we establish the global and local quadratic convergence of the method on certain assumptions. The proposed method uses a nonmonotone line search technique which includes the usual monotone line search as a special case. In addition, our method treats both the smoothing parameter $$\mu $$ and the regularization parameter $$\varepsilon $$ as independent variables. Preliminary numerical results are reported which indicate that the proposed method is effective.

Regularized Newton methods for x-ray phase contrast and general imaging problems.

Like many other advanced imaging methods, x-ray phase contrast imaging and tomography require mathematical inversion of the observed data to obtain real-space information. While an accurate forward model describing the generally nonlinear image formation from a given object to the observations is often available, explicit inversion formulas are typically not known. Moreover, the measured data might be insufficient for stable image reconstruction, in which case it has to be complemented by suitable a priori information. In this work, regularized Newton methods are presented as a general framework for the solution of such ill-posed nonlinear imaging problems. For a proof of principle, the approach is applied to x-ray phase contrast imaging in the near-field propagation regime. Simultaneous recovery of the phase- and amplitude from a single near-field diffraction pattern without homogeneity constraints is demonstrated for the first time. The presented methods further permit all-at-once phase contrast tomography, i.e. simultaneous phase retrieval and tomographic inversion. We demonstrate the potential of this approach by three-dimensional imaging of a colloidal crystal at 95nm isotropic resolution.

A Regularized Newton Method with Correction for Unconstrained Convex Optimization

In this paper, we present a regularized Newton method (M-RNM) with correction for minimizing a convex function whose Hessian matrices may be singular. At every iteration, not only a RNM step is computed but also two correction steps are computed. We show that if the objective function is LC2, then the method posses globally convergent. Numerical results show that the new algorithm performs very well.

A one-parametric class of smoothing functions and an improved regularization Newton method for the NCP

In this paper, we introduce a one-parametric class of smoothing functions, which enjoys some favourable properties and includes two famous smoothing functions as special cases. Based on this class of smoothing functions, we propose a regularization Newton method for solving the non-linear complementarity problem. The main feature of the proposed method is that it uses a perturbed Newton equation to obtain the direction. This not only allows our method to have global and local quadratic convergences without strict complementarity conditions, but also makes the regularization parameter converge to zero globally Q-linearly. In addition, we use a new non-monotone line search scheme to obtain the step size. Some numerical results are reported which confirm the good theoretical properties of the proposed method.

A non-monotone regularization Newton method for the second-order cone complementarity problem

Based on the smoothing Newton method and the Tikhonov regularization method, we construct a regularization Newton method for the second-order cone complementarity problem. The method uses a non-monotone line search scheme which contains the usual monotone line search as a special case. By using the theory of Euclidean Jordan algebras, we prove that the proposed method is globally and locally quadratically convergent under suitable assumptions. Some numerical results are reported which indicate the effectiveness of the method.

A globally and quadratically convergent primal–dual augmented Lagrangian algorithm for equality constrained optimization

We present a primal–dual augmented Lagrangian method to solve an equality constrained minimization problem. This is a Newton-like method applied to a perturbation of the optimality system that follows from a reformulation of the initial problem by introducing an augmented Lagrangian function. An important aspect of this approach is that, by a choice of suitable updating rules of parameters, the algorithm reduces to a regularized Newton method applied to a sequence of optimality systems. The global convergence is proved under mild assumptions. An asymptotic analysis is also presented and quadratic convergence is proved under standard regularity assumptions. Some numerical results show that the method is very efficient and robust.

A Regularized Newton Method with Correction for Unconstrained Nonconvex Optimization

In this paper, we present a modified regularized Newton method for minimizing a nonconvex function whose Hessian matrix may be singular. We show that if the gradient and Hessian of the objective function are Lipschitz continuous, then the method has a global convergence property. Under the local error bound condition which is weaker than nonsingularity, the method has cubic convergence.

A regularized Newton method without line search for unconstrained optimization

In this paper, we propose a regularized Newton method without line search. The proposed method controls a regularization parameter instead of a step size in order to guarantee the global convergence. We show that the proposed algorithm has the following convergence properties. (a) The proposed algorithm has global convergence under appropriate conditions. (b) It has superlinear rate of convergence under the local error bound condition. (c) An upper bound of the number of iterations required to obtain an approximate solution $$x$$x satisfying $$\Vert \nabla f(x) \Vert \le \varepsilon $$??f(x)?≤? is $$O(\varepsilon ^{-2})$$O(?-2), where $$f$$f is the objective function and $$\varepsilon $$? is a given positive constant.

A regularized Newton method for monotone nonlinear equations and its application

In this paper, we propose a regularized Newton method for the system of monotone nonlinear equations. The regularization parameter is taken as the norm of the residual, and a correction step with little additional calculations is also computed to compensate for the shorter trial step due to the introduction of the regularization parameter. Under the local error bound condition which is weaker than nonsingularity, we show that the new regularized Newton method with correction has quadratic convergence. We also apply the new method to the unconstrained convex optimization problems which may have singular Hessian at the solutions and develop a globally convergent regularized Newton algorithm by using trust region technique. Numerical results show that the algorithm is very efficient and robust.

Global Convergence of a Closed-Loop Regularized Newton Method for Solving Monotone Inclusions in Hilbert Spaces

We analyze the global convergence properties of some variants of regularized continuous Newton methods for convex optimization and monotone inclusions in Hilbert spaces. The regularization term is of Levenberg–Marquardt type and acts in an open-loop or closed-loop form. In the open-loop case the regularization term may be of bounded variation.