Newton-like Methods Research Articles

Scaling techniques for gradient projection-type methods in astronomical image deblurring

The aim of this paper is to present a computational study on scaling techniques in gradient projection-type (GP-type) methods for deblurring of astronomical images corrupted by Poisson noise. In this case, the imaging problem is formulated as a non-negatively constrained minimization problem in which the objective function is the sum of a fit-to-data term, the Kullback–Leibler divergence, and a Tikhonov regularization term. The considered GP-type methods are formulated by a common iteration formula, where the scaling matrix and the step-length parameter characterize the different algorithms. Within this formulation, both first-order and Newton-like methods are analysed, with particular attention to those implementation features and behaviours relevant for the image restoration problem. The numerical experiments show that suited scaling strategies can enable the GP methods to quickly approximate accurate reconstructions and then are useful for designing effective image deblurring algorithms.

An Enhanced Matrix-Free Secant Method via Predictor-Corrector Modified Line Search Strategies for Solving Systems of Nonlinear Equations

Diagonal updating scheme is among the cheapest Newton-like methods for solving system of nonlinear equations. Nevertheless, the method has some shortcomings. In this paper, we proposed an improved matrix-free secant updating scheme via line search strategies, by using the steps of backtracking in the Armijo-type line search as a step length predictor and Wolfe-Like condition as corrector. Our approach aims at improving the overall performance of diagonal secant updating scheme. Under mild assumptions, the global convergence results have been presented. Numerical experiments verify that the proposed approach is very promising.

An Improved Diagonal Jacobian Approximation via a New Quasi-Cauchy Condition for Solving Large-Scale Systems of Nonlinear Equations

We present a new diagonal quasi-Newton update with an improved diagonal Jacobian approximation for solving large-scale systems of nonlinear equations. In this approach, the Jacobian approximation is derived based on the quasi-Cauchy condition. The anticipation has been to further improve the performance of diagonal updating, by modifying the quasi-Cauchy relation so as to carry some additional information from the functions. The effectiveness of our proposed scheme is appraised through numerical comparison with some well-known Newton-like methods.

Expanding the applicability of two-point Newton-like methods under generalized conditions

We use a two-point Newton-like method to approximate a locally unique solution of a nonlinear equation containing a non-differentiable term in a Banach space setting. Using more precise majorizing sequences than in earlier studies, we present a tighter se

First-Order Continuous Newton-like Systems for Monotone Inclusions

In this paper, we propose and investigate nonautonomous first-order differential systems governed by monotone operators. They can be regarded as alternative formulations of second-order continuous Newton-like methods (with asymptotically vanishing hyperbolic regularization). Existence, uniqueness, and weak convergence results to equilibria are established in the setting of Hilbert spaces relative to any maximal monotone operator. Significant convergence rates are also obtained, while our systems involve only operator evaluation.

On the local convergence of fast two-step Newton-like methods for solving nonlinear equations

A local convergence analysis is presented for a fast two-step Newton-like method (TSNLM) for solving nonlinear equations in a Banach space setting. The TSNLM unifies earlier methods such as Newton’s, Secant, Newton-like, Chebyshev–Secant, Chebyshev–Newton, Steffensen, Stirling’s and other single or multistep methods. Numerical examples and a comparative study of these methods validating our theoretical results are also given in the concluding section of this paper.

Nonlinear least squares and Sobolev gradients

Least squares methods are effective for solving systems of partial differential equations. In the case of nonlinear systems the equations are usually linearized by a Newton iteration or successive substitution method, and then treated as a linear least squares problem. We show that it is often advantageous to form a sum of squared residuals first, and then compute a zero of the gradient with a Newton-like method. We present an effective method, based on Sobolev gradients, for treating the nonlinear least squares problem directly. The method is based on trust-region subproblems defined by a Sobolev norm and solved by a preconditioned conjugate gradient method with an effective preconditioner that arises naturally from the Sobolev space setting. The trust-region method is shown to be equivalent to a Levenberg–Marquardt method which blends a Newton or Gauss–Newton iteration with a gradient descent iteration, but uses a Sobolev gradient in place of the Euclidean gradient. We also provide an introduction to the Sobolev gradient method and discuss its relationship to operator preconditioning with equivalent operators.

Monotone convergence of Newton-like methods for M-matrix algebraic Riccati equations

For the algebraic Riccati equation whose four coefficient matrices form a nonsingular M-matrix or an irreducible singular M-matrix K, the minimal nonnegative solution can be found by Newton’s method and the doubling algorithm. When the two diagonal blocks of the matrix K have both large and small diagonal entries, the doubling algorithm often requires many more iterations than Newton’s method. In those cases, Newton’s method may be more efficient than the doubling algorithm. This has motivated us to study Newton-like methods that have higher-order convergence and are not much more expensive each iteration. We find that the Chebyshev method of order three and a two-step modified Chebyshev method of order four can be more efficient than Newton’s method. For the Riccati equation, these two Newton-like methods are actually special cases of the Newton–Shamanskii method. We show that, starting with zero initial guess or some other suitable initial guess, the sequence generated by the Newton–Shamanskii method converges monotonically to the minimal nonnegative solution.We also explain that the Newton-like methods can be used to great advantage when solving some Riccati equations involving a parameter.

New conditions for the convergence of Newton-like methods and applications

Newton-like method methods are often used for solving nonlinear equations in a Banach space setting. Using more precise majorizing sequences, we provide a tighter convergence analysis than in earlier studies such as [4,6,9,13,17–22,31–35,38–44]. Our results are illustrated by several numerical examples, for which older convergence conditions do not hold but for which our convergence criteria are satisfied.

GSGPEs: A MATLAB code for computing the ground state of systems of Gross–Pitaevskii equations

GSGPEs is a Matlab/GNU Octave suite of programs for the computation of the ground state of systems of Gross–Pitaevskii equations. It can compute the ground state in the defocusing case, for any number of equations with harmonic or quasi-harmonic trapping potentials, in spatial dimension one, two or three. The computation is based on a spectral decomposition of the solution into Hermite functions and direct minimization of the energy functional through a Newton-like method with an approximate line-search strategy. Program summaryProgram title: GSGPEsCatalogue identifier: AENT_v1_0Program summary URL: http://cpc.cs.qub.ac.uk/summaries/AENT_v1_0.htmlProgram obtainable from: CPC Program Library, Queen’s University, Belfast, N. IrelandLicensing provisions: Standard CPC licence, http://cpc.cs.qub.ac.uk/licence/licence.htmlNo. of lines in distributed program, including test data, etc.: 1417No. of bytes in distributed program, including test data, etc.: 13673Distribution format: tar.gzProgramming language: Matlab/GNU Octave.Computer: Any supporting Matlab/GNU Octave.Operating system: Any supporting Matlab/GNU Octave.RAM: About 100 MB for a single three-dimensional equation (test run output).Classification: 2.7, 4.9.Nature of problem:A system of Gross–Pitaevskii Equations (GPEs) is used to mathematically model a Bose–Einstein Condensate (BEC) for a mixture of different interacting atomic species. The equations can be used both to compute the ground state solution (i.e., the stationary order parameter that minimizes the energy functional) and to simulate the dynamics. For particular shapes of the traps, three-dimensional BECs can be also simulated by lower dimensional GPEs.Solution method:The ground state of a system of Gross–Pitaevskii equations is computed through a spectral decomposition into Hermite functions and the direct minimization of the energy functional.Running time:About 30 seconds for a single three-dimensional equation with d.o.f. 40 for each spatial direction (test run output).

An inexact restoration strategy for the globalization of the sSQP method

A globally convergent algorithm based on the stabilized sequential quadratic programming (sSQP) method is presented in order to solve optimization problems with equality constraints and bounds. This formulation has attractive features in the sense that constraint qualifications are not needed at all. In contrast with classic globalization strategies for Newton-like methods, we do not make use of merit functions. Our scheme is based on performing corrections on the solutions of the subproblems by using an inexact restoration procedure. The presented method is well defined and any accumulation point of the generated primal sequence is either a Karush-Kuhn-Tucker point or a stationary (maybe feasible) point of the problem of minimizing the infeasibility. Also, under suitable hypotheses, the sequence generated by the algorithm converges Q-linearly. Numerical experiments are given to confirm theoretical results.

Majorizing sequences for iterative procedures in Banach spaces

Newton-like methods are often used for solving nonlinear equations. In the present paper, we introduce very general majorizing sequences for Newton-like methods. Then, we provide semi-local convergence results for these methods. The new convergence results can be weaker than in earlier studies. These new results are illustrated by several numerical examples and special cases of Newton-like methods, for which the older convergence conditions do not hold but for which our weaker convergence conditions are satisfied.

Newton-like methods for efficient solutions in vector optimization

In this work we study the Newton-like methods for finding efficient solutions of the vector optimization problem for a map from a finite dimensional Hilbert space X to a Banach space Y, with respect to the partial order induced by a closed, convex and pointed cone C with a nonempty interior. We present both exact and inexact versions, in which the subproblems are solved approximately, within a tolerance. Furthermore, we prove that under reasonable hypotheses, the sequence generated by our method converges to an efficient solution of this problem.

Sample size selection in optimization methods for machine learning

This paper presents a methodology for using varying sample sizes in batch-type optimization methods for large-scale machine learning problems. The first part of the paper deals with the delicate issue of dynamic sample selection in the evaluation of the function and gradient. We propose a criterion for increasing the sample size based on variance estimates obtained during the computation of a batch gradient. We establish an $${O(1/\epsilon)}$$ complexity bound on the total cost of a gradient method. The second part of the paper describes a practical Newton method that uses a smaller sample to compute Hessian vector-products than to evaluate the function and the gradient, and that also employs a dynamic sampling technique. The focus of the paper shifts in the third part of the paper to L 1-regularized problems designed to produce sparse solutions. We propose a Newton-like method that consists of two phases: a (minimalistic) gradient projection phase that identifies zero variables, and subspace phase that applies a subsampled Hessian Newton iteration in the free variables. Numerical tests on speech recognition problems illustrate the performance of the algorithms.

Application of a modified semismooth Newton method to some elasto-plastic problems

Some elasto-plasticity models with hardening are discussed and some incremental finite element methods with different time discretisation schemes are considered. The corresponding one-time-step problems lead to variational equations with various non-linear operators. Common properties of the non-linear operators are derived and consequently a general problem is formulated. The problem can be solved by Newton-like methods. First, the semismooth Newton method is analysed. The local superlinear convergence is proved in dependence on the finite element discretisation parameter. Then it is introduced a modified semismooth Newton method which contain suitable “damping” in each Newton iteration in addition. The determination of the damping coefficients uses the fact that the investigated problem can be formulated as a minimisation one. The method is globally convergent, independently on the discretisation parameter. Moreover the local superlinear convergence also holds. The influence of inexact inner solvers is also discussed. The method is illustrated on a numerical example.

Some Newton-like methods with sharper error estimates for solving operator equations in Banach spaces

It is well known that the rate of convergence of S-iteration process introduced by Agarwal et al. (pp. 61-79) is faster than Picard iteration process for contraction operators. Following the ideas of S-iteration process, we introduce some Newton-like algorithms to solve the non-linear operator equation in Banach space setting. We study the semi-local as well as local convergence analysis of our algorithms. The rate of convergence of our algorithms are faster than the modified Newton method.Mathematics Subject Classification 2010: 49M15; 65K10; 47H10.

A Newton-like method for the numerical solution of nonlinear Fredholm-type operator equations

In a recent paper, Hernández and Salanova proposed an interesting Newton-like method for the solution of nonlinear Fredholm integral equations. The aim of the present paper is to carry over this method to nonlinear operator equations of similar type. By using a functional analytic approach, the presentation becomes much clearer and more general. Apart from this, some new results in the Banach space setting as well as in the applications are obtained.

Descentwise inexact proximal algorithms for smooth optimization

The proximal method is a standard regularization approach in optimization. Practical implementations of this algorithm require (i) an algorithm to compute the proximal point, (ii) a rule to stop this algorithm, (iii) an update formula for the proximal parameter. In this work we focus on (ii), when smoothness is present—so that Newton-like methods can be used for (i): we aim at giving adequate stopping rules to reach overall efficiency of the method. Roughly speaking, usual rules consist in stopping inner iterations when the current iterate is close to the proximal point. By contrast, we use the standard paradigm of numerical optimization: the basis for our stopping test is a “sufficient” decrease of the objective function, namely a fraction of the ideal decrease. We establish convergence of the algorithm thus obtained and we illustrate it on some ill-conditioned problems. The experiments show that combining the proposed inexact proximal scheme with a standard smooth optimization algorithm improves the numerical behaviour of the latter for those ill-conditioned problems.

Efficient, accurate and flexible finite element solvers for chemotaxis problems

In the framework of finite element discretizations, we introduce a fully nonlinear Newton-like method and a linearized second order approach in time applied to certain partial differential equations for chemotactic processes incorporating two entities, a chemical agent and the reacting population of certain biological organisms/species. We investigate the benefit of a corresponding monolithic approach and the decoupled variant. In particular, we analyze accuracy, efficiency and stability of different methods and their dependences on certain parameters in order to identify a well suited finite element solver for chemotaxis problems.

A line search filter algorithm with inexact step computations for equality constrained optimization

In this paper, a new line search filter algorithm for equality constrained optimization is presented. The approach belongs to the class of inexact Newton-like methods. It can also be regarded as an inexact version of generic sequential quadratic programming (SQP) methods. The trial step is obtained by truncatedly solving the primal–dual system based on any robust and efficient linear system solver. Practical termination tests for the linear system solver are established to ensure global convergence. Preliminary numerical results demonstrate the approach is potentially useful.