Newton-type Method Research Articles

Isolated Calmness of Perturbation Mappings and Superlinear Convergence of Newton-Type Methods

AbstractIn this paper, we characterize Lipschitzian properties of different multiplier-free and multiplier-dependent perturbation mappings associated with the stationarity system of a so-called generalized nonlinear program popularized by Rockafellar. Special emphasis is put on the investigation of the isolated calmness property at and around a point. The latter is decisive for the locally fast convergence of the so-called semismooth* Newton-type method by Gfrerer and Outrata. Our central result is the characterization of the isolated calmness at a point of a multiplier-free perturbation mapping via a combination of an explicit condition and a rather mild assumption, automatically satisfied e.g. for standard nonlinear programs. Isolated calmness around a point is characterized analogously by a combination of two stronger conditions. These findings are then related to so-called criticality of Lagrange multipliers, as introduced by Izmailov and extended to generalized nonlinear programming by Mordukhovich and Sarabi. We derive a new sufficient condition (a characterization for some problem classes) of nonexistence of critical multipliers, which has been also used in the literature as an assumption to guarantee local fast convergence of Newton-, SQP-, or multiplier-penalty-type methods. The obtained insights about critical multipliers seem to complement the vast literature on the topic.

Fast convergence to non-isolated minima: four equivalent conditions for $${\textrm{C}^{2}}$$ functions

AbstractOptimization algorithms can see their local convergence rates deteriorate when the Hessian at the optimum is singular. These singularities are inescapable when the optima are non-isolated. Yet, under the right circumstances, several algorithms preserve their favorable rates even when optima form a continuum (e.g., due to over-parameterization). This has been explained under various structural assumptions, including the Polyak–Łojasiewicz condition, Quadratic Growth and the Error Bound. We show that, for cost functions which are twice continuously differentiable ($$\textrm{C}^2$$ C 2 ), those three (local) properties are equivalent. Moreover, we show they are equivalent to the Morse–Bott property, that is, local minima form differentiable submanifolds, and the Hessian of the cost function is positive definite along its normal directions. We leverage this insight to improve local convergence guarantees for safe-guarded Newton-type methods under any (hence all) of the above assumptions. First, for adaptive cubic regularization, we secure quadratic convergence even with inexact subproblem solvers. Second, for trust-region methods, we argue capture can fail with an exact subproblem solver, then proceed to show linear convergence with an inexact one (Cauchy steps).

Bouligand–Newton type methods for non-smooth ill-posed problems

Abstract We consider Newton-type methods for solving nonlinear ill-posed inverse problems in Hilbert spaces where the forward operators are not necessarily G\^{a}teaux differentiable. Modifications are proposed with the non-existing Fr'{e}chet derivatives replaced by a family of bounded linear operators satisfying suitable properties. These bounded linear operators can be constructed by the Bouligand subderivatives which are defined as limits of Fr'{e}chet derivatives of the forward operator in differentiable points. The Bouligand subderivative mapping in general is not continuous unless the forward operator is G\^{a}teaux differentiable which
introduces challenges for convergence analysis of the corresponding Bouligand-Newton type methods. In this paper we will show that, under the discrepancy principle, these Bouligand-Newton type methods are iterative regularization methods of optimal order. Numerical results for an inverse problem arising from a non-smooth semi-linear elliptic equation are presented to test the performance of the methods.

An adaptive regularized proximal Newton-type methods for composite optimization over the Stiefel manifold

An adaptive regularized proximal Newton-type methods for composite optimization over the Stiefel manifold

Differentiable Voronoi Diagrams for Simulation of Cell-Based Mechanical Systems

Navigating topological transitions in cellular mechanical systems is a significant challenge for existing simulation methods. While abstract models lack predictive capabilities at the cellular level, explicit network representations struggle with topology changes, and per-cell representations are computationally too demanding for large-scale simulations. To address these challenges, we propose a novel cell-centered approach based on differentiable Voronoi diagrams. Representing each cell with a Voronoi site, our method defines shape and topology of the interface network implicitly. In this way, we substantially reduce the number of problem variables, eliminate the need for explicit contact handling, and ensure continuous geometry changes during topological transitions. Closed-form derivatives of network positions facilitate simulation with Newton-type methods for a wide range of per-cell energies. Finally, we extend our differentiable Voronoi diagrams to enable coupling with arbitrary rigid and deformable boundaries. We apply our approach to a diverse set of examples, highlighting splitting and merging of cells as well as neighborhood changes. We illustrate applications to inverse problems by matching soap foam simulations to real-world images. Comparative analysis with explicit cell models reveals that our method achieves qualitatively comparable results at significantly faster computation times.

Data-Driven Method for Robust Recovery in 1-Bit Compressive Sensing with the Minimax Concave Penalty

With the advent of large-scale data, the demand for information is increasing, which makes signal sampling technology and digital processing methods particularly important. The utilization of 1-bit compressive sensing in sparse recovery has garnered significant attention due to its cost-effectiveness in hardware implementation and storage. In this paper, we first leverage the minimax concave penalty equipped with the least squares to recover a high-dimensional true signal x∈Rp with k-sparse from n-dimensional 1-bit measurements and discuss the regularization by combing the nonconvex sparsity-inducing penalties. Moreover, we give an analysis of the complexity of the method with minimax concave penalty in certain conditions and derive the general theory for the model equipped with the family of sparsity-inducing nonconvex functions. Then, our approach employs a data-driven Newton-type method with stagewise steps to solve the proposed method. Numerical experiments on the synthesized and real data verify the competitiveness of the proposed method.

A novel Newton method for inverse elastic scattering problems

This work is concerned with an inverse elastic scattering problem of identifying the unknown rigid obstacle embedded in an open space filled with a homogeneous and isotropic elastic medium. A Newton-type iteration method relying on the boundary condition is designed to identify the boundary curve of the obstacle. Based on the Helmholtz decomposition and the Fourier–Bessel expansion, we explicitly derive the approximate scattered field and its derivative on each iterative curve. Rigorous mathematical justifications for the proposed method are provided. Numerical examples are presented to verify the effectiveness of the proposed method.

Analytic rotation-invariant modelling of anisotropic finite elements

Anisotropic hyperelastic distortion energies are used to solve many problems in fields like computer graphics and engineering with applications in shape analysis, deformation, design, mesh parameterization, biomechanics and more. However, formulating a robust anisotropic energy that is low-order and yet sufficiently non-linear remains a challenging problem for achieving the convergence promised by Newton-type methods in numerical optimization. In this paper, we propose a novel analytic formulation of an anisotropic energy that is smooth everywhere, low-order, rotationally-invariant and at-least twice differentiable. At its core, our approach utilizes implicit rotation factorizations with invariants of the Cauchy-Green tensor that arises from the deformation gradient. The versatility and generality of our analysis is demonstrated through a variety of examples, where we also show that the constitutive law suggested by the anisotropic version of the well-known As-Rigid-As-Possible energy is the foundational parametric description of both passive and active elastic materials. The generality of our approach means that we can systematically derive the force and force-Jacobian expressions for use in implicit and quasistatic numerical optimization schemes, and we can also use our analysis to rewrite, simplify and speedup several existing anisotropic and isotropic distortion energies with guaranteed inversion-safety.

On the Kantorovich Theory for Nonsingular and Singular Equations

We develop a new Kantorovich-like convergence analysis of Newton-type methods to solve nonsingular and singular nonlinear equations in Banach spaces. The outer or generalized inverses are exchanged by a finite sum of linear operators making the implementation of these methods easier than in earlier studies. The analysis uses relaxed generalized continuity of the derivatives of operators involved required to control the derivative and on real majorizing sequences. The same approach can also be implemented on other iterative methods with inverses. The examples complement the theory by verifying the convergence conditions and demonstrating the performance of the methods.

Real-time time-varying economic nonlinear model predictive control for wind turbines

Economic nonlinear model predictive control is a great choice to tackle the control issues appearing in the current wind energy industry. In this paper, instead of generator power, aerodynamic power is employed in the economic cost function to establish the required conditions of stability and convergence of the economic performance, such as turnpike and problem convexity. However, since aerodynamic power depends on time-varying wind speed, conventional time-invariant economic nonlinear model predictive controllers cannot guarantee the stability and convergence of economic performance. This paper proposes a new time-varying economic nonlinear model predictive controller for wind turbine control that considers an economic trajectory, instead of a steady-state, in its optimization problem. The proposed time-varying economic cost function directly considers aerodynamic power, the activity of pitch angle and generator torque, and fatigue loads on the shaft and tower. Therefore, this controller can maximize power extraction and reduce fatigue load on the tower, drivetrain, and actuators. Furthermore, a fast-parallel Newton-type method is used to implement the proposed controller in actual wind turbines. An accurate aeroelastic model is used to validate the performance of the proposed control scheme. The proposed controller is also compared with two tracking nonlinear model predictive controllers, the baseline controller, and a newly developed method, under fatigue and extreme load scenarios in the presence and absence of uncertainty. The simulation results show the superior economic performance of the proposed approach. Moreover, the real-time results verify the computational speed that the proposed controller requires to deploy actual wind turbines.

A dimension expanded Newton-type method for absolute value equations

A dimension expanded Newton-type method for absolute value equations

An extended Gauss-Newton method for full-waveform inversion

Full-waveform inversion (FWI) is a large-scale nonlinear ill-posed problem for which computationally expensive Newton-type methods can become trapped in undesirable local minima, particularly when the initial model lacks a low-wavenumber component and the recorded data lack low-frequency content. A modification to the Gauss-Newton (GN) method is developed to address these issues. The standard GN system for multisource multireceiver FWI is reformulated into an equivalent matrix equation form, with the solution becoming a diagonal matrix rather than a vector as in the standard system. The search direction is transformed from a vector to a matrix by relaxing the diagonality constraint, effectively adding a degree of freedom to the subsurface offset axis. The relaxed system can be explicitly solved with only the inversion of two small matrices that deblur the data residual matrix along the source and receiver dimensions, which simplifies the inversion of the Hessian matrix. When used to solve the extended-source FWI objective function, the extended GN (EGN) method integrates the benefits of model and source extension. The EGN method effectively combines the computational effectiveness of the reduced FWI method with the robustness characteristics of extended formulations and offers a promising solution for addressing the challenges of FWI. It bridges the gap between these extended formulations and the reduced FWI method, enhancing inversion robustness while maintaining computational efficiency. The robustness and stability of the EGN algorithm for waveform inversion are demonstrated numerically.

Asymptotically Newton-Type Methods without Inverses for Solving Equations

The implementation of Newton’s method for solving nonlinear equations in abstract domains requires the inversion of a linear operator at each step. Such an inversion may be computationally very expensive or impossible to find. That is why alternative iterative methods are developed in this article that require no inversion or only one inversion of a linear operator at each step. The inverse of the operator is replaced by a frozen sum of linear operators depending on the Fréchet derivative of an operator. The numerical examples illustrate that for all practical purposes, the new methods are as effective as Newton’s but much cheaper to implement. The same methodology can be used to create similar alternatives to other methods using inversions of linear operators such as divided differences or other linear operators.

Communication Efficient Distributed Newton Method over Unreliable Networks

Distributed optimization in resource constrained devices demands both communication efficiency and fast convergence rates. Newton-type methods are getting preferable due to their superior convergence rates compared to the first-order methods. In this paper, we study a new problem in regard to the second-order distributed optimization over unreliable networks. The working devices are power-limited or operate in unfavorable wireless channels, experiencing packet losses during their uplink transmission to the server. Our scenario is very common in real-world and leads to instability of classical distributed optimization methods especially the second-order methods because of their sensitivity to the imprecision of local Hessian matrices. To achieve robustness to high packet loss, communication efficiency and fast convergence rates, we propose a novel distributed second-order method, called RED-New (Packet loss Resilient Distributed Approximate Newton). Each iteration of RED-New comprises two rounds of light-weight and lossy transmissions, in which the server aggregates the local information with a new developed scaling strategy. We prove the linear-quadratic convergence rate of RED-New. Experimental results demonstrate its advantage over first-order and second-order baselines, and its tolerance to packet loss rate ranging from 5% to 40%.

On the Existence Theorem of a Three-Step Newton-Type Method Under Weak L-Average

On the Existence Theorem of a Three-Step Newton-Type Method Under Weak L-Average

On Local Behavior of Newton-Type Methods Near Critical Solutions of Constrained Equations

On Local Behavior of Newton-Type Methods Near Critical Solutions of Constrained Equations

Variant of Trapezoidal-Newton Method for Solving Nonlinear Equations and its Dynamics

This article introduces a novel approach resulting from the adaptation of Trapezoidal-Newton method variants. The iterative process is enhanced through the incorporation of a numerical integral strategy derived from two-partition Trapezoidal method. Through rigorous error analysis, the study establishes a third order convergence for this method. It emerges as a viable alternative for solving nonlinear equations, a conclusion substantiated by computational costs conducted on diverse nonlinear equation forms. Furthermore, an exploration of basin of attraction analyses that this method exhibits faster convergence compared to other Newton-type methods, albeit with a slightly expanded divergent region with a variant of Newton Simpson’s method.

Location, separation and approximation of solutions of nonlinear Hammerstein-type integral equations

From Newton's method, we construct a Newton-type iterative method that allows studying a class of nonlinear Hammerstein-type integral equations. This method is reduced to Newton's method if the kernel of the integral equation is separable and, unlike Newton's method, can be applied to approximate a solution if the kernel is nonseparable. In addition, from an analysis of the global convergence of the method, we can locate and separate solutions of the nonlinear Hammerstein-type integral equations involved. For this study of the global convergence, we use auxiliary functions and obtain restricted global convergence domains that are usually balls.

Behavior of Newton-Type Methods Near Critical Solutions of Nonlinear Equations with Semismooth Derivatives

Behavior of Newton-Type Methods Near Critical Solutions of Nonlinear Equations with Semismooth Derivatives

The domain derivative in inverse obstacle scattering with nonlinear impedance boundary condition

In this paper an inverse obstacle scattering problem for the Helmholtz equation with nonlinear impedance boundary condition is considered. For a certain class of nonlinearities, well-posedness of the direct scattering problem is proven. Furthermore, differentiability of solutions with respect to the boundary is shown by the variational method. A characterization of the derivative allows for iterative regularization schemes in solving the inverse problem, which consists in reconstructing the scattering obstacle from the far field pattern of a scattered wave. An all-at-once Newton-type regularization method is developed to illustrate the use of the domain derivative by some numerical examples.