Newton-like Methods Research Articles

An efficient Newton-like conjugate gradient method with restart strategy and its application

In an attempt to enhance the theoretical structure of the Hestenes and Stiefel (HS) conjugate gradient method, several modifications of the method are provided, most of which rely on a double-truncated property to analyze its convergence properties. In this paper, a spectral HS method is proposed, which is sufficiently descent and converges globally using Powell’s restart strategy. This modification makes it possible to relax the double bounded property associated with the earlier versions of the HS method. Furthermore, the spectral parameter is motivated by some interesting theoretical features of the generalized conjugacy condition, as well as the quadratic convergence property of the Newton method. Based on some standard test problems, the numerical results reveal the advantages of the method compared to some popular conjugate gradient methods. Additionally, the method also demonstrates reliable results when applied to solve image reconstruction models.

A Unified Kantorovich-type Convergence Analysis of Newton-like Methods for Solving Generalized Equations under the Aubin Property

Numerous applications from diverse disciplines reduce to solving generalized equations in a Banach space setting. These equations are solved mostly iteratively, when a sequence is generated approximating a solution provided that certain conditions are valid on the starting point and the operators appearing on the method. In particular, Newton-like methods are developed whose specializations reduce to well known methods such as Newton, modified Newton, Secant, Kurchatov and Steffensen to mention a few. A unified semi-local analysis of these methods is presented using the contraction mapping principle under the Aubin property of a set valued operator, and generalized continuity assumption on the operators on these methods.

A Non-monotone Alternating Newton-Like Directional Method for Low-Rank and Sparse Matrix Compressive Recovery

A Non-monotone Alternating Newton-Like Directional Method for Low-Rank and Sparse Matrix Compressive Recovery

Recursion Newton-Like Algorithm for l2,0 -ReLU Deep Neural Networks.

Rectified linear unit (ReLU) deep neural network (DNN) is a classical model in deep learning and has achieved great success in many applications. However, this model is characterized by too many parameters, which not only requires huge memory but also imposes unbearable computation burden. The l2,0 regularization has become a useful technique to cope with this trouble. In this article, we design a recursion Newton-like algorithm (RNLA) to simultaneously train and compress ReLU-DNNs with l2,0 regularization. First, we reformulate the multicomposite training model into a constrained optimization problem by explicitly introducing the network nodes as the variables of the optimization. Based on the penalty function of the reformulation, we obtain two types of minimization subproblems. Second, we build the first-order optimality conditions for acquiring P-stationary points of the two subproblems, and these P-stationary points enable us to equivalently derive two sequences of stationary equations, which are piecewise linear matrix equations. We solve these equations by the column Newton-like method in group sparse subspace with lower computational scale and cost. Finally, numerical experiments are conducted on real datasets, and the results demonstrate that the proposed method RNLA is effective and applicable.

Interpolated Coefficient Mixed Finite Elements for Semilinear Time Fractional Diffusion Equations

In this paper, we consider a fully discrete interpolated coefficient mixed finite element method for semilinear time fractional reaction–diffusion equations. The classic L1 scheme based on graded meshes and new mixed finite element based on triangulation is used for the temporal and spatial discretization, respectively. The interpolation coefficient technique is used to deal with the semilinear term, and the discrete nonlinear system is solved by a Newton-like iterative method. Stability and convergence results for both the original variable and its flux are derived. Numerical experiments confirm our theoretical analysis.

Chebyshev-series solutions for nonlinear systems with hypersonic gliding trajectory example

In order to design a drag-based predictor-corrector entry guidance in the future, new analytical formulae, expressed as finite-term Chebyshev series, are developed for fast predicting a three-dimensional (3D) Hypersonic Gliding Trajectory (HGT) by proposing a Newton-like method to solve a highly nonlinear entry dynamics model. Specifically, first, a new reduced-order dynamics model is developed by properly simplifying the original dynamics model such that the drag acceleration (AD) and the ratio of the horizontal component of the lift to the drag (LDz), which are important for governing the 3D trajectory, emerge as key parameters of the dynamical equations. Then, AD and LDz are planned as polynomials of speed. However, the simplified model is still not easy to solve due to its high nonlinearity. To deal with the difficulty, a Newton-like method is proposed to transform the simplified model into Linear Differential Recurrence Equations (LDREs). By evaluating the LDREs repeatedly, a sequence of approximations of the 3D HGT can be generated, the limit of which is the trajectory specified by that simplified model. In fact, due to the fast convergence speed of the Newton-like method, the LDREs need be solved only twice to achieve high accuracy. By using Chebyshev series to approximate some complex terms appearing in the LDREs, the LDREs become analytically solvable. As a result, the approximate analytical solutions to the 3D HGT are obtained. Simulation results show that the analytical solutions need to consume less than 1 ms of time on a laptop computer and have errors of less than 4 percentage. The computational efficiency is fundamental for onboard predictor-corrector guidance.

Computing minimum-volume enclosing ellipsoids

For a given multidimensional data set (point cloud), we investigate methods for computing the minimum-volume enclosing ellipsoid (MVEE), which provides an efficient representation of the data that is useful in many applications, including data analysis, optimal design, and computational geometry. Contrary to conventional wisdom, we demonstrate that careful exploitation of problem structure can enable high-order (Newton and Newton-like) methods with superlinear convergence rates to scale to very large MVEE problems. We also introduce a hybrid method that combines the benefits of both high-order and low-order methods, along with new initialization schemes that further enhance performance. Observing that computational cost depends significantly on the particular distribution of the data, we demonstrate that kurtosis serves as an excellent indicator of problem difficulty and provides useful guidance in choosing an appropriate solution algorithm and initialization.

A family of predictor–corrector schemes for a class of variational inequalities

This work is concerned with an abstract problem in the form of a variational inequality, or equivalently a minimization problem involving a non-differential functional. The problem is inspired by a formulation of the initial–boundary value problem of elastoplasticity. The objective of this work is to revisit the predictor–corrector algorithms that are commonly used in computational applications, and to establish conditions under which these are convergent or, at least, under which they lead to decreasing sequences of the functional for the problem. The focus is on the predictor step, given that the corrector step by definition leads to a decrease in the functional. The predictor step may be formulated as a minimization problem. Attention is given to the tangent predictor, a line search approach, the method of steepest descent, and a Newton-like method. These are all shown to lead to decreasing sequences.

Newton-like Normal S-iteration under Weak Conditions

In the present paper, we introduced a quadratically convergent Newton-like normal S-iteration method free from the second derivative for the solution of nonlinear equations permitting f′(x)=0 at some points in the neighborhood of the root. Our proposed method works well when the Newton method fails and performs even better than some higher-order converging methods. Numerical results verified that the Newton-like normal S-iteration method converges faster than Fang et al.’s method. We studied different aspects of the normal S-iteration method regarding the faster convergence to the root. Lastly, the dynamic results support the numerical results and explain the convergence, divergence, and stability of the proposed method.

Extended Newton-like Midpoint Method for Solving Equations in Banach Space

In this study, we present a convergence analysis of a Newton-like midpoint method for solving nonlinear equations in a Banach space setting. The semilocal convergence is analyzed in two different ways. The first one is shown by replacing the existing conditions with weaker and tighter continuity conditions, thereby enhancing its applicability. The second one uses more general ω-continuity conditions and the majorizing principle. This approach includes only the first order Fréchet derivative and is applicable for problems that were otherwise hard to solve by using approaches seen in the literature. Moreover, the local convergence is established along with the existence and uniqueness region of the solution. The method is useful for solving Engineering and Applied Science problems. The paper ends with numerical examples that show the applicability of our convergence theorems in cases not covered in earlier studies.

A generic power flow formulation for flexible modeling and fast solving for large-scale unbalanced networks

This paper proposes a new three-phase power flow formulation for fast solving and systematic modeling based on generic formulation and modular approach. In this method, each component of the system is modeled independently as a subsystem while extra-equations are added to establish the connection between components. The augmented system equations are solved using Newton-like method and KLU sparse matrix solver. The mismatch equations are formulated with dynamic parameters to avoid the repetitive symbolic factorizations required after each change in nonzero pattern of the Jacobian matrix. Compared to conventional solution methods, the proposed solution is faster and offers greater modeling flexibility, which makes it more efficient for simulating complex and large-scale networks. The IEEE standard 34-bus and 8500-bus are used as test systems to evaluate the numerical performance, the convergence characteristics, and the accuracy of the results. The comparison with the traditional power flow methods shows significant reduction of resolution time without compromising on solution accuracy.

Neighboring extremal nonlinear model predictive control of a rigid body on SO(3)

AbstractThe issue of implementing nonlinear model predictive control (NMPC) on mechanical systems evolving on special orthogonal group (SO(3)) is taken into consideration in the first place. Necessary conditions of optimality are extracted based on Lie group variational integrators, leading to a two-point boundary value problem (TPBVP) which is solved using sensitivity derivatives and indirect shooting methods. Fast Newton-like methods referred to as fast solvers which are commonly used to solve the TPBVP are established based on the repetition of a nonlinear process. The numerical schemes employed to alleviate the computation burden consist of eliminating some constraint-related but non-essential terms in the trend of sensitivity derivatives calculation and for solving the TPBVP equations. As another claim, assuming that a first attempt to resolve the NMPC problem is accessible, the problem subjected to some changes in its initial conditions (due to some re-planning schemes) can be resolved cost-effectively based on it. Instead of solving the whole optimization process from the scratch, the optimal control inputs and states of the system are updated based on the neighboring extremal (NE) method. For this purpose, two approaches are considered: applying NE method on the first solution that leads to a neighboring optimal solution, or assisting this latter by updating the NMPC-related optimization using exact TPBVP equations at some predefined intermediate steps. It is shown through an example that the first method is not accurate enough due to error accumulations. In contrast, the second method preserves the accuracy while reducing the computation time significantly.

Dynamics of Newton-like root finding methods

When exploring the literature, it can be observed that the operator obtained when applying Newton-like root finding algorithms to the quadratic polynomials z2 − c has the same form regardless of which algorithm has been used. In this paper, we justify why this expression is obtained. This is done by studying the symmetries of the operators obtained after applying Newton-like algorithms to a family of degree d polynomials p(z) = zd − c. Moreover, we provide an iterative procedure to obtain the expression of new Newton-like algorithms. We also carry out a dynamical study of the given generic operator and provide general conclusions of this type of methods.

On the convergence of Newton-like MS method with dynamics and applications

On the convergence of Newton-like MS method with dynamics and applications

Error Estimates for Adaptive Spectral Decompositions

Adaptive spectral (AS) decompositions associated with a piecewise constant function, u, yield small subspaces where the characteristic functions comprising u are well approximated. When combined with Newton-like optimization methods for the solution of inverse medium problems, AS decompositions have proved remarkably efficient in providing at each nonlinear iteration a low-dimensional search space. Here, we derive L^2-error estimates for the AS decomposition of u, truncated after K terms, when u is piecewise constant and consists of K characteristic functions over Lipschitz domains and a background. Our estimates apply both to the continuous and the discrete Galerkin finite element setting. Numerical examples illustrate the accuracy of the AS decomposition for media that either do, or do not, satisfy the assumptions of the theory.

The Dynamics of a Continuous Newton-like Method

The objective of the current work is to invent and introduce the continuous version of Newton’s method. This scheme is used to establish some interesting properties with examples. We have plotted the fractal pattern graphs for a Newton-like method and a Damped Newton-like method in the discrete case and hence we have introduced a new concept of streamline for the continuous version of the Newton-like method. The graph and streamlines of these patterns are in agreement with numerical results and describe the convergence and stability of the proposed method to different roots when the Newton method fails.

Monotone convergence of Newton-like iteration for a structured nonlinear eigen-problem

A structured eigen-problem Ax+F(x)=λx is studied in this paper, where in applications A∈Rn×n is an irreducible Stieltjes matrix. Under certain restrictions, this problem has a unique positive solution. We show that, starting from a multiple of the positive eigenvector of A, the Newton-like algorithm for this eigen-problem is well defined and converges monotonically. Numerical results illustrate the effectiveness of this Newton-like method.

Solution of an acoustic transmission inverse problem by extended inversion

Study of a simple single-trace transmission example shows how an extended source formulation of full-waveform inversion can produce an optimization problem without spurious local minima (‘cycle skipping’), hence efficiently solvable via Newton-like local optimization methods. The data consist of a single trace extracted from a causal pressure field, propagating in a homogeneous fluid according to linear acoustics, and recorded at a given distance from a transient point energy source. The source intensity (‘wavelet’) is presumed quasi-impulsive, with zero energy for time lags greater than a specified maximum lag. The inverse problem is: from the recorded trace, recover both the sound velocity or slowness and source wavelet with specified support, so that the data is fit with prescribed RMS relative error. The least-squares objective function has multiple large residual minimizers. The extended inverse problem permits source energy to spread in time, and replaces the maximum lag constraint by a weighted quadratic penalty. A companion paper shows that for proper choice of weight operator, any stationary point of the extended objective produces a good approximation of the global minimizer of the least squares objective, with slowness error bounded by a multiple of the maximum lag and the assumed noise level. This paper summarizes the theory developed in the companion paper and presents numerical experiments demonstrating the accuracy of the predictions in concrete instances. We also show how to dynamically adjust the penalty scale during iterative optimization to improve the accuracy of the slowness estimate.

Newton acceleration on manifolds identified by proximal gradient methods

Proximal methods are known to identify the underlying substructure of nonsmooth optimization problems. Even more, in many interesting situations, the output of a proximity operator comes with its structure at no additional cost, and convergence is improved once it matches the structure of a minimizer. However, it is impossible in general to know whether the current structure is final or not; such highly valuable information has to be exploited adaptively. To do so, we place ourselves in the case where a proximal gradient method can identify manifolds of differentiability of the nonsmooth objective. Leveraging this manifold identification, we show that Riemannian Newton-like methods can be intertwined with the proximal gradient steps to drastically boost the convergence. We prove the superlinear convergence of the algorithm when solving some nondegenerated nonsmooth nonconvex optimization problems. We provide numerical illustrations on optimization problems regularized by \(\ell _1\)-norm or trace-norm.

Market Clearing and Krusell-Smith Algorithm in an Economy with Multiple Assets

This paper proposes a novel method to compute the Krusell and Smith (1997, 1998) algorithm, used for solving heterogeneous-agents models with aggregate risk and incomplete markets when households can save in more than one asset. When used to solve a model with more than one asset, the standard algorithm has to impose equilibria for each additional asset (find the market-clearing price) in each period simulated. This procedure entails root-finding in each period, which is computationally expensive. I show that it is possible to avoid root-finding at this level by not imposing equilibria each period, but instead temporarily suspending market clearing. The proposed method then updates the law of motion for asset prices by using the information on the excess demand for assets via a Newton-like method. Since the method avoids the root-finding for each time period simulated, it leads to a significant reduction in computation time. In the example model with two assets, the proposed version of the algorithm leads to an 80% decrease in computational time, even when measured conservatively. This method is potentially useful in computing general equilibrium asset-pricing models with risky and safe assets, featuring both aggregate and uninsurable idiosyncratic risk, since methods that use linearisation in the neighborhood of the aggregate steady state are considered to be less accurate than global solution methods for such models.