Local Quadratic Convergence Research Articles

Accelerated-gradient-based generalized Levenberg–Marquardt method with oracle complexity bound and local quadratic convergence

AbstractMinimizing the sum of a convex function and a composite function appears in various fields. The generalized Levenberg–Marquardt (LM) method, also known as the prox-linear method, has been developed for such optimization problems. The method iteratively solves strongly convex subproblems with a damping term. This study proposes a new generalized LM method for solving the problem with a smooth composite function. The method enjoys three theoretical guarantees: iteration complexity bound, oracle complexity bound, and local convergence under a Hölderian growth condition. The local convergence results include local quadratic convergence under the quadratic growth condition; this is the first to extend the classical result for least-squares problems to a general smooth composite function. In addition, this is the first LM method with both an oracle complexity bound and local quadratic convergence under standard assumptions. These results are achieved by carefully controlling the damping parameter and solving the subproblems by the accelerated proximal gradient method equipped with a particular termination condition. Experimental results show that the proposed method performs well in practice for several instances, including classification with a neural network and nonnegative matrix factorization.

Local quadratic convergence of the SQP method for an optimal control problem governed by a regularized fracture propagation model

We prove local quadratic convergence of the sequential quadratic programming (SQP) method for an optimal control problem of tracking type governed by one time step of the Euler-Lagrange equation of a time discrete regularized fracture or damage energy minimization problem. This lower-level energy minimization problem % describing the fracture process contains a penalization term for violation of the irreversibility condition in the fracture growth process and a viscous regularization term. Conditions on the latter, corresponding to a time step restriction, guarantee strict convexity and thus unique solvability of the Euler Lagrange equations. Nonetheless, these are quasilinear and the control problem is nonconvex. For the convergence proof with $L^\infty$ localization of the SQP-method, we follow the approach from \cite{Troeltzsch:1999}, utilizing strong regularity of generalized equations and arguments from \cite{HoppeNeitzel:2020} for $L^2$-localization. 2020 Mathematics Subject Classification 90C55, 49M41, 49M15.

The solvability of weighted complementarity problems and a smoothing Newton algorithm under the local error bound

The weighted Complementarity Problem (wCP) can be used for modelling a larger class of problems from science and engineering. In this paper, we first extend the solvability of the Complementarity Problems (CP) obtained by Yoshise [SIAM J. Optim. 17, 1129–1153 (2006)] to the wCP. Especially, we prove that the solution set of the wCP is bounded. We then propose a smoothing Newton algorithm to solve the wCP and prove that it is globally convergent. Moreover, we establish the local quadratic convergence of the proposed algorithm under the local error bound condition which is weaker than the nonsingularity condition used in previous smoothing Newton-type algorithms. Further, we apply the proposed algorithm to solve the weighted horizontal linear complementarity problem and report some numerical results.

A Newton’s iteration converges quadratically to nonisolated solutions too

The textbook Newton’s iteration is practically inapplicable on nonisolated solutions of unregularized nonlinear systems. With a simple modification, a version of Newton’s iteration regains its local quadratic convergence to nonisolated zeros of smooth mappings assuming the solutions are semiregular as properly defined regardless of whether the system is square, underdetermined or overdetermined. Furthermore, the iteration serves as a de facto regularization mechanism for computing singular zeros from empirical data. Even if the given system is perturbed so that the nonisolated solution disappears, the iteration still locally converges to a stationary point that approximates a solution of the underlying system with an error bound in the same order of the data accuracy. Geometrically, the iteration approximately converges to the nearest point on the solution manifold. This extension simplifies nonlinear system modeling by eliminating the zero isolation process and enables a wide range of applications in algebraic computation.

A versatile volume-based modeling technique of distributed local quadratic convergence for aeroengines

For advanced aero-engine design and research, modeling and simulation in a digital environment is indispensable, especially for engines of complicated configurations, such as variable cycle engines (VCE) and adaptive cycle engines (ACE). Also, in the research of future smart engines, reliable real-time digital twins are paramount. However, the 2 dominant methods that used in solving the simulation models, Newton-Raphson (N-R) method and volume-based method, are not fully qualified for the study requirements, because neither of them reaches the satisfactory balance of convergence rate and calculating efficiency. In this study, by deeply analyzing the mathematical principle of these 2 methods, a novel modeling and solving method for aero-engine simulation, which integrates the advantages of both N-R and volume-based methods, is established. It has distributed architecture and local quadratic convergence rate. And a novel modeling method for variable area bypass injectors (VABI) is put forward. These facilitate simulation of various configurations of aero-engines. The modeling cases, including a high bypass-ratio (BPR) turbofan and an ACE, illustrate that the novel technique decreases the iterations by about two-thirds comparing with volume-based method, while the success rate of convergence remains over 99%. This proves its superiority in both convergence and calculating efficiency over the conventional ones. This technique can be used in advanced gas turbine engine design and control strategy optimization, and study of digital twins.

Geometry and convergence of natural policy gradient methods

We study the convergence of several natural policy gradient (NPG) methods in infinite-horizon discounted Markov decision processes with regular policy parametrizations. For a variety of NPGs and reward functions we show that the trajectories in state-action space are solutions of gradient flows with respect to Hessian geometries, based on which we obtain global convergence guarantees and convergence rates. In particular, we show linear convergence for unregularized and regularized NPG flows with the metrics proposed by Kakade and Morimura and co-authors by observing that these arise from the Hessian geometries of conditional entropy and entropy respectively. Further, we obtain sublinear convergence rates for Hessian geometries arising from other convex functions like log-barriers. Finally, we interpret the discrete-time NPG methods with regularized rewards as inexact Newton methods if the NPG is defined with respect to the Hessian geometry of the regularizer. This yields local quadratic convergence rates of these methods for step size equal to the inverse penalization strength.

Fast iterative method for local steric Poisson–Boltzmann theories in biomolecular solvation

This work proposes a fast iterative method for local steric Poisson–Boltzmann (PB) theories, in which the electrostatic potential is governed by the Poisson's equation and ionic concentrations satisfy equilibrium conditions. To present the method, we focus on a local steric PB theory derived from a lattice-gas model, as an example. The advantages of the proposed method in efficiency are achieved by treating ionic concentrations as scalar implicit functions of the electrostatic potential, though such functions are only numerically achievable. The existence, uniqueness, boundness, and smoothness of such functions are rigorously established. A Newton iteration method with truncation is proposed to solve a nonlinear system discretized from the generalized PB equations. The existence and uniqueness of the solution to the discretized nonlinear system are established by showing that it is a unique minimizer of a constructed convex energy. Thanks to the boundness of ionic concentrations, truncation bounds for the potential are obtained by using the extremum principle. The truncation step in iterations is shown to be energy and error decreasing. To further speed-up computations, we propose a novel precomputing-interpolation strategy, which is applicable to other local steric PB theories and makes the proposed methods for solving steric PB theories as efficient as for solving the classical PB theory. Analysis on the Newton iteration method with truncation shows local quadratic convergence for the proposed numerical methods. Applications to realistic biomolecular solvation systems reveal that counterions with steric hindrance stratify in an order prescribed by the parameter of ionic valence-to-volume ratio. Finally, we remark that the proposed iterative methods for local steric PB theories can be readily incorporated in well-known classical PB solvers.

Majorization-minimization-based Levenberg–Marquardt method for constrained nonlinear least squares

A new Levenberg–Marquardt (LM) method for solving nonlinear least squares problems with convex constraints is described. Various versions of the LM method have been proposed, their main differences being in the choice of a damping parameter. In this paper, we propose a new rule for updating the parameter so as to achieve both global and local convergence even under the presence of a convex constraint set. The key to our results is a new perspective of the LM method from majorization-minimization methods. Specifically, we show that if the damping parameter is set in a specific way, the objective function of the standard subproblem in LM methods becomes an upper bound on the original objective function under certain standard assumptions. Our method solves a sequence of the subproblems approximately using an (accelerated) projected gradient method. It finds an varepsilon-stationary point after O(varepsilon ^{-2}) computation and achieves local quadratic convergence for zero-residual problems under a local error bound condition. Numerical results on compressed sensing and matrix factorization show that our method converges faster in many cases than existing methods.

Newton Method for Stochastic Control Problems

We develop a new iterative method based on Pontryagin principle to solve stochastic control problems. This method is nothing else than the Newton method extended to the framework of stochastic controls, where the state dynamics is given by an ODE with stochastic coefficients. Each iteration of the method is made of two ingredients: computing the Newton direction, and finding an adapted step length. The Newton direction is obtained by solving an affine-linear Forward-Backward Stochastic Differential Equation (FBSDE) with random coefficients. This is done in the setting of a general filtration. We prove that solving such an FBSDE reduces to solving a Riccati Backward Stochastic Differential Equation (BSDE) and an affine-linear BSDE, as expected in the framework of linear FBSDEs or Linear-Quadratic stochastic control problems. We then establish convergence results for this Newton method. In particular, sufficient regularity of the second-order derivative of the cost functional is required to obtain (local) quadratic convergence. A restriction to the space of essentially bounded stochastic processes is needed to obtain such regularity. To choose an appropriate step length while fitting our choice of space of processes, an adapted backtracking line-search method is developed. We then prove global convergence of the Newton method with the proposed line-search procedure, which occurs at a quadratic rate after finitely many iterations. An implementation with regression techniques to solve BSDEs arising in the computation of the Newton step is developed. We apply it to the control problem of a large number of batteries providing ancillary services to an electricity network.

A new modified Newton iteration for computing nonnegative [formula omitted]-eigenpairs of nonnegative tensors

We propose a new modification of Newton iteration for finding some nonnegative Z-eigenpairs of a nonnegative tensor. The method has local quadratic convergence to a nonnegative eigenpair of a nonnegative tensor, under the usual assumption guaranteeing the local quadratic convergence of the original Newton iteration.

Iterative refinement of Schur decompositions

The Schur decomposition of a square matrix A is an important intermediate step of state-of-the-art numerical algorithms for addressing eigenvalue problems, matrix functions, and matrix equations. This work is concerned with the following task: Compute a (more) accurate Schur decomposition of A from a given approximate Schur decomposition. This task arises, for example, in the context of parameter-dependent eigenvalue problems and mixed precision computations. We have developed a Newton-like algorithm that requires the solution of a triangular matrix equation and an approximate orthogonalization step in every iteration. We prove local quadratic convergence for matrices with mutually distinct eigenvalues and observe fast convergence in practice. In a mixed low-high precision environment, our algorithm essentially reduces to only four high-precision matrix-matrix multiplications per iteration. When refining double to quadruple precision, it often needs only 3–4 iterations, which reduces the time of computing a quadruple precision Schur decomposition by up to a factor of 10–20.

Sequential Quadratic Optimization for Nonlinear Optimization Problems on Riemannian Manifolds

We consider optimization problems on Riemannian manifolds with equality and inequality constraints, which we call Riemannian nonlinear optimization (RNLO) problems. Although they have numerous applications, the existing studies on them are limited especially in terms of algorithms. In this paper, we propose Riemannian sequential quadratic optimization (RSQO) that uses a line-search technique with an $\ell_{1}$ penalty function as an extension of the standard SQO algorithm for constrained nonlinear optimization problems in Euclidean spaces to Riemannian manifolds. We prove its global convergence to a Karush--Kuhn--Tucker point of the RNLO problem by means of parallel transport and the exponential mapping. Furthermore, we establish its local quadratic convergence by analyzing the relationship between sequences generated by RSQO and the Riemannian Newton method. Ours is the first algorithm that has both global and local convergence properties for constrained nonlinear optimization on Riemannian manifolds. Empirical results show that RSQO finds solutions more stably and with higher accuracy compared with the existing Riemannian penalty and augmented Lagrangian methods.

New convergence analysis of a class of smoothing Newton-type methods for second-order cone complementarity problem

<abstract><p>In this paper we propose a class of smoothing Newton-type methods for solving the second-order cone complementarity problem (SOCCP). The proposed method design is based on a special regularized Chen-Harker-Kanzow-Smale (CHKS) smoothing function. When the solution set of the SOCCP is nonempty, our method has the following convergence properties: (ⅰ) it generates a bounded iteration sequence; (ⅱ) the value of the merit function converges to zero; (ⅲ) any accumulation point of the generated iteration sequence is a solution of the SOCCP; (ⅳ) it has the local quadratic convergence rate under suitable assumptions. Some numerical results are reported.</p></abstract>

Dynamic Programming Through the Lens of Semismooth Newton-Type Methods

Policy iteration and value iteration are at the core of many (approximate) dynamic programming methods. For Markov Decision Processes with finite state and action spaces, we show that they are instances of semismooth Newton-type methods for solving the Bellman equation. In particular, we prove that policy iteration is equivalent to the exact semismooth Newton method and enjoys a local quadratic convergence rate. This finding is corroborated by extensive numerical evidence in the fields of control and operations research, which confirms that policy iteration generally requires relatively few iterations to achieve convergence even in presence of a large number of admissible policies. We then show that value iteration is an instance of the fixed-point iteration method and develop a novel locally accelerated version of value iteration with global convergence guarantees and negligible extra computational costs.

Riemannian Newton optimization methods for the symmetric tensor approximation problem

The Symmetric Tensor Approximation problem (STA) consists of approximating a symmetric tensor or a homogeneous polynomial by a linear combination of symmetric rank-1 tensors or powers of linear forms of low symmetric rank. We present two new Riemannian Newton-type methods for low rank approximation of symmetric tensor with complex coefficients.The first method uses the parametrization of the set of tensors of rank at most r by weights and unit vectors. Exploiting the properties of the apolar product on homogeneous polynomials combined with efficient tools from complex optimization, we provide an explicit and tractable formulation of the Riemannian gradient and Hessian, leading to Newton iterations with local quadratic convergence. We prove that under some regularity conditions on non-defective tensors in the neighborhood of the initial point, the Newton iteration (completed with a trust-region scheme) is converging to a local minimum.The second method is a Riemannian Gauss–Newton method on the Cartesian product of Veronese manifolds. An explicit orthonormal basis of the tangent space of this Riemannian manifold is described. We deduce the Riemannian gradient and the Gauss–Newton approximation of the Riemannian Hessian. We present a new retraction operator on the Veronese manifold.We analyze the numerical behavior of these methods, with an initial point provided by Simultaneous Matrix Diagonalisation (SMD). Numerical experiments show the good numerical behavior of the two methods in different cases and in comparison with existing state-of-the-art methods.

A cosh-based smoothing Newton algorithm for the real-time pricing problem in smart grid

In a smart grid system, different types of electricity users such as residential, commercial and industrial users have different utility functions. This paper proposes a nonlinear constrained optimization model for real-time pricing that maximizes the total utilities of all the different user groups. A Karush–Kuhn–Tucker equation system is employed to solve the model. However, it is a challenging task to solve the system due to the nonlinear complementary condition. To tackle the challenge, a cosh-based smoothing approximation function is proposed to substitute the nonlinear complementary condition. Subsequently, the smoothing Newton algorithm is developed to solve the new equation system. The global convergence and local quadratic convergence of the algorithm are proved. Numerical experiments are performed to test the smoothing method and compare the solutions of real-time pricing, fixed pricing and time-of-use pricing strategies applied in the smart grid system. The results show that the real-time pricing mechanism is the most suitable in saving energy and reducing peaks and troughs in energy consumption. This also indicates that it is effective to use the smoothing Newton algorithm to solve the problem of real-time electricity pricing for smart grid. We obtain that the smoothing approximation function is effective for the supply–demand balance constraints in smart grid.

A modified damped Gauss–Newton method for non-monotone weighted linear complementarity problems

We consider the weighted linear complementarity problem (denoted by wLCP). Many numerical algorithms have been proposed for solving the monotone wLCP. In this paper, we propose a damped Gauss–Newton method to solve the non-monotone wLCP which is designed based on a derivative-free non-monotone line search. We show that the proposed method is well defined and it is globally convergent without any problem assumptions. Moreover, we analyze the local quadratic convergence of the proposed method under the non-singularity condition and the local error bound condition, respectively. Our method not only has encouraging local convergence properties but also can be used to solve non-monotone wLCPs. Preliminary numerical results are reported.

A Nonmonotone Smoothing Newton Algorithm for Weighted Complementarity Problem

The weighted complementarity problem (denoted by WCP) significantly extends the general complementarity problem and can be used for modeling a larger class of problems from science and engineering. In this paper, by introducing a one-parametric class of smoothing functions which includes the weight vector, we propose a smoothing Newton algorithm with nonmonotone line search to solve WCP. We show that any accumulation point of the iterates generated by this algorithm, if exists, is a solution of the considered WCP. Moreover, when the solution set of WCP is nonempty, under assumptions weaker than the Jacobian nonsingularity assumption, we prove that the iteration sequence generated by our algorithm is bounded and converges to one solution of WCP with local superlinear or quadratic convergence rate. Promising numerical results are also reported.

A Nonmonotone Trust Region Method for Unconstrained Optimization Problems on Riemannian Manifolds

We propose a nonmonotone trust region method for unconstrained optimization problems on Riemannian manifolds. Global convergence to the first-order stationary points is proved under some reasonable conditions. We also establish local R-linear, super-linear and quadratic convergence rates. Preliminary experiments show that the algorithm is efficient.

Morley finite element methods for the stationary quasi-geostrophic equation

In this paper, we propose and analyze a nonconforming Morley finite element method for the stationary quasi-geostrophic equation in the ocean circulation. Stability and the inf–sup condition for the discrete solution are proved, and the local existence of a unique solution to the discrete nonlinear system is established based on the assumption of the existence of an isolated solution to the linearized problem and Banach fixed point argument. One principal tool employed is to exploit the enrichment operator from nonconforming space to H02(Ω). Thereby, not only smallness assumption on data is avoided, but also optimal error estimates in H2- and H1-norms are proved under minimal regularity condition on the exact solution. Then, for the nonlinear discrete system, the Newton method is applied, which is shown to preserve local quadratic convergence. Moreover, a posteriori error estimator for an adaptive algorithm is derived. Finally, several numerical experiments with a benchmark problem are considered to confirm our theoretical findings.