Local Superlinear Convergence Research Articles

Convergence and stability analysis of iteratively reweighted least squares for noisy block sparse recovery

This paper considers the theoretical properties of iteratively reweighted least squares algorithm for noisy block sparse recovery problem (BIRLS for short). Li et al. used numerical experiments to show the remarkable performance of BIRLS algorithm for recovering a block sparse signal in noiseless measurement case, but no convergence analysis was given. In this paper, we focus on providing convergence, convergence rate and stability analysis of BIRLS algorithm for block sparse recovery in the presence of noise. The convergence of BIRLS is proved strictly. Furthermore, when the linear measurement matrix A satisfies the block restricted isometry property (abbreviated as block RIP), we show that BIRLS algorithm is stable and give the error analysis of BIRLS algorithm. We also characterize the convergence rate of the BIRLS algorithm, which implies global linear convergence for p=1 and local super-linear convergence for 0<p<1. The simplicity of BIRLS algorithm, along with the theoretical guarantees provided in this paper, make a compelling case for its adoption as a standard tool for block sparse recovery.

Generalized Continuation Newton Methods and the Trust-Region Updating Strategy for the Underdetermined System

This paper considers the generalized continuation Newton method and the trust-region updating strategy for the underdetermined system of nonlinear equations. Moreover, in order to improve its computational efficiency, the new method will not update the Jacobian matrix when the current Jacobian matrix performs well. The numerical results show that the new method is more robust and faster than the traditional optimization method such as the Levenberg–Marquardt method (a variant of trust-region methods, the built-in subroutine fsolve.m of the MATLAB R2020a environment). The computational time of the new method is about 1/8 to 1/50 of that of fsolve. Furthermore, it also proves the global convergence and the local superlinear convergence of the new method under some standard assumptions.

Continuation Newton methods with the residual trust-region time-stepping scheme for nonlinear equations

For nonlinear equations, the homotopy methods (continuation methods) are popular in engineering fields since their convergence regions are large and they are quite reliable to find a solution. The disadvantage of the classical homotopy methods is that their computational time is heavy since they need to solve many auxiliary nonlinear systems during the intermediate continuation processes. In order to overcome this shortcoming, we consider the special explicit continuation Newton method with the residual trust-region time-stepping scheme for this problem. According to our numerical experiments, the new method is more robust and faster than the traditional optimization method (the built-in subroutine fsolve.m of the MATLAB environment) and the homotopy continuation methods (HOMPACK90 and NAClab). Furthermore, we analyze the global convergence and the local superlinear convergence of the new method.

A penalty scheme and policy iteration for nonlocal HJB variational inequalities with monotone nonlinearities

We propose a class of numerical schemes for nonlocal HJB variational inequalities (HJBVIs) with monotone nonlinearities arising from mixed optimal stopping and control of processes with infinite activity jumps, where the objective is specified by a monotone recursive preference. The solution and free boundary of the HJBVI are constructed from a sequence of penalized equations, for which the penalization error is estimated. The penalized equation is then discretized by a class of semi-implicit monotone approximations. We present a novel analysis technique for the well-posedness of the discrete equation, and demonstrate the convergence of the scheme, which subsequently gives a constructive proof for the existence of a solution to the penalized equation and variational inequality. We further propose an efficient iterative algorithm with local superlinear convergence for solving the discrete equation. Numerical experiments are presented for an optimal investment problem under ambiguity and a two-dimensional recursive consumption-portfolio allocation problem.

On Solving the Convex Semi-Infinite Minimax Problems via Superlinear 𝒱𝒰 Incremental Bundle Technique with Partial Inexact Oracle

In this paper, we study convex semi-infinite programming involving minimax problems. One of the difficulties in solving these problems is that the maximum type functions are not differentiable. Due to the nonsmooth nature of the problem, we apply the special proximal bundle scheme on the basis of [Formula: see text]-decomposition theory to solve the nonsmooth convex semi-infinite minimax problems. The proposed scheme requires an evaluation within some accuracy for all the components of the objective function. Regarding the incremental method, we only need one component function value and one subgradient which are estimated to update the bundle information and produce the search direction. Under some mild assumptions, we present global convergence and local superlinear convergence of the proposed bundle method. Numerical results of several example problems are reported to show the effectiveness of the new scheme.

A competitive inexact nonmonotone filter SQP method: convergence analysis and numerical results

We propose an inexact nonmonotone successive quadratic programming (SQP) algorithm for solving nonlinear programming problems with equality constraints and bounded variables. Regarding the value of the current feasibility violation and the minimum value of its linear approximation over a trust region, several scenarios are envisaged. In one scenario, a possible infeasible stationary point is detected. In other scenarios, the search direction is computed using an inexact (truncated) solution of a feasible strictly convex quadratic program (QP). The search direction is shown to be a descent direction for the objective function or the feasibility violation in the feasible or infeasible iterations, respectively. A new penalty parameter updating formula is proposed to turn the search direction into a descent direction for an -penalty function. In certain iterations, an accelerator direction is developed to obtain a superlinear local convergence rate of the algorithm. Using a nonmonotone filter strategy, the global convergence of the algorithm and a superlinear local rate of convergence are guaranteed. The main advantage of the algorithm is that the global convergence of the algorithm is established using inexact solutions of the QPs. Furthermore, the use of inexact solutions instead of exact solutions of the subproblems enhances the robustness and efficiency of the algorithm. The algorithm is implemented using MATLAB and the program is tested on a wide range of test problems from the CUTEst library. Comparison of the obtained numerical results with those obtained by testing some similar SQP algorithms affirms the efficiency and robustness of the proposed algorithm.

A Riemannian optimization approach to the radial distribution network load flow problem

The Load Flow (LF) equations in power networks are the foundation of several applications on active and reactive power flow control, distributed and real-time control and optimization. In this paper, we formulate the LF problem in radial electricity distribution networks as an unconstrained Riemannian optimization problem, consisting of two manifolds, and we consider alternative retractions and initialization options. We introduce a Riemannian approximate Newton method tailored to the LF problem, as an exact solution method guaranteeing monotonic descent and local superlinear convergence rate. Extensive numerical comparisons on several test networks illustrate that the proposed method outperforms standard Riemannian optimization methods (Gradient Descent, Newton’s), and achieves comparable performance with the traditional Newton–Raphson method (in Euclidean coordinates), albeit besting it by a guarantee to convergence. We also consider an approximate LF solution obtained by the first iteration of the proposed method, and we show that it significantly outperforms other approximants in the LF literature. Lastly, we derive an interesting analogy with the well-known Backward–Forward Sweep (BFS) method showing that BFS iterations move on the power flow manifold, and highlighting the advantage of our method in converging under high constant impedance loading conditions, whereas BFS may diverge.

A Nonmonotone Smoothing Newton Algorithm for Weighted Complementarity Problem

The weighted complementarity problem (denoted by WCP) significantly extends the general complementarity problem and can be used for modeling a larger class of problems from science and engineering. In this paper, by introducing a one-parametric class of smoothing functions which includes the weight vector, we propose a smoothing Newton algorithm with nonmonotone line search to solve WCP. We show that any accumulation point of the iterates generated by this algorithm, if exists, is a solution of the considered WCP. Moreover, when the solution set of WCP is nonempty, under assumptions weaker than the Jacobian nonsingularity assumption, we prove that the iteration sequence generated by our algorithm is bounded and converges to one solution of WCP with local superlinear or quadratic convergence rate. Promising numerical results are also reported.

A trust‐region method for the parameterized generalized eigenvalue problem with nonsquare matrix pencils

AbstractThe l parameterized generalized eigenvalue problems for the nonsquare matrix pencils, proposed by Chu and Golub [SIAM J. Matrix Anal. Appl., 28(2006), pp. 770‐787], can be formulated as an optimization problem on a corresponding complex product Stiefel manifold. Some early proposed algorithms are based on the first‐order information of the objective function, and fast convergence could not be expected. In this article, we turn the generic Riemannian trust‐region method of Absil et al. into a practical algorithm for solving the underlying problem, which enjoys the global convergence and local superlinear convergence rate. Numerical experiments are provided to illustrate the efficiency of the proposed method. Detailed comparisons with some existing results (l = 1 and l = n) are given first. For the case l = n, the algorithm can yield exactly the same optimal solution as Ito and Murota's algorithm, which is essentially a direct method to solve the problem. The algorithm runs faster than Boutry et al.'s algorithm for the case l = 1. Further comparisons with some early proposed gradient‐based algorithms, and some latest infeasible methods for solving manifold optimization problems are also provided to show the merits of the proposed approach.

Limited-memory BFGS with displacement aggregation

A displacement aggregation strategy is proposed for the curvature pairs stored in a limited-memory BFGS (a.k.a. L-BFGS) method such that the resulting (inverse) Hessian approximations are equal to those that would be derived from a full-memory BFGS method. This means that, if a sufficiently large number of pairs are stored, then an optimization algorithm employing the limited-memory method can achieve the same theoretical convergence properties as when full-memory (inverse) Hessian approximations are stored and employed, such as a local superlinear rate of convergence under assumptions that are common for attaining such guarantees. To the best of our knowledge, this is the first work in which a local superlinear convergence rate guarantee is offered by a quasi-Newton scheme that does not either store all curvature pairs throughout the entire run of the optimization algorithm or store an explicit (inverse) Hessian approximation. Numerical results are presented to show that displacement aggregation within an adaptive L-BFGS scheme can lead to better performance than standard L-BFGS.

New Results on Superlinear Convergence of Classical Quasi-Newton Methods

We present a new theoretical analysis of local superlinear convergence of classical quasi-Newton methods from the convex Broyden class. As a result, we obtain a significant improvement in the currently known estimates of the convergence rates for these methods. In particular, we show that the corresponding rate of the Broyden–Fletcher–Goldfarb–Shanno method depends only on the product of the dimensionality of the problem and the logarithm of its condition number.

A Nonmonotone Trust Region Method for Unconstrained Optimization Problems on Riemannian Manifolds

We propose a nonmonotone trust region method for unconstrained optimization problems on Riemannian manifolds. Global convergence to the first-order stationary points is proved under some reasonable conditions. We also establish local R-linear, super-linear and quadratic convergence rates. Preliminary experiments show that the algorithm is efficient.

Local convergence of tensor methods

In this paper, we study local convergence of high-order Tensor Methods for solving convex optimization problems with composite objective. We justify local superlinear convergence under the assumption of uniform convexity of the smooth component, having Lipschitz-continuous high-order derivative. The convergence both in function value and in the norm of minimal subgradient is established. Global complexity bounds for the Composite Tensor Method in convex and uniformly convex cases are also discussed. Lastly, we show how local convergence of the methods can be globalized using the inexact proximal iterations.

Greedy Quasi-Newton Methods with Explicit Superlinear Convergence

In this paper, we study greedy variants of quasi-Newton methods. They are based on the updating formulas from a certain subclass of the Broyden family. In particular, this subclass includes the well-known DFP, BFGS, and SR1 updates. However, in contrast to the classical quasi-Newton methods, which use the difference of successive iterates for updating the Hessian approximations, our methods apply basis vectors, greedily selected so as to maximize a certain measure of progress. For greedy quasi-Newton methods, we establish an explicit nonasymptotic bound on their rate of local superlinear convergence, as applied to minimizing strongly convex and strongly self-concordant functions (and, in particular, to strongly convex functions with Lipschitz continuous Hessian). The established superlinear convergence rate contains a contraction factor, which depends on the square of the iteration counter. We also show that greedy quasi-Newton methods produce Hessian approximations whose deviation from the exact Hessians linearly converges to zero.

On a Semismooth* Newton Method for Solving Generalized Equations

In the paper, a Newton-type method for the solution of generalized equations (GEs) is derived, where the linearization concerns both the single-valued and the multivalued part of the considered GE. The method is based on the new notion of semismoothness${}^*$, which, together with a suitable regularity condition, ensures the local superlinear convergence. An implementable version of the new method is derived for a class of GEs, frequently arising in optimization and equilibrium models.

An Interior-Point Approach for Solving Risk-Averse PDE-Constrained Optimization Problems with Coherent Risk Measures

The prevalence of uncertainty in models of engineering and the natural sciences necessitates the inclusion of random parameters in the underlying partial differential equations (PDEs). The resulting decision problems governed by the solution of such random PDEs are infinite dimensional stochastic optimization problems. In order to obtain risk-averse optimal decisions in the face of such uncertainty, it is common to employ risk measures in the objective function. This leads to risk-averse PDE-constrained optimization problems. We propose a method for solving such problems in which the risk measures are convex combinations of the mean and conditional value-at-risk (CVaR). Since these risk measures can be evaluated by solving a related inequality-constrained optimization problem, we suggest a log-barrier technique to approximate the risk measure. This leads to a new continuously differentiable convex risk measure: the log-barrier risk measure. We show that the log-barrier risk measure fits into the setting of optimized certainty equivalents of Ben-Tal and Teboulle and the expectation quadrangle of Rockafellar and Uryasev. Using the differentiability of the log-barrier risk measure, we derive first-order optimality conditions reminiscent of classical primal and primal-dual interior-point approaches in nonlinear programming. We derive the associated Newton system, propose a reduced symmetric system to calculate the steps, and provide a sufficient condition for local superlinear convergence in the continuous setting. Furthermore, we provide a $\Gamma$-convergence result for the log-barrier risk measures to prove convergence of the minimizers to the original nonsmooth problem. The results are illustrated by a numerical study.

A smoothing quasi-Newton method for solving general second-order cone complementarity problems

Recently, there are much interests in studying smoothing Newton method for solving montone second-order cone complementarity problem (SOCCP) or SOCCPs with Cartesian $$P/P_0$$ -property. In this paper, we propose a smoothing quasi-Newon method for solving general SOCCP. We show that the proposed method is well-defined without any additional assumption and has global convergence under standard conditions. Moreover, under the Jacobian nonsingularity assumption, the method is shown to have local superlinear or quadratic convergence rate. Our preliminary numerical experiments show the method could be very effective for solving SOCCPs.

Performance seeking control of minimum infrared characteristic on double bypass variable cycle engine

In order to study the performance seeking control of double bypass variable cycle engine, a Feasible Sequential Quadratic Programming (FSQP) algorithm with global and local superlinear convergence characteristics is adopted. The engine model established a backward infrared radiation intensity prediction method for exhaust system, which could calculate the infrared radiation intensity of the high temperature components and nozzle stream in the system. The infrared radiation absorption effect of the nozzle stream on the high temperature components is considered in the method. According to the above model, a minimum infrared characteristic performance seeking control based on infrared prediction model is proposed. The optimal control simulations of the maximum thrust, minimum specific fuel consumption and minimum infrared characteristic mode on the double bypass variable cycle engine are carried out. Finally, the simulation results show that the algorithm could obtain the optimal operating conditions of double bypass variable cycle engine in different working conditions, which achieves the steady-state performance seeking control of the engine.

Smoothing inexact Newton method based on a new derivative-free nonmonotone line search for the NCP over circular cones

In this paper we consider the nonlinear complementarity problem over circular cones (CCNCP) which contains a lot of circular cone optimization problems. We study a one-parametric class of smoothing functions which can be used to reformulate the CCNCP as a system of smooth nonlinear equations. Based on the equivalent reformulation, we propose a smoothing inexact Newton method to solve the CCNCP. In each iteration, the proposed method solves the nonlinear equations only approximately. Since the inexact direction is not necessarily descent, a new derivative-free nonmonotone line search is developed to ensure that the proposed method has global and local superlinear and quadratical convergence. Some numerical results are also reported.

A hybrid semismooth quasi-Newton method for nonsmooth optimal control with PDEs

We propose a semismooth Newton-type method for nonsmooth optimal control problems. Its particular feature is the combination of a quasi-Newton method with a semismooth Newton method. This reduces the computational costs in comparison to semismooth Newton methods while maintaining local superlinear convergence. The method applies to Hilbert space problems whose objective is the sum of a smooth function, a regularization term, and a nonsmooth convex function. In the theoretical part of this work we establish the local superlinear convergence of the method in an infinite-dimensional setting and discuss its application to sparse optimal control of the heat equation subject to box constraints. We verify that the assumptions for local superlinear convergence are satisfied in this application and we prove that convergence can take place in stronger norms than that of the Hilbert space if initial error and problem data permit. In the numerical part we provide a thorough study of the hybrid approach on two optimal control problems, including an engineering problem from magnetic resonance imaging that involves bilinear control of the Bloch equations. We use this problem to demonstrate that the new method is capable of solving nonconvex, nonsmooth large-scale real-world problems. Among others, the study addresses mesh independence, globalization techniques, and limited-memory methods. We observe throughout that algorithms based on the hybrid methodology are several times faster in runtime than their semismooth Newton counterparts.