Convergence Rate Analysis Research Articles

Comparison of the full-discretization methods for milling stability analysis by using different high-order polynomials to interpolate both state term and delayed term

This paper compares different full-discretization methods for milling stability analysis by using different high-order polynomials to interpolate both state term and delayed term (HFDMs) from the aspects of accuracy and efficiency. The dynamic model of milling process with consideration of regeneration effect is described by time periodic delay-differential equation (DDE) in the state-space. Different high-order interpolation polynomials are used to approximate the state term and delayed term. The state transition matrix is obtained on the basis of direct integration scheme. The rates of convergence of different HFDMs are compared with those of the benchmark methods using different process parameter points, the results indicate that it is difficult to evaluate the accuracy of different HFDMs through the convergence rate analysis of limited process parameter points. Then, mean differences and variances between the referenced and predicted critical depths of cut are employed for accuracy analysis. The 3rd-2nd HFDM is, on the whole, proved to be more accurate than the other methods. The efficiencies of different HFDMs are also verified through time-consuming study for both single degree of freedom (1-DOF) and two degree of freedom (2-DOF) milling system. The 3rd-2nd HFDM is proved to be an efficient method by comparing with the other methods. Besides, the HFDMs are available for predicting the stability lobe diagrams under both large immersion condition and low immersion condition.

Convergence Rate of the Modified Landweber Method for Solving Inverse Potential Problems

In this paper, we present the convergence rate analysis of the modified Landweber method under logarithmic source condition for nonlinear ill-posed problems. The regularization parameter is chosen according to the discrepancy principle. The reconstructions of the shape of an unknown domain for an inverse potential problem by using the modified Landweber method are exhibited.

A modified limited memory steepest descent method motivated by an inexact super-linear convergence rate analysis

AbstractHow to choose the step size of gradient descent method has been a popular subject of research. In this paper we propose a modified limited memory steepest descent method (MLMSD). In each iteration we propose a selection rule to pick a unique step size from a candidate set, which is calculated by Fletcher’s limited memory steepest descent method (LMSD), instead of going through all the step sizes in a sweep, as in Fletcher’s original LMSD algorithm. MLMSD is motivated by an inexact super-linear convergence rate analysis. The R-linear convergence of MLMSD is proved for a strictly convex quadratic minimization problem. Numerical tests are presented to show that our algorithm is efficient and robust.

Analysis of biased stochastic gradient descent using sequential semidefinite programs

We present a convergence rate analysis for biased stochastic gradient descent (SGD), where individual gradient updates are corrupted by computation errors. We develop stochastic quadratic constraints to formulate a small linear matrix inequality (LMI) whose feasible points lead to convergence bounds of biased SGD. Based on this LMI condition, we develop a sequential minimization approach to analyze the intricate trade-offs that couple stepsize selection, convergence rate, optimization accuracy, and robustness to gradient inaccuracy. We also provide feasible points for this LMI and obtain theoretical formulas that quantify the convergence properties of biased SGD under various assumptions on the loss functions.

The convergence rate analysis of the symmetric ADMM for the nonconvex separable optimization problems

The symmetric alternating direction method of multipliers is an efficient algorithm, which updates the Lagrange multiplier twice at each iteration and the variables are treated in a symmetric manner. Considering that the convergence range of the parameters plays an important role in the implementation of the algorithm. In this paper, we analyze the convergence rate of the symmetric ADMM with a more relaxed parameter range for solving the two block nonconvex separable optimization problem under the assumption that the generated sequence is bounded. Two cases are considered. In the first case, both components of the objective function are nonconvex, we prove the convergence of the augmented Lagrangian function sequence, and establish the $ O(1/\sqrt{k}) $ worst-case complexity measured by the difference of two consecutive iterations. In the second case, one component of the objective function is convex and the error bound condition is assumed, then we can prove that the iterative sequence converges locally to a KKT point in a R-linear rate; and an auxiliary sequence converges in a Q-linear rate. Furthermore, a practical inexact symmetric ADMM with relative error criteria is proposed, and the associated convergence analysis is established under the same conditions.

The Glowinski–Le Tallec splitting method revisited: A general convergence and convergence rate analysis

In this paper, we focus on a splitting method called the \begin{document}$ \theta $\end{document} -scheme proposed by Glowinski and Le Tallec in [ 17 , 20 , 27 ]. First, we present an elaborative convergence analysis in a Hilbert space and propose a general convergent inexact \begin{document}$ \theta $\end{document} -scheme. Second, for unconstrained problems, we prove the convergence of the \begin{document}$ \theta $\end{document} -scheme and show a sublinear convergence rate in terms of objective value. Furthermore, a practical inexact \begin{document}$ \theta $\end{document} -scheme is derived to solve \begin{document}$ l_2 $\end{document} -loss based problems and its convergence is proved. Third, for constrained problems, even though the convergence of the \begin{document}$ \theta $\end{document} -scheme is available in the literature, yet its sublinear convergence rate is unknown until we provide one via a variational reformulation of the solution set. Besides, in order to relax the condition imposed on the \begin{document}$ \theta $\end{document} -scheme, we propose a new variant and show its convergence. Finally, some preliminary numerical experiments demonstrate the efficiency of the \begin{document}$ \theta $\end{document} -scheme and our proposed methods.

Convergence rate analysis of proximal gradient methods with applications to composite minimization problems

First-order methods such as proximal gradient, which use Forward–Backward Splitting techniques have proved to be very effective in solving nonsmooth convex minimization problem, which is useful in solving various practical problems in different fields such as machine learning and image processing. In this paper, we propose few new forward–backward splitting algorithms, which consume less number of iterations to converge to an optimum. In addition, we derive convergence rates for the proposed formulations and show that the speed of convergence of these algorithms is significantly better than the traditional forward–backward algorithm. To demonstrate the practical applicability, we apply them to two real-world problems of machine learning and image processing. The first issue deals with the regression on high-dimensional datasets, whereas the second one is the image deblurring problem. Numerical experiments have been conducted on several publicly available real datasets to verify the obtained theoretical results. Results demonstrate the superiority of our algorithms in terms of accuracy, the number of iterations required to converge and the rate of convergence against the classical first-order methods.

Convergence Analysis of Inexact Randomized Iterative Methods

In this paper we present a convergence rate analysis of inexact variants of several randomized iterative methods for solving three closely related problems: a convex stochastic quadratic optimizati...

PDE acceleration: a convergence rate analysis and applications to obstacle problems

This paper provides a rigorous convergence rate and complexity analysis for a recently introduced framework, called PDE acceleration, for solving problems in the calculus of variations and explores applications to obstacle problems. PDE acceleration grew out of a variational interpretation of momentum methods, such as Nesterov’s accelerated gradient method and Polyak’s heavy ball method, that views acceleration methods as equations of motion for a generalized Lagrangian action. Its application to convex variational problems yields equations of motion in the form of a damped nonlinear wave equation rather than nonlinear diffusion arising from gradient descent. These accelerated PDEs can be efficiently solved with simple explicit finite difference schemes where acceleration is realized by an improvement in the CFL condition from $$\mathrm{d}t\sim \mathrm{d}x^2$$ for diffusion equations to $$\mathrm{d}t\sim \mathrm{d}x$$ for wave equations. In this paper, we prove a linear convergence rate for PDE acceleration for strongly convex problems, provide a complexity analysis of the discrete scheme, and show how to optimally select the damping parameter for linear problems. We then apply PDE acceleration to solve minimal surface obstacle problems, including double obstacles with forcing, and stochastic homogenization problems with obstacles, obtaining state-of-the-art computational results.

Convergence rate analysis of an iterative algorithm for solving the multiple-sets split equality problem

This paper considers an iterative algorithm of solving the multiple-sets split equality problem (MSSEP) whose step size is independent of the norm of the related operators, and investigates its sublinear and linear convergence rate. In particular, we present a notion of bounded Hölder regularity property for the MSSEP, which is a generalization of the well-known concept of bounded linear regularity property, and give several sufficient conditions to ensure it. Then we use this property to conclude the sublinear and linear convergence rate of the algorithm. In the end, some numerical experiments are provided to verify the validity of our consequences.

Harmonized salp chain-built optimization

As an optimization paradigm, Salp Swarm Algorithm (SSA) outperforms various population-based optimizers in the perspective of the accuracy of obtained solutions and convergence rate. However, SSA gets stuck into sub-optimal solutions and degrades accuracy while solving the complex optimization problems. To relieve these shortcomings, a modified version of the SSA is proposed in the present work, which tries to establish a more stable equilibrium between the exploration and exploitation cores. This method utilizes two different strategies called opposition-based learning and levy-flight (LVF) search. The algorithm is named m-SSA, and its validation is performed on a well-known set of 23 classical benchmark problems. To observe the strength of the proposed method on the scalability of the test problems, the dimension of these problems is varied from 50 to 1000. Furthermore, the proposed m-SSA is also used to solve some real engineering optimization problems. The analysis of results through various statistical measures, convergence rate, and statistical analysis ensures the effectiveness of the proposed strategies integrated with the m-SSA. The comparison of the m-SSA with the conventional SSA, variants of SSA and some other state-of-the-art algorithms illustrate its enhanced search efficiency.

Fractional differential equation approach for convex optimization with convergence rate analysis

A fractional differential equation (FDE) based algorithm for convex optimization is presented in this paper, which generalizes ordinary differential equation (ODE) based algorithm by providing an additional tunable parameter $$\alpha \in (0,1]$$. The convergence of the algorithm is analyzed. For the strongly convex case, the algorithm achieves at least the Mittag-Leffler convergence, while for the general case, the algorithm achieves at least an $$O(1/t^\alpha )$$ convergence rate. Numerical simulations show that the FDE based algorithm may have faster or slower convergence speed than the ODE based one, depending on specific problems.

On the Nonergodic Convergence Rate of an Inexact Augmented Lagrangian Framework for Composite Convex Programming

In this paper, we consider the linearly constrained composite convex optimization problem, whose objective is a sum of a smooth function and a possibly nonsmooth function. We propose an inexact augmented Lagrangian (IAL) framework for solving the problem. The stopping criterion used in solving the augmented Lagrangian subproblem in the proposed IAL framework is weaker and potentially much easier to check than the one used in most of the existing IAL frameworks/methods. We analyze the global convergence and the nonergodic convergence rate of the proposed IAL framework. Preliminary numerical results are presented to show the efficiency of the proposed IAL framework and the importance of the nonergodic convergence and convergence rate analysis.

A conforming primal–dual mixed formulation for the 2D multiscale porous media flow problem

In this paper, a new primal–dual mixed finite-element method is introduced, aimed to model multiscale problems with several geometric subregions in the domain of interest. In each of these regions, porous media fluid flow takes place, but governed by physical parameters at a different scale; in addition, a fluid exchange through contact interfaces occurs between neighboring regions. The well-posedness of the primal–dual mixed finite-element formulation on bounded simply connected polygonal domains of the plane is presented. Next, the convergence of the discrete solution to the exact solution of the problem is discussed, together with the convergence rate analysis. Finally, the numerical examples illustrate the method’s capabilities to handle multiscale problems and interface discontinuities as well as experimental rates of convergence.

Iterative Reweighted Minimization for Generalized Norm/Quasi-Norm Difference Regularized Unconstrained Nonlinear Programming

In this paper, we study the minimization of l p-q (0 <; p ≤ 1, q ≥ 1, p=6q), the general difference of l p and l q norms/quasi-norms, as a nonconvex metric for solving unconstrained nonlinear programming. We first establish an exact (stable) sparse recovery condition for the l p-q constrained problem under an extended restricted p-isometry property and then propose an iterative algorithm for the l p-q regularized unconstrained minimization based on the t-variant of the iterative reweighted minimization method (t ≥ 1) and e-approximation. We theoretically prove that the proposed algorithm converges to a stationary point satisfying the first-order optimality condition. In particular, we provide a convergence rate analysis of the method and show that the local convergence is superlinear under a certain condition. Our extensive experimental results demonstrate that if the sensing matrix satisfies the restricted p-isometry property, the proposed iterative reweighted minimization method for the l p-q unconstrained problem generally outperforms the existing methods (especially for those methods based on the difference of norms). For the ill-conditioned sensing matrix, a variant of our method via the difference of convex functions algorithm (DCA) shows better performance on the frequency of success for signal sparse recovery. Likewise, our methods are illustrated to be valid and generally outperform the existing methods for real images.

Proximally Guided Stochastic Subgradient Method for Nonsmooth, Nonconvex Problems

In this paper, we introduce a stochastic projected subgradient method for weakly convex (i.e., uniformly prox-regular) nonsmooth, nonconvex functions---a wide class of functions which includes the additive and convex composite classes. At a high level, the method is an inexact proximal-point iteration in which the strongly convex proximal subproblems are quickly solved with a specialized stochastic projected subgradient method. The primary contribution of this paper is a simple proof that the proposed algorithm converges at the same rate as the stochastic gradient method for smooth nonconvex problems. This result appears to be the first convergence rate analysis of a stochastic (or even deterministic) subgradient method for the class of weakly convex functions. In addition, a two-phase variant is proposed that significantly reduces the variance of the solutions returned by the algorithm. Finally, preliminary numerical experiments are also provided.

Riemannian Stochastic Variance Reduced Gradient Algorithm with Retraction and Vector Transport

In recent years, stochastic variance reduction algorithms have attracted considerable attention for minimizing the average of a large but finite number of loss functions. This paper proposes a novel Riemannian extension of the Euclidean stochastic variance reduced gradient (R-SVRG) algorithm to a manifold search space. The key challenges of averaging, adding, and subtracting multiple gradients are addressed with retraction and vector transport. For the proposed algorithm, we present a global convergence analysis with a decaying step size as well as a local convergence rate analysis with a fixed step size under some natural assumptions. In addition, the proposed algorithm is applied to the computation problem of the Riemannian centroid on the symmetric positive definite (SPD) manifold as well as the principal component analysis and low-rank matrix completion problems on the Grassmann manifold. The results show that the proposed algorithm outperforms the standard Riemannian stochastic gradient descent algorithm in each case.

Variational source condition for ill-posed backward nonlinear Maxwell’s equations

This paper examines the mathematical analysis of an electromagnetic inverse problem governed by nonlinear evolutionary Maxwell’s equations. The aim of the inverse problem is to recover electromagnetic fields at the past time by noisy measurement data at the present time. We consider the Tikhonov regularization method to cope with the ill-posedness of the governing backward nonlinear Maxwell’s equations. By means of the semigroup theory, we study its convergence analysis and derive optimality conditions through a rigorous first-order analysis and adjoint calculus. The final part of the paper is focused on the convergence rate analysis of the Tikhonov regularization method under a variational source condition (VSC), which leads to power-type convergence rates. Employing the spectral theory, the complex interpolation theory and fractional Sobolev spaces, we validate the proposed VSC on account of an appropriate regularity assumption on the exact initial data and the material parameters.

Time-varying low gain feedback for linear systems with unknown input delay

In this paper, the traditional low gain feedback design with a time-invariant feedback parameter is generalized to time-varying parameter design for linear systems with delayed input. For an unknown delay with a known upper bound, a time-varying low gain feedback law, constructed by using the parametric Lyapunov equation based approach, globally regulates a system with all open loop poles at the origin as long as the time-varying low gain parameter has a continuous second derivative and approaches a sufficiently small constant with its derivative approaching zero as time goes to infinity. Improvement of the closed-loop performance is addressed in a convergence rate analysis and then observed in simulation compared with the traditional constant parameter low gain design.

Convergence rate analysis for the higher order power method in best rank one approximations of tensors

A popular and classical method for finding the best rank one approximation of a real tensor is the higher order power method (HOPM). It is known in the literature that the iterative sequence genera...