Global Convergence Rate Research Articles

Stochastic analysis of an adaptive cubic regularization method under inexact gradient evaluations and dynamic Hessian accuracy

We here adapt an extended version of the adaptive cubic regularization method with dynamic inexact Hessian information for nonconvex optimization in Bellavia et al. [Adaptive cubic regularization methods with dynamic inexact hessian information and applications to finite-sum minimization. IMA Journal of Numerical Analysis. 2021;41(1):764–799] to the stochastic optimization setting. While exact function evaluations are still considered, this novel variant inherits the innovative use of adaptive accuracy requirements for Hessian approximations introduced in the just quoted paper and additionally employs inexact computations of the gradient. Without restrictions on the variance of the errors, we assume that these approximations are available within a sufficiently large, but fixed, probability and we extend, in the spirit of Cartis and Scheinberg [Global convergence rate analysis of unconstrained optimization methods based on probabilistic models. Math Program Ser A. 2018;159(2):337–375], the deterministic analysis of the framework to its stochastic counterpart, showing that the expected number of iterations to reach a first-order stationary point matches the well-known worst-case optimal complexity. This is, in fact, still given by , with respect to the first-order ϵ tolerance. Finally, numerical tests on nonconvex finite-sum minimization confirm that using inexact first- and second-order derivatives can be beneficial in terms of the computational savings.

A trust‐region method for the parameterized generalized eigenvalue problem with nonsquare matrix pencils

AbstractThe l parameterized generalized eigenvalue problems for the nonsquare matrix pencils, proposed by Chu and Golub [SIAM J. Matrix Anal. Appl., 28(2006), pp. 770‐787], can be formulated as an optimization problem on a corresponding complex product Stiefel manifold. Some early proposed algorithms are based on the first‐order information of the objective function, and fast convergence could not be expected. In this article, we turn the generic Riemannian trust‐region method of Absil et al. into a practical algorithm for solving the underlying problem, which enjoys the global convergence and local superlinear convergence rate. Numerical experiments are provided to illustrate the efficiency of the proposed method. Detailed comparisons with some existing results (l = 1 and l = n) are given first. For the case l = n, the algorithm can yield exactly the same optimal solution as Ito and Murota's algorithm, which is essentially a direct method to solve the problem. The algorithm runs faster than Boutry et al.'s algorithm for the case l = 1. Further comparisons with some early proposed gradient‐based algorithms, and some latest infeasible methods for solving manifold optimization problems are also provided to show the merits of the proposed approach.

An efficient multilevel thresholding based satellite image segmentation approach using a new adaptive cuckoo search algorithm

It is the most challenging and difficult task to segment a satellite image because of its complete randomness, multiple regions of interest, weak correlation with pixels, and regions of ambiguity. There are several Nature-inspired algorithms available, which are used to overcome these difficulties and those are more efficient to generate the best threshold value for the segmentation of satellite images. Though various modern methodologies opt for better results but methods have some drawbacks too like techniques are computationally expensive and time-consuming. In this paper, we have proposed a more effective satellite image segmentation approach using a new adaptive cuckoo search (ACS) algorithm. The result obtained from the projected technique is compared with CSMcCulloch incorporating McCulloch’s method for levy flight generation in Cuckoo Search (CS) algorithm by using two different objective functions namely Otsu’s method and Tsallis entropy function. The measurement techniques such as PSNR, MSE, FSIM, SSIM, UIQI, and computational time in term of CPU running time have been considered for validating and evaluating the proposed method. This proposed algorithm technique has resulted in improve segmentation quality of satellite images and reduced computational time. The analysis of the convergence rate proves that ACS is superior to the CSMcCulloch algorithm for reaching the global convergence rate. These experimental outcomes help to encourage researchers in different domains such as computer vision, application of medical image analysis, machine learning as well as deep learning.

A QCQP-based splitting SQP algorithm for two-block nonconvex constrained optimization problems with application

This paper discusses a class of two-block nonconvex smooth optimization problems with nonlinear constraints. Based on a quadratically constrained quadratic programming (QCQP) approximation, an augmented Lagrangian function (ALF), and a Lagrangian splitting technique into small-scale subproblems, we propose a novel sequential quadratic programming (SQP) algorithm. First, inspired by the augmented Lagrangian method (ALM), we penalize the quadratic equality constraints associated with the QCQP approximation subproblem in its objective by means of the ALF, and then split the resulting subproblem into two small-scale ones, but both of them are not quadratic programming (QP) due to the square of the quadratic equality constraints in the objective. Second, by ignoring the three-order infinitesimal arising from the squared term, the two small-scale subproblems are reduced to two standard QP subproblems, which can yield an improved search direction. Third, taking the ALF of the discussed problem as a merit function, the next iterate point is generated by the Armijo line search. As a result, a new SQP method, called QCQP-based splitting SQP method, is proposed. Under suitable conditions, the global convergence, strong convergence, iteration complexity and convergence rate of the proposed method are analyzed and obtained. Finally, preliminary numerical experiments and applications were carried out, and these show that the proposed method is promising.

Accelerated Graph Learning From Smooth Signals

We consider network topology identification subject to a signal smoothness prior on the nodal observations. A fast dual-based proximal gradient algorithm is developed to efficiently tackle a strongly convex, smoothness-regularized network inverse problem known to yield high-quality graph solutions. Unlike existing solvers, the novel iterations come with global convergence rate guarantees and do not require additional step-size tuning. Reproducible simulated tests demonstrate the effectiveness of the proposed method in accurately recovering random and real-world graphs, markedly faster than state-of-the-art alternatives and without incurring an extra computational burden.

Global Convergence Rate Analysis of a Generic Line Search Algorithm with Noise

In this paper, we develop convergence analysis of a modified line search method for objective functions whose value is computed with noise and whose gradient estimates are inexact and possibly random. The noise is assumed to be bounded in absolute value without any additional assumptions. We extend the framework based on stochastic methods from [C. Cartis and K. Scheinberg, Math. Program., 169 (2018), pp. 337--375] which was developed to provide analysis of a standard line search method with exact function values and random gradients to the case of noisy functions. We introduce two alternative conditions on the gradient which, when satisfied with some sufficiently large probability at each iteration, guarantees convergence properties of the line search method. We derive expected complexity bounds to reach a near optimal neighborhood for convex, strongly convex and nonconvex functions. The exact dependence of the convergence neighborhood on the noise is specified.

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

In this paper, we consider a class of constrained nonsmooth convex optimization problems, which have wide applications in signal processing, imaging recovery, machine learning, etc. Recently, theoretical analysis and numerical experiments show that the extrapolation can improve the convergence rate of the algorithms efficiently, which let the proximal gradient algorithm with extrapolation have significant advantages in solving large scale optimization problems. Motivated by the smoothing techniques, and the fast iterative shrinkage-thresholding algorithm (FISTA) proposed by Beck and Teboulle, we bring forward a new accelerated algorithm for solving the considered constrained nonsmooth convex optimization problems. We prove that any accumulated point of the algorithm is an optimal solution of the problem. In the analysis of the algorithm, we consider several different update methods for the smoothing parameter, and give the global convergence rate $O(\\ln~k/k)$ on the objective function values. Moreover, we analyze the convergence rate of the difference on the iterates $\\lim_{k\\rightarrow}\\|x^{k+1}-x^k\\|=0$. Finally, our numerical experiments illustrate the good performance of the proposed algorithm in solving two classes of sparse optimization problems and the positive influence of the extrapolation on the convergence rate.

A class of smooth exact penalty function methods for optimization problems with orthogonality constraints

ABSTRACT Updating the augmented Lagrangian multiplier by closed-form expression yields efficient first-order infeasible approach for optimization problems with orthogonality constraints. Hence, parallelization becomes tractable in solving this type of problems. Inspired by this closed-form updating scheme, we propose a novel penalty function with compact convex constraints (PenC). We show that PenC can act as an exact penalty model which shares the same global minimizers as the original problem with orthogonality constraints. Based on PenC, we first propose a first-order algorithm called PenCF and establish its global convergence and local linear convergence rate under some mild assumptions. For the case that the computation and storage of Hessian is achievable, and we pursue high precision solution and fast local convergence rate, a second-order approach called PenCS is proposed for solving PenC. To avoid expensive calculation or solving a hard subproblem in computing the Newton step, we propose a new strategy to do it approximately which still leads to quadratic convergence locally. Moreover, the main iterations of both PenCF and PenCS are orthonormalization-free and hence parallelizable. Numerical experiments illustrate that PenCF is comparable with the existing first-order methods. Furthermore, PenCS shows its stability and high efficiency in obtaining high precision solution comparing with the existing second-order methods.

Estimation for functional linear semiparametric model

We study a functional linear semiparametric model which is not only an extension of partially functional linear models, but also an extension of semiparametric models. We consider the case that a response is related to a functional predictor and several scalar variables and the functional predictor is observed at a set of discrete points with noise. We propose a new estimation procedure which combines functional principal component analysis and B-spline methods to estimate unknown parameters and functions in model. The asymptotic distribution of the estimators of slope parameters is derived and the global convergence rate of the estimator of unknown slope function is established. The convergence rate of the mean squared prediction error for a predictor is also established. Simulation studies are conducted to investigate the finite sample performance of the proposed estimators. A real data example based on real estate data is used to illustrate our proposed methodology.

Linearized symmetric multi-block ADMM with indefinite proximal regularization and optimal proximal parameter

The proximal term plays a significant role in the literature of proximal Alternating Direction Method of Multipliers (ADMM), since (positive-definite or indefinite) proximal terms can promote convergence of ADMM and further simplify the involved subproblems. However, an overlarge proximal parameter decelerates the convergence numerically, though the convergence can be established with it. In this paper, we thus focus on a Linearized Symmetric ADMM (LSADMM) with proper proximal terms for solving a family of multi-block separable convex minimization models, and we determine an optimal (smallest) value of the proximal parameter while convergence of this LSADMM can be still ensured. Our LSADMM partitions the data into two group variables and updates the Lagrange multiplier twice in different forms with suitable step sizes. The region of the proximal parameter, involved in the second group subproblems, is partitioned into the union of three different sets. We show the global convergence and sublinear ergodic convergence rate of LSADMM for the two cases, while a counter-example is given to illustrate that convergence of LSADMM can not be guaranteed for the remaining case. Theoretically, we obtain the optimal lower bound of the proximal parameter. Numerical experiments on the so-called latent variable Gaussian graphical model selection problems are presented to demonstrate performance of the proposed algorithm and the significant advantage of the optimal lower bound of the proximal parameter.

On the Convergence Rate of Inexact Majorized sGS ADMM with Indefinite Proximal Terms for Convex Composite Programming

In this paper, we propose an inexact majorized symmetric Gauss–Seidel (sGS) alternating direction method of multipliers (ADMM) with indefinite proximal terms for multi-block convex composite programming. This method is a specific form of the inexact majorized ADMM which is further proposed to solve a general two-block separable optimization problem. The new methods adopt certain relative error criteria to solve the involving subproblems approximately, and the step-sizes allow to choose in the scope [Formula: see text]. Under more general conditions, we establish the global convergence and Q-linear convergence rate of the proposed methods.

Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods

In this paper we study several classes of stochastic optimization algorithms enriched with heavy ball momentum. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic dual subspace ascent. This is the first time momentum variants of several of these methods are studied. We choose to perform our analysis in a setting in which all of the above methods are equivalent: convex quadratic problems. We prove global non-asymptotic linear convergence rates for all methods and various measures of success, including primal function values, primal iterates, and dual function values. We also show that the primal iterates converge at an accelerated linear rate in a somewhat weaker sense. This is the first time a linear rate is shown for the stochastic heavy ball method (i.e., stochastic gradient descent method with momentum). Under somewhat weaker conditions, we establish a sublinear convergence rate for Cesaro averages of primal iterates. Moreover, we propose a novel concept, which we call stochastic momentum, aimed at decreasing the cost of performing the momentum step. We prove linear convergence of several stochastic methods with stochastic momentum, and show that in some sparse data regimes and for sufficiently small momentum parameters, these methods enjoy better overall complexity than methods with deterministic momentum. Finally, we perform extensive numerical testing on artificial and real datasets, including data coming from average consensus problems.

Jump or kink: on super-efficiency in segmented linear regression breakpoint estimation

Summary We consider the problem of segmented linear regression with a single breakpoint, with the focus on estimating the location of the breakpoint. If $n$ is the sample size, we show that the global minimax convergence rate for this problem in terms of the mean absolute error is $O(n^{-1/3})$. On the other hand, we demonstrate the construction of a super-efficient estimator that achieves the pointwise convergence rate of either $O(n^{-1})$ or $O(n^{-1/2})$ for every fixed parameter value, depending on whether the structural change is a jump or a kink. The implications of this example and a potential remedy are discussed.

Convergence rates of Gaussian ODE filters

A recently introduced class of probabilistic (uncertainty-aware) solvers for ordinary differential equations (ODEs) applies Gaussian (Kalman) filtering to initial value problems. These methods model the true solution x and its first q derivatives a priori as a Gauss–Markov process {varvec{X}}, which is then iteratively conditioned on information about {dot{x}}. This article establishes worst-case local convergence rates of order q+1 for a wide range of versions of this Gaussian ODE filter, as well as global convergence rates of order q in the case of q=1 and an integrated Brownian motion prior, and analyses how inaccurate information on {dot{x}} coming from approximate evaluations of f affects these rates. Moreover, we show that, in the globally convergent case, the posterior credible intervals are well calibrated in the sense that they globally contract at the same rate as the truncation error. We illustrate these theoretical results by numerical experiments which might indicate their generalizability to q in {2,3,ldots }.

Convergence rates for an inexact ADMM applied to separable convex optimization

Convergence rates are established for an inexact accelerated alternating direction method of multipliers (I-ADMM) for general separable convex optimization with a linear constraint. Both ergodic and non-ergodic iterates are analyzed. Relative to the iteration number k, the convergence rate is O(1/k) in a convex setting and O(1/k^2) in a strongly convex setting. When an error bound condition holds, the algorithm is 2-step linearly convergent. The I-ADMM is designed so that the accuracy of the inexact iteration preserves the global convergence rates of the exact iteration, leading to better numerical performance in the test problems.

Inertial proximal gradient methods with Bregman regularization for a class of nonconvex optimization problems

This paper proposes an inertial Bregman proximal gradient method for minimizing the sum of two possibly nonconvex functions. This method includes two different inertial steps and adopts the Bregman regularization in solving the subproblem. Under some general parameter constraints, we prove the subsequential convergence that each generated sequence converges to the stationary point of the considered problem. To overcome the parameter constraints, we further propose a nonmonotone line search strategy to make the parameter selections more flexible. The subsequential convergence of the proposed method with line search is established. When the line search is monotone, we prove the stronger global convergence and linear convergence rate under Kurdyka–Łojasiewicz framework. Moreover, numerical results on SCAD and MCP nonconvex penalty problems are reported to demonstrate the effectiveness and superiority of the proposed methods and line search strategy.

A sharp error estimate of piecewise polynomial collocation for nonlocal problems with weakly singular kernels

Abstract As is well known, piecewise linear polynomial collocation (PLC) and piecewise quadratic polynomial collocation (PQC) are used to approximate the weakly singular integrals $$\begin{equation*}I(a,b,x) =\int^b_a \frac{u(y)}{|x-y|^\gamma}\textrm{d}y, \quad x \in (a,b),\quad 0< \gamma <1,\end{equation*}$$which have local truncation errors $\mathcal{O} (h^2 )$ and $\mathcal{O} (h^{4-\gamma } )$, respectively. Moreover, for Fredholm weakly singular integral equations of the second kind, i.e., $\lambda u(x)- I(a,b,x) =f(x)$, $\lambda \neq 0$, the global convergence rates are also $\mathcal{O} (h^2 )$ and $\mathcal{O} (h^{4-\gamma } )$ by PLC and PQC in Atkinson (2009, The Numerical Solution of Integral Equations of the Second Kind, Cambridge University Press). In this work we study the following nonlocal problems, which are similar to the above Fredholm integral equations: $$\begin{equation*}\int^b_a \frac{u(x)-u(y)}{|x-y|^\gamma}\textrm{d}y =f(x), \quad x \in (a,b),\quad 0< \gamma <1. \end{equation*}$$In the first part of this paper we prove that the weakly singular integrals $I(a,b,x)$ have optimal local truncation error $\mathcal{O}(h^4\eta _i^{-\gamma } )$ by PQC, where $\eta _i=\min \left \{x_i-a,b-x_i\right \}$ and $x_i$ coincides with an element junction point. Then the sharp global convergence orders $\mathcal{O}\left (h\right )$ and $\mathcal{O} (h^3)$ by PLC and PQC, respectively, are established for nonlocal problems. Finally, numerical experiments are shown to illustrate the effectiveness of the presented methods.

A dual symmetric Gauss-Seidel alternating direction method of multipliers for hyperspectral sparse unmixing

Since sparse unmixing has emerged as a promising approach to hyperspectral unmixing, some spatial-contextual information in the hyperspectral images has been exploited to improve the performance of the unmixing recently. The total variation (TV) has been widely used to promote the spatial homogeneity as well as the smoothness between adjacent pixels. However, the computation task for hyperspectral sparse unmixing with a TV regularization term is heavy. Besides, the convergence of the primal alternating direction method of multipliers (ADMM) for the hyperspectral sparse unmixing with a TV regularization term has not been explained in detail. In this paper, we design an efficient and convergent dual symmetric Gauss-Seidel ADMM (sGS-ADMM) for hyperspectral sparse unmixing with a TV regularization term. We also present the global convergence and local linear convergence rate analysis for this algorithm. As demonstrated in numerical experiments, our algorithm can obviously improve the efficiency of the unmixing compared with the state-of-the-art algorithm. More importantly, we can obtain images with higher quality.

An Improved Convergence Analysis for Decentralized Online Stochastic Non-Convex Optimization

In this paper, we study decentralized online stochastic non-convex optimization over a network of nodes. Integrating a technique called gradient tracking in decentralized stochastic gradient descent, we show that the resulting algorithm, GT-DSGD, enjoys certain desirable characteristics towards minimizing a sum of smooth non-convex functions. In particular, for general smooth non-convex functions, we establish non-asymptotic characterizations of GT-DSGD and derive the conditions under which it achieves network-independent performances that match the centralized minibatch SGD. In contrast, the existing results suggest that GT-DSGD is always network-dependent and is therefore strictly worse than the centralized minibatch SGD. When the global non-convex function additionally satisfies the Polyak-Łojasiewics (PL) condition, we establish the linear convergence of GT-DSGD up to a steady-state error with appropriate constant step-sizes. Moreover, under stochastic approximation step-sizes, we establish, for the first time, the optimal global sublinear convergence rate on almost every sample path, in addition to the asymptotically optimal sublinear rate in expectation. Since strongly convex functions are a special case of the functions satisfying the PL condition, our results are not only immediately applicable but also improve the currently known best convergence rates and their dependence on problem parameters.

Mean-square convergence of numerical methods for random ordinary differential equations

The aim of this work is to analyze the mean-square convergence rates of numerical schemes for random ordinary differential equations (RODEs). First, a relation between the global and local mean-square convergence order of one-step explicit approximations is established. Then, the global mean-square convergence rates are investigated for RODE-Taylor schemes for general RODEs, Affine-RODE-Taylor schemes for RODEs with affine noise, and Ito-Taylor schemes for RODEs with Ito noise, respectively. The theoretical convergence results are demonstrated through numerical experiments.