Step-size Rule Research Articles

A linearly convergent doubly stochastic Gauss–Seidel algorithm for solving linear equations and a certain class of over-parameterized optimization problems

Consider the classical problem of solving a general linear system of equations $Ax=b$. It is well known that the (successively over relaxed) Gauss-Seidel scheme and many of its variants may not converge when $A$ is neither diagonally dominant nor symmetric positive definite. Can we have a linearly convergent G-S type algorithm that works for {\it any} $A$? In this paper we answer this question affirmatively by proposing a doubly stochastic G-S algorithm that is provably linearly convergent (in the mean square error sense) for any feasible linear system of equations. The key in the algorithm design is to introduce a {\it nonuniform double stochastic} scheme for picking the equation and the variable in each update step as well as a stepsize rule. These techniques also generalize to certain iterative alternating projection algorithms for solving the linear feasibility problem $A x\le b$ with an arbitrary $A$, as well as high-dimensional minimization problems for training over-parameterized models in machine learning. Our results demonstrate that a carefully designed randomization scheme can make an otherwise divergent G-S algorithm converge.

Non-stationary Douglas–Rachford and alternating direction method of multipliers: adaptive step-sizes and convergence

We revisit the classical Douglas–Rachford (DR) method for finding a zero of the sum of two maximal monotone operators. Since the practical performance of the DR method crucially depends on the step-sizes, we aim at developing an adaptive step-size rule. To that end, we take a closer look at a linear case of the problem and use our findings to develop a step-size strategy that eliminates the need for step-size tuning. We analyze a general non-stationary DR scheme and prove its convergence for a convergent sequence of step-sizes with summable increments in the case of maximally monotone operators. This, in turn, proves the convergence of the method with the new adaptive step-size rule. We also derive the related non-stationary alternating direction method of multipliers. We illustrate the efficiency of the proposed methods on several numerical examples.

Modified extragradient-like algorithms with new stepsizes for variational inequalities

The paper concerns with an algorithm for approximating solutions of a variational inequality problem involving a Lipschitz continuous and monotone operator in a Hilbert space. The algorithm uses a new stepsize rule which does not depend on the Lipschitz constant and without any linesearch procedure. The resulting algorithm only requires to compute a projection on feasible set and a value of operator over each iteration. The convergence and the convergence rate of the algorithm are established. Some experiments are performed to show the numerical behavior of the proposed algorithm and also to compare its performance with those of others.

A Gradient Sampling Method on Algebraic Varieties and Application to Nonsmooth Low-Rank Optimization

In this paper, a nonsmooth optimization method for locally Lipschitz functions on real algebraic varieties is developed. To this end, the set-valued map $\varepsilon$-conditional subdifferential $x\to \partial_{\varepsilon}^{N} f(x):= \partial_{\varepsilon}f(x)+N(x)$ is introduced, where $\partial_{\varepsilon}f(x)$ is the Goldstein-$\varepsilon$-subdifferential and $N(x)$ is a closed convex cone at $x$. It is proved that negative of the shortest $\varepsilon$-conditional subgradient provides a descent direction in $T(x)$, which denotes the polar of $N(x)$. The $\varepsilon$-conditional subdifferential at an iterate $x_{\ell}$ can be approximated by a convex hull of a finite set of projected gradients at sampling points in $x_\ell+\varepsilon_{\ell} B_{T(x_{\ell})}(0,1)$ to $T(x_{\ell})$, where $T(x_{\ell})$ is a linear space in the Bouligand tangent cone and $ B_{T(x_{\ell})}(0,1)$ denotes the unit ball in $T(x_{\ell})$. The negative of the shortest vector in this convex hull is shown to be a descent direction in the Bouligand tangent cone at $x_{\ell}$. The proposed algorithm makes a step along this descent direction with a certain step-size rule, followed by a retraction to lift back to points on the algebraic variety $\mathcal{M}$. The convergence of the resulting algorithm to a critical point is proved. For numerical illustration, the considered method is applied to some nonsmooth problems on varieties of low-rank matrices $\mathcal{M}_{\leq r}$ of real $M\times N$ matrices of rank at most $r$, specifically robust low-rank matrix approximation and recovery in the presence of outliers.

Extremum Seeking Algorithms based on Non-Commutative Maps

In this work, discrete-time extremum seeking algorithms for unconstrained optimization problems are developed. A general class of non-commutative maps and one- and two point function evaluation polices are presented to approximate a gradient-descent algorithm, suitable for extremum seeking problems. Moreover, adaptive step size rules are discussed to achieve faster convergence and vanishing steady state oscillations.

Convergence Rate of Incremental Gradient and Incremental Newton Methods

The incremental gradient method is a prominent algorithm for minimizing a finite sum of smooth convex functions, used in many contexts including large-scale data processing applications and distributed optimization over networks. It is a first-order method that processes the functions one at a time based on their gradient information. The incremental Newton method, on the other hand, is a second-order variant which exploits additionally the curvature information of the underlying functions and can therefore be faster. In this paper, we focus on the case when the objective function is strongly convex and present fast convergence results for the incremental gradient and incremental Newton methods under the constant and diminishing stepsizes. For a decaying stepsize rule $\alpha_k = \Theta(1/k^s)$ with $s \in (0,1]$, we show that the distance of the IG iterates to the optimal solution converges at rate ${\cal O}(1/k^{s})$ (which translates into ${\cal O}(1/k^{2s})$ rate in the suboptimality of the objective value). For $s>1/2$, this improves the previous ${\cal O}(1/\sqrt{k})$ results in distances obtained for the case when functions are non-smooth. We show that to achieve the fastest ${\cal O}(1/k)$ rate, incremental gradient needs a stepsize that requires tuning to the strong convexity parameter whereas the incremental Newton method does not. The results are based on viewing the incremental gradient method as a gradient descent method with gradient errors, devising efficient upper bounds for the gradient error to derive inequalities that relate distances of the consecutive iterates to the optimal solution and finally applying Chung's lemmas from the stochastic approximation literature to these inequalities to determine their asymptotic behavior. In addition, we construct examples to show tightness of our rate results.

Self-adaptive iterative method for solving boundedly Lipschitz continuous and strongly monotone variational inequalities

In this paper we introduce a new self-adaptive iterative algorithm for solving the variational inequalities in real Hilbert spaces, denoted by operatorname{VI}(C, F). Here Csubseteq mathcal{H} is a nonempty, closed and convex set and F: Crightarrow mathcal{H} is boundedly Lipschitz continuous (i.e., Lipschitz continuous on any bounded subset of C) and strongly monotone operator. One of the advantages of our algorithm is that it does not require the knowledge of the Lipschitz constant of F on any bounded subset of C or the strong monotonicity coefficient a priori. Moreover, the proposed self-adaptive step size rule only adds a small amount of computational effort and hence guarantees fast convergence rate. Strong convergence of the method is proved and a posteriori error estimate of the convergence rate is obtained.Primary numerical results illustrate the behavior of our proposed scheme and also suggest that the convergence rate of the method is comparable with the classical gradient projection method for solving variational inequalities.

Tseng type methods for solving inclusion problems and its applications

In this paper, we introduce two modifications of the forward–backward splitting method with a new step size rule for inclusion problems in real Hilbert spaces. The modifications are based on Mann and viscosity-ideas. Under standard assumptions, such as Lipschitz continuity and monotonicity (also maximal monotonicity), we establish strong convergence of the proposed algorithms. We present two numerical examples, the first in infinite dimensional spaces, which illustrates mainly the strong convergence property of the algorithm. For the second example, we illustrate the performances of our scheme, compared with the classical forward–backward splitting method for the problem of recovering a sparse noisy signal. Our result extend some related works in the literature and the primary experiments might also suggest their potential applicability.

A convergence analysis of the method of codifferential descent

This paper is devoted to a detailed convergence analysis of the method of codifferential descent (MCD) developed by professor V.F. Demyanov for solving a large class of nonsmooth nonconvex optimization problems. We propose a generalization of the MCD that is more suitable for applications than the original method, and that utilizes only a part of a codifferential on every iteration, which allows one to reduce the overall complexity of the method. With the use of some general results on uniformly codifferentiable functions obtained in this paper, we prove the global convergence of the generalized MCD in the infinite dimensional case. Also, we propose and analyse a quadratic regularization of the MCD, which is the first general method for minimizing a codifferentiable function over a convex set. Apart from convergence analysis, we also discuss the robustness of the MCD with respect to computational errors, possible step size rules, and a choice of parameters of the algorithm. In the end of the paper we estimate the rate of convergence of the MCD for a class of nonsmooth nonconvex functions that arises, in particular, in cluster analysis. We prove that under some general assumptions the method converges with linear rate, and it convergence quadratically, provided a certain first order sufficient optimality condition holds true.

Membership overlay design optimization with resource constraints (accelerated on GPU)

Centralized overlay network management, such as for Service Overlay Networks, is an important topicfor Internet based services. The computational efficiency of the central controller node is of paramount importance to guarantee the quality of the service. The paper considers the problem where the network can be also asymmetric and each node requires to be connected with nodes which provide some given resources. This work proposes a comparison of alternative approaches to the parallelization of a dynamic overlay network reconfiguration algorithm, implemented as a distributed Lagrangian algorithm. The proposed approach is based on a subgradient algorithm which makes use of the quasi-constant step-size rule specifically studied for a parallel/distributed implementation. The method is implemented using MPI, OpenMP and CUDA on GPU and was focussed on the membership overlay reconfiguration for test instances up to 1000 nodes.

String-averaging incremental stochastic subgradient algorithms

ABSTRACTWe present a method to solve constrained convex stochastic optimization problems when the objective is a finite sum of convex functions . Our method is based on Incremental Stochastic Subgradient Algorithms and String-Averaging techniques, with an assumption that the subgradient directions are affected by random errors in each iteration. Our analysis allows the method to perform approximate projections onto the feasible set in each iteration. We provide convergence results for the case where a diminishing step-size rule is used. We test our method in a large set of random instances of a stochastic convex programming problem and we compare its performance with the robust mirror descent stochastic approximation algorithm proposed in Nemirovski et al. (Robust stochastic approximation approach to stochastic programming, SIAM J Optim 19 (2009), pp. 15741609).

Totally relaxed, self-adaptive algorithm for solving variational inequalities over the intersection of sub-level sets

In this paper we study the classical Variational Inequality (VI) over the intersection of sub-level sets of finite convex functions in real Hilbert spaces. The Subgradient Extragradient method of Censor et al. extend Korpelevich’s extragradient method by introducing an additional constructible set and then there is the need to calculate only one orthogonal projection onto the feasible set per each iteration instead of two as in. Motivated by this result, we propose a new extension, called the Totally Relaxed and Self-adaptive Subgradient Extragradient Method, which does not require the calculation of any exact projections onto the VI’s feasible set. Hence, any general convex feasible sets can be involved in the VI, such as the finite intersection of sub-level sets of convex functions. In our new scheme we also introduce an adaptive step-size rule which avoids the need to know a priori the Lipschitz constant of the VI associated mapping. Under mild and standard assumptions, we prove weak convergence of the proposed method at rate in the ergodic sense. Two numerical examples are presented to illustrate the behaviour and performances of out proposed scheme.

Abstract convergence theorem for quasi-convex optimization problems with applications

Quasi-convex optimization is fundamental to the modelling of many practical problems in various fields such as economics, finance and industrial organization. Subgradient methods are practical iterative algorithms for solving large-scale quasi-convex optimization problems. In the present paper, focusing on quasi-convex optimization, we develop an abstract convergence theorem for a class of sequences, which satisfy a general basic inequality, under some suitable assumptions on parameters. The convergence properties in both function values and distances of iterates from the optimal solution set are discussed. The abstract convergence theorem covers relevant results of many types of subgradient methods studied in the literature, for either convex or quasi-convex optimization. Furthermore, we propose a new subgradient method, in which a perturbation of the successive direction is employed at each iteration. As an application of the abstract convergence theorem, we obtain the convergence results of the proposed subgradient method under the assumption of the Hölder condition of order p and by using the constant, diminishing or dynamic stepsize rules, respectively. A preliminary numerical study shows that the proposed method outperforms the standard, stochastic and primal-dual subgradient methods in solving the Cobb–Douglas production efficiency problem.

Strong convergence of a double projection-type method for monotone variational inequalities in Hilbert spaces

We introduce a projection-type algorithm for solving monotone variational inequality problems in real Hilbert spaces without assuming Lipschitz continuity of the corresponding operator. We prove that the whole sequence of iterates converges strongly to a solution of the variational inequality. The method uses only two projections onto the feasible set in each iteration in contrast to other strongly convergent algorithms which either require plenty of projections within a step size rule or have to compute projections on possibly more complicated sets. Some numerical results illustrate the behavior of our method.

An Incremental Subgradient Method on Riemannian Manifolds

In this paper, we propose and analyze an incremental subgradient method with a diminishing stepsize rule for a convex optimization problem on a Riemannian manifold, where the object function consisted of the sum of a large number of component functions. This type of function is useful in lots of fields. We establish an important inequality about the sequence generated by the method, provided the sectional curvature of the manifold is nonnegative. Using the inequality, we prove Proposition 3.1, and then we obtain some convergence results of the incremental subgradient method.

Decentralized Frank–Wolfe Algorithm for Convex and Nonconvex Problems

Decentralized optimization algorithms have received much attention due to the recent advances in network information processing. However, conventional decentralized algorithms based on projected gradient descent are incapable of handling high-dimensional constrained problems, as the projection step becomes computationally prohibitive. To address this problem, this paper adopts a projection-free optimization approach, a.k.a. the Frank-Wolfe (FW) or conditional gradient algorithm. We first develop a decentralized FW (DeFW) algorithm from the classical FW algorithm. The convergence of the proposed algorithm is studied by viewing the decentralized algorithm as an inexact FW algorithm. Using a diminishing step size rule and letting t be the iteration number, we show that the DeFW algorithm's convergence rate is O(1/t) for convex objectives; is O(1/t <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) for strongly convex objectives with the optimal solution in the interior of the constraint set; and is O(1/√t) toward a stationary point for smooth but nonconvex objectives. We then show that a consensus-based DeFW algorithm meets the above guarantees with two communication rounds per iteration. We demonstrate the advantages of the proposed DeFW algorithm on low-complexity robust matrix completion and communication efficient sparse learning. Numerical results on synthetic and real data are presented to support our findings.

Weak and strong convergence theorems for variational inequality problems

In this paper, we study the weak and strong convergence of two algorithms for solving Lipschitz continuous and monotone variational inequalities. The algorithms are inspired by Tseng's extragradient method and the viscosity method with Armijo-like step size rule. The main advantages of our algorithms are that the construction of solution approximations and the proof of convergence of the algorithms are performed without the prior knowledge of the Lipschitz constant of cost operators. Finally, we provide numerical experiments to show the efficiency and advantage of the proposed algorithms.

On the convergence of s-dependent GFR conjugate gradient method for unconstrained optimization

In this paper, the authors present an s-dependent conjugate gradient method for unconstrained optimization problem and make two different kinds of estimations of upper bounds of β k with respect to $$\beta_{k}^{\mathrm{FR}}$$ which are called dependent ratio. The global convergence of s-dependent GFR conjugate gradient method using several step-size rules is obtained.

On the worst-case evaluation complexity of non-monotone line search algorithms

A general class of non-monotone line search algorithms has been proposed by Sachs and Sachs (Control Cybern 40:1059–1075, 2011) for smooth unconstrained optimization, generalizing various non-monotone step size rules such as the modified Armijo rule of Zhang and Hager (SIAM J Optim 14:1043–1056, 2004). In this paper, the worst-case complexity of this class of non-monotone algorithms is studied. The analysis is carried out in the context of non-convex, convex and strongly convex objectives with Lipschitz continuous gradients. Despite de nonmonotonicity in the decrease of function values, the complexity bounds obtained agree in order with the bounds already established for monotone algorithms.

Numerical algorithms for scatter-to-attenuation reconstruction in PET: empirical comparison of convergence, acceleration, and the effect of subsets.

MLGA was combined with the Armijo step size rule; and accelerated using conjugate gradients, Nesterov's momentum method, and data subsets of different sizes. In 2BP, we varied the subset size, an important determinant of convergence speed and computational burden. We used three sets of simulation data to evaluate the impact of a spatial scale factor. The Armijo step size allowed 10-fold increased step sizes compared to native MLGA. Conjugate gradients and Nesterov momentum lead to slightly faster, yet non-uniform convergence; improvements were mostly confined to later iterations, possibly due to the non-linearity of the problem. MLGA with data subsets achieved faster, uniform, and predictable convergence, with a speed-up factor equivalent to the number of subsets and no increase in computational burden. By contrast, 2BP computational burden increased linearly with the number of subsets due to repeated evaluation of the objective function, and convergence was limited to the case of many (and therefore small) subsets, which resulted in high computational burden. Possibilities of improving 2BP appear limited. While general-purpose acceleration methods appear insufficient for MLGA, results suggest that data subsets are a promising way of improving MLGA performance.