Strong Convexity Research Articles

Convergence Rate of Incremental Gradient and Incremental Newton Methods

The incremental gradient method is a prominent algorithm for minimizing a finite sum of smooth convex functions, used in many contexts including large-scale data processing applications and distributed optimization over networks. It is a first-order method that processes the functions one at a time based on their gradient information. The incremental Newton method, on the other hand, is a second-order variant which exploits additionally the curvature information of the underlying functions and can therefore be faster. In this paper, we focus on the case when the objective function is strongly convex and present fast convergence results for the incremental gradient and incremental Newton methods under the constant and diminishing stepsizes. For a decaying stepsize rule $\alpha_k = \Theta(1/k^s)$ with $s \in (0,1]$, we show that the distance of the IG iterates to the optimal solution converges at rate ${\cal O}(1/k^{s})$ (which translates into ${\cal O}(1/k^{2s})$ rate in the suboptimality of the objective value). For $s>1/2$, this improves the previous ${\cal O}(1/\sqrt{k})$ results in distances obtained for the case when functions are non-smooth. We show that to achieve the fastest ${\cal O}(1/k)$ rate, incremental gradient needs a stepsize that requires tuning to the strong convexity parameter whereas the incremental Newton method does not. The results are based on viewing the incremental gradient method as a gradient descent method with gradient errors, devising efficient upper bounds for the gradient error to derive inequalities that relate distances of the consecutive iterates to the optimal solution and finally applying Chung's lemmas from the stochastic approximation literature to these inequalities to determine their asymptotic behavior. In addition, we construct examples to show tightness of our rate results.

Convergence Rates for Deterministic and Stochastic Subgradient Methods without Lipschitz Continuity

We extend the classic convergence rate theory for subgradient methods to apply to non-Lipschitz functions. For the deterministic projected subgradient method, we present a global $O(1/\sqrt{T})$ convergence rate for any convex function which is locally Lipschitz around its minimizers. This approach is based on Shor's classic subgradient analysis and implies generalizations of the standard convergence rates for gradient descent on functions with Lipschitz or Hölder continuous gradients. Further, we show a $O(1/\sqrt{T})$ convergence rate for the stochastic projected subgradient method on convex functions with at most quadratic growth, which improves to $O(1/T)$ under either strong convexity or a weaker quadratic lower bound condition.

Determination of stress state in coal seam based on inverse problem solution using acoustic sounding data

The paper presents an algorithm for calculating the horizontal, vertical and tangential stresses in a horizontal coal seam lying between galleries. These stresses are expressed in terms of the Airy function, which satisfies a homogeneous biharmonic equation. For its numerical solution it is necessary to set boundary conditions. Practical limitations do not allow us to determine the tangential stresses on horizontal boundaries of the coal seam. Calculations of stresses are proposed to be carried out in two steps. The first step is to solve the inverse problem for the biharmonic equation to find unknown tangential stresses on horizontal boundaries of the coal seam. The inverse problem is solved by minimizing the residual functional. Its strong convexity is proved, which implies the existence and uniqueness of the solution of the inverse problem. The second step is to solve the boundary value problem for the biharmonic equation to calculate stresses in the coal seam. The result of a numerical experiment is presented.

A Sparse Stochastic Method With Optimal Convergence Rate for Optimizing Pseudo-Linear Performance Measures

Optimizing pseudo-linear performance measures has gained increasing interest in recent years. Compared with the existing works focusing on L2-norm, it is more challenging to design the model with L1-norm, especially, when data are of large scale. To address the issue, in this paper, we propose a sparse stochastic method to optimize the pseudo-linear metrics. The algorithm begins by formulating the problem as a cost-sensitive classification with L1 regularization, then composite objective mirror descent (COMID) is adopted for the inner optimization, which can obtain a sparse solution with promising performance. However, the original COMID method only has $\mathcal {O}(log(T)/T)$ convergence rate with the strong convex objective. To this end, a simple yet efficient polynomial-decay average strategy is suggested. We prove that with this strategy, the proposed algorithm can not only achieve the optimal convergence rate of $\mathcal {O}(1/T)$ but also obtain the classifier with higher sparsity. The empirical studies on several public benchmark data sets demonstrate the superior performance of the proposed algorithm in terms of both efficiency and sparsity.

A Coordinate-Descent Primal-Dual Algorithm with Large Step Size and Possibly Nonseparable Functions

This paper introduces a coordinate descent version of the V\~u-Condat algorithm. By coordinate descent, we mean that only a subset of the coordinates of the primal and dual iterates is updated at each iteration, the other coordinates being maintained to their past value. Our method allows us to solve optimization problems with a combination of differentiable functions, constraints as well as non-separable and non-differentiable regularizers. We show that the sequences generated by our algorithm converge to a saddle point of the problem at stake, for a wider range of parameter values than previous methods. In particular, the condition on the step-sizes depends on the coordinate-wise Lipschitz constant of the differentiable function's gradient, which is a major feature allowing classical coordinate descent to perform so well when it is applicable. We then prove a sublinear rate of convergence in general and a linear rate of convergence if the objective enjoys strong convexity properties. We illustrate the performances of the algorithm on a total-variation regularized least squares regression problem and on large scale support vector machine problems.

Convergence of the Iterates in Mirror Descent Methods

We consider centralized and distributed mirror descent (MD) algorithms over a finite-dimensional Hilbert space, and prove that the problem variables converge to an optimizer of a possibly nonsmooth function when the step sizes are square summable but not summable. Prior literature has focused on the convergence of the function value to its optimum. However, applications from distributed optimization and learning in games require the convergence of the variables to an optimizer, which is generally not guaranteed without assuming strong convexity of the objective function. We provide numerical simulations comparing entropic MD and standard subgradient methods for the robust regression problem.

Distributed Stochastic Subgradient Projection Algorithms Based on Weight‐Balancing over Time‐Varying Directed Graphs

We consider a distributed constrained optimization problem over graphs, where cost function of each agent is private. Moreover, we assume that the graphs are time‐varying and directed. In order to address such problem, a fully decentralized stochastic subgradient projection algorithm is proposed over time‐varying directed graphs. However, since the graphs are directed, the weight matrix may not be a doubly stochastic matrix. Therefore, we overcome this difficulty by using weight‐balancing technique. By choosing appropriate step‐sizes, we show that iterations of all agents asymptotically converge to some optimal solutions. Further, by our analysis, convergence rate of our proposed algorithm is O(ln Γ/Γ) under local strong convexity, where Γ is the number of iterations. In addition, under local convexity, we prove that our proposed algorithm can converge with rate . In addition, we verify the theoretical results through simulations.

Distributed constrained optimization for multi-agent networks with nonsmooth objective functions

This paper investigates the distributed constrained optimization problem in multi-agent systems, where agents cooperatively minimize an objective function being the sum of each agent’s objective function while meeting equality and inequality constraints. By virtue of nonsmooth analysis and graph theory, a novel distributed continuous-time algorithm is proposed to solve such a kind of optimization problems. Different from existing continuous-time results relying on the differentiability or strict (strong) convexity of local objective functions, the proposed approach considers more general local objective functions which are only convex and not necessarily smooth. In addition, the proposed approach considers more general local constraints, not just box constraints considered in most existing studies. The optimality of the proposed algorithm is ensured under certain initial condition. Based on set-valued LaSalle invariance principle, the convergence of the proposed scheme is rigorously proved. Finally, simulation examples are applied to validate the effectiveness of the proposed algorithm.

Guaranteed Recovery of One-Hidden-Layer Neural Networks via Cross Entropy

We study model recovery for data classification, where the training labels are generated from a one-hidden-layer neural network with sigmoid activations, also known as a single-layer feedforward network, and the goal is to recover the weights of the neural network. We consider two network models, the fully-connected network (FCN) and the non-overlapping convolutional neural network (CNN). We prove that with Gaussian inputs, the empirical risk based on cross entropy exhibits strong convexity and smoothness uniformly in a local neighborhood of the ground truth, as soon as the sample complexity is sufficiently large. This implies that if initialized in this neighborhood, gradient descent converges linearly to a critical point that is provably close to the ground truth. Furthermore, we show such an initialization can be obtained via the tensor method. This establishes the global convergence guarantee for empirical risk minimization using cross entropy via gradient descent for learning one-hidden-layer neural networks, at the near-optimal sample and computational complexity with respect to the network input dimension without unrealistic assumptions such as requiring a fresh set of samples at each iteration.

Randomized Primal–Dual Proximal Block Coordinate Updates

In this paper, we propose a randomized primal–dual proximal block coordinate updating framework for a general multi-block convex optimization model with coupled objective function and linear constraints. Assuming mere convexity, we establish its O(1 / t) convergence rate in terms of the objective value and feasibility measure. The framework includes several existing algorithms as special cases such as a primal–dual method for bilinear saddle-point problems (PD-S), the proximal Jacobian alternating direction method of multipliers (Prox-JADMM) and a randomized variant of the ADMM for multi-block convex optimization. Our analysis recovers and/or strengthens the convergence properties of several existing algorithms. For example, for PD-S our result leads to the same order of convergence rate without the previously assumed boundedness condition on the constraint sets, and for Prox-JADMM the new result provides convergence rate in terms of the objective value and the feasibility violation. It is well known that the original ADMM may fail to converge when the number of blocks exceeds two. Our result shows that if an appropriate randomization procedure is invoked to select the updating blocks, then a sublinear rate of convergence in expectation can be guaranteed for multi-block ADMM, without assuming any strong convexity. The new approach is also extended to solve problems where only a stochastic approximation of the subgradient of the objective is available, and we establish an $$O(1/\sqrt{t})$$ convergence rate of the extended approach for solving stochastic programming.

Restricted strong convexity implies weak submodularity

We connect high-dimensional subset selection and submodular maximization. Our results extend the work of Das and Kempe [In ICML (2011) 1057–1064] from the setting of linear regression to arbitrary objective functions. For greedy feature selection, this connection allows us to obtain strong multiplicative performance bounds on several methods without statistical modeling assumptions. We also derive recovery guarantees of this form under standard assumptions. Our work shows that greedy algorithms perform within a constant factor from the best possible subset-selection solution for a broad class of general objective functions. Our methods allow a direct control over the number of obtained features as opposed to regularization parameters that only implicitly control sparsity. Our proof technique uses the concept of weak submodularity initially defined by Das and Kempe. We draw a connection between convex analysis and submodular set function theory which may be of independent interest for other statistical learning applications that have combinatorial structure.

Tight Global Linear Convergence Rate Bounds for Operator Splitting Methods

In this paper, we establish necessary and sufficient conditions for global linear convergence rate bounds in operator splitting methods for a general class of convex optimization problems, where the associated fixed-point operator is strongly quasi-nonexpansive. We also provide a tight bound on the achievable convergence rate. Most existing results establishing global linear convergence in such methods require restrictive assumptions regarding strong convexity and smoothness of the constituent functions in the optimization problem. However, there are several examples in the literature showing that linear convergence is possible even when these properties do not hold. We provide a unifying analysis method for establishing global linear convergence based on linear regularity and show that many existing results are special cases of our approach. Moreover, we propose a novel linearly convergent splitting method for linear programming.

A Modified Proximal Gradient Method for a Class of Sparse Optimization Problems

In this paper, we propose a modified proximal gradient method for a class of sparse optimization problems, which arise in many contemporary statistical and signal processing applications. The new method uses a new scheme to construct the descent direction based on the proximal gradient method. It is proven that the modified proximal gradient method is Q-linearly convergent without the assumption of the strong convexity of the objective function. Numerical experiments have been conducted to evaluate the proposed method.

A Projected Primal–Dual Method for Solving Constrained Monotone Inclusions

In this paper, we provide an algorithm for solving constrained composite primal–dual monotone inclusions, i.e., monotone inclusions in which a priori information on primal–dual solutions is represented via closed and convex sets. The proposed algorithm incorporates a projection step onto the a priori information sets and generalizes methods proposed in the literature for solving monotone inclusions. Moreover, under the presence of strong monotonicity, we derive an accelerated scheme inspired on the primal–dual algorithm applied to the more general context of constrained monotone inclusions. In the particular case of convex optimization, our algorithm generalizes several primal–dual optimization methods by allowing a priori information on solutions. In addition, we provide an accelerated scheme under strong convexity. An application of our approach with a priori information is constrained convex optimization problems, in which available primal–dual methods impose constraints via Lagrange multiplier updates, usually leading to slow algorithms with unfeasible primal iterates. The proposed modification forces primal iterates to satisfy a selection of constraints onto which we can project, obtaining a faster method as numerical examples exhibit. The obtained results extend and improve several results in the literature.

A Bregman inexact linesearch–based forward–backward algorithm for nonsmooth nonconvex optimization

In this paper, we present a forward-backward linesearch-based algorithm suited for the minimization of the sum of a smooth (possibly nonconvex) function and a convex (possibly nonsmooth) term. Such algorithm first computes inexactly the proximal operator with respect to a given Bregman distance, and then ensures a sufficient decrease condition by performing a linesearch along the descent direction. The proposed approach can be seen as an instance of the more general class of descent methods presented in [7], however, unlike in [7], we do not assume the strong convexity of the Bregman distance used in the proximal evaluation. We prove that each limit point of the iterates sequence is stationary, we show how to compute an approximate proximal–gradient point with respect to a Bregman distance and, finally, we report the good numerical performance of the algorithm on a large scale image restoration problem.

On starting and stopping criteria for nested primal-dual iterations

The importance of an adequate inner loop starting point (as opposed to a sufficient inner loop stopping rule) is discussed in the context of a numerical optimization algorithm consisting of nested primal-dual proximal-gradient iterations. While the number of inner iterations is fixed in advance, convergence of the whole algorithm is still guaranteed by virtue of a warm-start strategy for the inner loop, showing that inner loop “starting rules” can be just as effective as “stopping rules” for guaranteeing convergence. The algorithm itself is applicable to the numerical solution of convex optimization problems defined by the sum of a differentiable term and two possibly non-differentiable terms. One of the latter terms should take the form of the composition of a linear map and a proximable function, while the differentiable term needs an accessible gradient. The algorithm reduces to the classical proximal gradient algorithm in certain special cases and it also generalizes other existing algorithms. In addition, under some conditions of strong convexity, we show a linear rate of convergence.

Primal Averaging: A New Gradient Evaluation Step to Attain the Optimal Individual Convergence.

Many well-known first-order gradient methods have been extended to cope with large-scale composite problems, which often arise as a regularized empirical risk minimization in machine learning. However, their optimal convergence is attained only in terms of the weighted average of past iterative solutions. How to make the individual convergence of stochastic gradient descent (SGD) optimal, especially for strongly convex problems has now become a challenging problem in the machine learning community. On the other hand, Nesterov's recent weighted averaging strategy succeeds in achieving the optimal individual convergence of dual averaging (DA) but it fails in the basic mirror descent (MD). In this paper, a new primal averaging (PA) gradient operation step is presented, in which the gradient evaluation is imposed on the weighted average of all past iterative solutions. We prove that simply modifying the gradient operation step in MD by PA strategy suffices to recover the optimal individual rate for general convex problems. Along this line, the optimal individual rate of convergence for strongly convex problems can also be achieved by imposing the strong convexity on the gradient operation step. Furthermore, we extend PA-MD to solve regularized nonsmooth learning problems in the stochastic setting, which reveals that PA strategy is a simple yet effective extra step toward the optimal individual convergence of SGD. Several real experiments on sparse learning and SVM problems verify the correctness of our theoretical analysis.

Stochastic comparison bounds for an M1, M2/G1, G2/1 retrial queue with two way communication

The main goal in the present paper is to provide a technique that considers the stochastic comparison approach for investigating monotonicity and comparability of an $M_{1},M_{2}/G_{1},G_{2}/1$ retrial queues with two way communication. This approach is developed for comparing a non Markov process to Markov process with many possible stochastic orderings. Particularly, we show the monotonicity of the transition operator of the embedded Markov chain relative to the strong stochastic ordering and convex ordering, as well as the comparability of two transition operators. Bounds are also obtained for the stationary distribution of the number of customers at departure epochs. Additionally, the performance measures of the system considered can be estimated by those of an $M_1,M_2/M_1,M_2/1$ retrial queue with two way communication when the service time distribution is NBUE (respectively NWUE). Finally, we validate stochastic comparison results by presenting a numerical example illustrating the interest of the approach.

Viscosity Approximation Methods for Generalized Multi-Valued Nonexpansive Mappings with Applications

In this article, we study viscosity approximation methods for generalized multi-valued nonexpansive mappings and we present some new results related to strong convergence, variational inequality, convex optimization, split and common split feasibility problems (SFPs). Some numerical computations are also presented to illustrate our results.

Proximal alternating penalty algorithms for nonsmooth constrained convex optimization

We develop two new proximal alternating penalty algorithms to solve a wide range class of constrained convex optimization problems. Our approach mainly relies on a novel combination of the classical quadratic penalty, alternating minimization, Nesterov’s acceleration, adaptive strategy for parameters. The first algorithm is designed to solve generic and possibly nonsmooth constrained convex problems without requiring any Lipschitz gradient continuity or strong convexity, while achieving the best-known $$\mathcal {O}\left( \frac{1}{k}\right) $$ -convergence rate in a non-ergodic sense, where k is the iteration counter. The second algorithm is also designed to solve non-strongly convex, but semi-strongly convex problems. This algorithm can achieve the best-known $$\mathcal {O}\left( \frac{1}{k^2}\right) $$ -convergence rate on the primal constrained problem. Such a rate is obtained in two cases: (1) averaging only on the iterate sequence of the strongly convex term, or (2) using two proximal operators of this term without averaging. In both algorithms, we allow one to linearize the second subproblem to use the proximal operator of the corresponding objective term. Then, we customize our methods to solve different convex problems, and lead to new variants. As a byproduct, these algorithms preserve the same convergence guarantees as in our main algorithms. We verify our theoretical development via different numerical examples and compare our methods with some existing state-of-the-art algorithms.