Stochastic Subgradient Research Articles

High Probability Bounds for Stochastic Subgradient Schemes with Heavy Tailed Noise]

High Probability Bounds for Stochastic Subgradient Schemes with Heavy Tailed Noise]

Nonsmooth projection-free optimization with functional constraints

AbstractThis paper presents a subgradient-based algorithm for constrained nonsmooth convex optimization that does not require projections onto the feasible set. While the well-established Frank–Wolfe algorithm and its variants already avoid projections, they are primarily designed for smooth objective functions. In contrast, our proposed algorithm can handle nonsmooth problems with general convex functional inequality constraints. It achieves an $$\epsilon $$ ϵ -suboptimal solution in $$\mathcal {O}(\epsilon ^{-2})$$ O ( ϵ - 2 ) iterations, with each iteration requiring only a single (potentially inexact) Linear Minimization Oracle call and a (possibly inexact) subgradient computation. This performance is consistent with existing lower bounds. Similar performance is observed when deterministic subgradients are replaced with stochastic subgradients. In the special case where there are no functional inequality constraints, our algorithm competes favorably with a recent nonsmooth projection-free method designed for constraint-free problems. Our approach utilizes a simple separation scheme in conjunction with a new Lagrange multiplier update rule.

Distributed Heterogeneous Multi-Agent Optimization with Stochastic Sub-Gradient

Distributed Heterogeneous Multi-Agent Optimization with Stochastic Sub-Gradient

Mini-Batch Risk Forms

Risk forms are real functionals of two arguments: a bounded measurable function on a Polish space and a probability measure on that space. They are convenient mathematical structures adapting the coherent risk measures to the situation of a variable reference probability measure. We introduce a new class of risk forms called mini-batch forms. We construct them by using a random empirical probability measure as the second argument and by post-composition with the expected value operator. We prove that coherent and law invariant risk forms generate mini-batch risk forms which are well defined on the space of integrable random variables, and we derive their dual representation. We demonstrate how unbiased stochastic subgradients of such risk forms can be constructed. Then, we consider pre-compositions of mini-batch risk forms with nonsmooth and nonconvex functions, which are differentiable in a generalized way, and we derive generalized subgradients and unbiased stochastic subgradients of such compositions. Finally, we study the dependence of risk forms and mini-batch risk forms on perturbation of the probability measure and establish quantitative stability in terms of optimal transport metrics. We obtain finite-sample expected error estimates for mini-batch risk forms involving functions on a finite-dimensional space.

Mini-batch stochastic subgradient for functional constrained optimization

In this paper, we consider finite sum composite optimization problems with many functional constraints. The objective function is expressed as a finite sum of two terms, one of which admits easy computation of (sub)gradients while the other is amenable to proximal evaluations. We assume a generalized bounded gradient condition on the objective which allows us to simultaneously tackle both smooth and nonsmooth problems. We also consider the cases of both with and without a quadratic functional growth property. Further, we assume that each constraint set is given as the level set of a convex but not necessarily a differentiable function. We reformulate the constrained finite sum problem into a stochastic optimization problem for which the stochastic subgradient projection method from Necoara and Singh [Stochastic subgradient projection methods for composite optimization with functional constraints; 2022 Journal of Machine Learning Research, 23, 1–35] specializes in a collection of mini-batch variants, with different mini-batch sizes for the objective function and functional constraints, respectively. More specifically, at each iteration, our algorithm takes a mini-batch stochastic proximal subgradient step aimed at minimizing the objective function and then a subsequent mini-batch subgradient projection step minimizing the feasibility violation. By specializing in different mini-batching strategies, we derive exact expressions for the stepsizes as a function of the mini-batch size and in some cases we also derive insightful stepsize-switching rules which describe when one should switch from a constant to a decreasing stepsize regime. We also prove sublinear convergence rates for the mini-batch subgradient projection algorithm which depend explicitly on the mini-batch sizes and on the properties of the objective function. Numerical results also show a better performance of our mini-batch scheme over its single-batch counterpart.

A Stochastic Subgradient Method for Distributionally Robust Non-convex and Non-smooth Learning

A Stochastic Subgradient Method for Distributionally Robust Non-convex and Non-smooth Learning

Adaptive primal-dual stochastic gradient method for expectation-constrained convex stochastic programs

Stochastic gradient methods (SGMs) have been widely used for solving stochastic optimization problems. A majority of existing works assume no constraints or easy-to-project constraints. In this paper, we consider convex stochastic optimization problems with expectation constraints. For these problems, it is often extremely expensive to perform projection onto the feasible set. Several SGMs in the literature can be applied to solve the expectation-constrained stochastic problems. We propose a novel primal-dual type SGM based on the Lagrangian function. Different from existing methods, our method incorporates an adaptiveness technique to speed up convergence. At each iteration, our method inquires an unbiased stochastic subgradient of the Lagrangian function, and then it renews the primal variables by an adaptive-SGM update and the dual variables by a vanilla-SGM update. We show that the proposed method has a convergence rate of $$O(1/\sqrt{k})$$ in terms of the objective error and the constraint violation. Although the convergence rate is the same as those of existing SGMs, we observe its significantly faster convergence than an existing non-adaptive primal-dual SGM and a primal SGM on solving the Neyman–Pearson classification and quadratically constrained quadratic programs. Furthermore, we modify the proposed method to solve convex–concave stochastic minimax problems, for which we perform adaptive-SGM updates to both primal and dual variables. A convergence rate of $$O(1/\sqrt{k})$$ is also established to the modified method for solving minimax problems in terms of primal-dual gap. Our code has been released at https://github.com/RPI-OPT/APriD .

Stochastic Subgradient for Large-Scale Support Vector Machine Using the Generalized Pinball Loss Function

In this paper, we propose a stochastic gradient descent algorithm, called stochastic gradient descent method-based generalized pinball support vector machine (SG-GPSVM), to solve data classification problems. This approach was developed by replacing the hinge loss function in the conventional support vector machine (SVM) with a generalized pinball loss function. We show that SG-GPSVM is convergent and that it approximates the conventional generalized pinball support vector machine (GPSVM). Further, the symmetric kernel method was adopted to evaluate the performance of SG-GPSVM as a nonlinear classifier. Our suggested algorithm surpasses existing methods in terms of noise insensitivity, resampling stability, and accuracy for large-scale data scenarios, according to the experimental results.

First-Order Newton-Type Estimator for Distributed Estimation and Inference

This article studies distributed estimation and inference for a general statistical problem with a convex loss that could be nondifferentiable. For the purpose of efficient computation, we restrict ourselves to stochastic first-order optimization, which enjoys low per-iteration complexity. To motivate the proposed method, we first investigate the theoretical properties of a straightforward divide-and-conquer stochastic gradient descent approach. Our theory shows that there is a restriction on the number of machines and this restriction becomes more stringent when the dimension p is large. To overcome this limitation, this article proposes a new multi-round distributed estimation procedure that approximates the Newton step only using stochastic subgradient. The key component in our method is the proposal of a computationally efficient estimator of , where is the population Hessian matrix and w is any given vector. Instead of estimating (or ) that usually requires the second-order differentiability of the loss, the proposed first-order Newton-type estimator (FONE) directly estimates the vector of interest as a whole and is applicable to nondifferentiable losses. Our estimator also facilitates the inference for the empirical risk minimizer. It turns out that the key term in the limiting covariance has the form of , which can be estimated by FONE.

Inexact stochastic subgradient projection method for stochastic equilibrium problems with nonmonotone bifunctions: application to expected risk minimization in machine learning

This paper discusses a stochastic equilibrium problem for which the function is in the form of the expectation of nonmonotone bifunctions and the constraint set is closed and convex. This problem includes various applications such as stochastic variational inequalities, stochastic Nash equilibrium problems, and nonconvex stochastic optimization problems. For solving this stochastic equilibrium problem, we propose an inexact stochastic subgradient projection method. The proposed method sets a random realization of the bifunction and then updates its approximation by using both its stochastic subgradient and the projection onto the constraint set. The main contribution of this paper is to present a convergence analysis showing that, under certain assumptions, any accumulation point of the sequence generated by the proposed method using a constant step size almost surely belongs to the solution set of the stochastic equilibrium problem. A convergence rate analysis of the method is also provided to illustrate the method’s efficiency. Another contribution of this paper is to show that a machine learning algorithm based on the proposed method achieves the expected risk minimization for a class of least absolute selection and shrinkage operator (lasso) problems in statistical learning with sparsity. Numerical comparisons of the proposed machine learning algorithm with existing machine learning algorithms for the expected risk minimization using LIBSVM datasets demonstrate the effectiveness and superior classification accuracy of the proposed algorithm.

The pattern recognition of multisource partial discharge in transformers based on parallel feature domain

The traditional diagnosis methods for the multisource partial discharge in transformer have several problems, such as the poor effect of signal separation, the inaccuracy of the extracted signal features and the over fitting of the neural network algorithm used for pattern recognition. In order to solve the above problems, the improved algorithm of stacked auto-encoder is adopted to extract the adaptive features of partial discharge one dimensional defect signals in time domain and two dimensional defect signal in time frequency domain, and a parallel feature space combining the features in time domain with that the time frequency domain is constructed. For the first time, the transformed L1 norm is utilized as the regular term of the loss function which is solved by the proximally guided stochastic subgradient in the improved auto-encoder. The trained neural network model can be utilized for pattern recognition of the different multisource partial discharge signals. The results show that the proposed method has a higher recognition rate for the different multisource partial discharge defects than that in the traditional method, and alleviates the problem of over fitting to a certain extent.

Primal-Dual Stochastic Gradient Method for Convex Programs with Many Functional Constraints

Stochastic gradient method (SGM) has been popularly applied to solve optimization problems with objective that is stochastic or an average of many functions. Most existing works on SGMs assume that the underlying problem is unconstrained or has an easy-to-project constraint set. In this paper, we consider problems that have a stochastic objective and also many functional constraints. For such problems, it could be extremely expensive to project a point to the feasible set, or even compute subgradient and/or function value of all constraint functions. To find solutions of these problems, we propose a novel (adaptive) SGM based on the classical augmented Lagrangian function. Within every iteration, it inquires a stochastic subgradient of the objective, and a subgradient and the function value of one randomly sampled constraint function. Hence, the per-iteration complexity is low. We establish its convergence rate for convex problems and also problems with strongly convex objective. It can achieve the optimal $O(1/\sqrt{k})$ convergence rate for convex case and nearly optimal $O\big((\log k)/k\big)$ rate for strongly convex case. Numerical experiments on a sample approximation problem of the robust portfolio selection and quadratically constrained quadratic programming are conducted to demonstrate its efficiency.

A new convergent hybrid learning algorithm for two-stage stochastic programs

This study proposes a new hybrid learning algorithm to approximate the expected recourse function for two-stage stochastic programs. The proposed algorithm, which is called projected stochastic hybrid learning algorithm, is a hybrid of piecewise linear approximation and stochastic subgradient methods. Piecewise linear approximations are updated adaptively by using stochastic subgradient and sample information on the objective function itself. In order to achieve a global optimum, a projection step that implements the stochastic subgradient method is performed to jump out from a local optimum. For general two-stage stochastic programs, we prove the convergence of the algorithm. Furthermore, the algorithm can drop the projection steps for two-stage stochastic programs with network recourse. Therefore, the pure piecewise linear approximation method is convergent when the initial piecewise linear functions are properly constructed. Computational results indicate that the algorithm exhibits rapid convergence.

Stochastic (Approximate) Proximal Point Methods: Convergence, Optimality, and Adaptivity

We develop model-based methods for solving stochastic convex optimization problems, introducing the approximate-proximal point, or aProx, family, which includes stochastic subgradient, proximal point, and bundle methods. When the modeling approaches we propose are appropriately accurate, the methods enjoy stronger convergence and robustness guarantees than classical approaches, even though the model-based methods typically add little to no computational overhead over stochastic subgradient methods. For example, we show that improved models converge with probability 1 and enjoy optimal asymptotic normality results under weak assumptions; these methods are also adaptive to a natural class of what we term easy optimization problems, achieving linear convergence under appropriate strong growth conditions on the objective. Our substantial experimental investigation shows the advantages of more accurate modeling over standard subgradient methods across many smooth and non-smooth optimization problems.

Distributed Stochastic Subgradient Projection Algorithms Based on Weight‐Balancing over Time‐Varying Directed Graphs

We consider a distributed constrained optimization problem over graphs, where cost function of each agent is private. Moreover, we assume that the graphs are time‐varying and directed. In order to address such problem, a fully decentralized stochastic subgradient projection algorithm is proposed over time‐varying directed graphs. However, since the graphs are directed, the weight matrix may not be a doubly stochastic matrix. Therefore, we overcome this difficulty by using weight‐balancing technique. By choosing appropriate step‐sizes, we show that iterations of all agents asymptotically converge to some optimal solutions. Further, by our analysis, convergence rate of our proposed algorithm is O(ln Γ/Γ) under local strong convexity, where Γ is the number of iterations. In addition, under local convexity, we prove that our proposed algorithm can converge with rate . In addition, we verify the theoretical results through simulations.

Convergence Rates for Deterministic and Stochastic Subgradient Methods without Lipschitz Continuity

We extend the classic convergence rate theory for subgradient methods to apply to non-Lipschitz functions. For the deterministic projected subgradient method, we present a global $O(1/\sqrt{T})$ convergence rate for any convex function which is locally Lipschitz around its minimizers. This approach is based on Shor's classic subgradient analysis and implies generalizations of the standard convergence rates for gradient descent on functions with Lipschitz or Hölder continuous gradients. Further, we show a $O(1/\sqrt{T})$ convergence rate for the stochastic projected subgradient method on convex functions with at most quadratic growth, which improves to $O(1/T)$ under either strong convexity or a weaker quadratic lower bound condition.

String-averaging incremental stochastic subgradient algorithms

ABSTRACTWe present a method to solve constrained convex stochastic optimization problems when the objective is a finite sum of convex functions . Our method is based on Incremental Stochastic Subgradient Algorithms and String-Averaging techniques, with an assumption that the subgradient directions are affected by random errors in each iteration. Our analysis allows the method to perform approximate projections onto the feasible set in each iteration. We provide convergence results for the case where a diminishing step-size rule is used. We test our method in a large set of random instances of a stochastic convex programming problem and we compare its performance with the robust mirror descent stochastic approximation algorithm proposed in Nemirovski et al. (Robust stochastic approximation approach to stochastic programming, SIAM J Optim 19 (2009), pp. 15741609).

Optimal distributed stochastic mirror descent for strongly convex optimization

In this paper we consider convergence rate problems for stochastic strongly-convex optimization in the non-Euclidean sense with a constraint set over a time-varying multi-agent network. We propose two efficient non-Euclidean stochastic subgradient descent algorithms based on the Bregman divergence as distance-measuring function rather than the Euclidean distances that were employed by the standard distributed stochastic projected subgradient algorithms. For distributed optimization of non-smooth and strongly convex functions whose only stochastic subgradients are available, the first algorithm recovers the best previous known rate of O(ln(T)∕T) (where T is the total number of iterations). The second algorithm is an epoch variant of the first algorithm that attains the optimal convergence rate of O(1∕T), matching that of the best previously known centralized stochastic subgradient algorithm. Finally, we report some simulation results to illustrate the proposed algorithms.

一种非光滑强凸函数的随机次梯度镜面下降算法

一种非光滑强凸函数的随机次梯度镜面下降算法

Parallel Algorithms and Probability of Large Deviation for Stochastic Convex Optimization Problems

We consider convex stochastic optimization problems under different assumptions on the properties of available stochastic subgradient. It is known that, if the value of the objective function is available, one can obtain, in parallel, several independent approximate solutions in terms of the objective residual expectation. Then, choosing the solution with the minimum function value, one can control the probability of large deviation of the objective residual. On the contrary, in this short paper, we address the situation, when the value of the objective function is unavailable or is too expensive to calculate. Under "‘light-tail"’ assumption for stochastic subgradient and in general case with moderate large deviation probability, we show that parallelization combined with averaging gives bounds for probability of large deviation similar to a serial method. Thus, in these cases, one can benefit from parallel computations and reduce the computational time without loss in the solution quality.