Stochastic Approximation Scheme Research Articles

Stochastic Approximation for Estimating the Price of Stability in Stochastic Nash Games

The goal in this paper is to approximate the Price of Stability (PoS) in stochastic Nash games using stochastic approximation (SA) schemes. PoS is amongst the most popular metrics in game theory and provides an avenue for estimating the efficiency of Nash games. In particular, knowing the value of PoS can help with designing efficient networked systems, including transportation networks and power market mechanisms. Motivated by the absence of efficient methods for computing the PoS, first we consider stochastic optimization problems with a nonsmooth and merely convex objective function and a merely monotone stochastic variational inequality (SVI) constraint. This problem appears in the numerator of the PoS ratio. We develop a randomized block-coordinate stochastic extra-(sub)gradient method where we employ a novel iterative penalization scheme to account for the mapping of the SVI in each of the two gradient updates of the algorithm. We obtain an iteration complexity of the order ϵ − 4 that appears to be best known result for this class of constrained stochastic optimization problems, where ϵ denotes an arbitrary bound on suitably defined infeasibility and suboptimality metrics. Second, we develop an SA-based scheme for approximating the PoS and derive lower and upper bounds on the approximation error. To validate the theoretical findings, we provide preliminary simulation results on a networked stochastic Nash Cournot competition.

Stochastic approximation based confidence regions for stochastic variational inequalities

The sample average approximation (SAA) and the stochastic approximation (SA) are two popular schemes for solving the stochastic variational inequalities problem (SVIP). In the past decades, theories on the consistency of the SAA solutions and SA solutions have been well studied. More recently, the asymptotic confidence regions of the true solution to SVIP have been constructed when the SAA scheme is implemented. It is of fundamental interest to develop confidence regions of the true solution to the SVIP when the SA scheme is employed. In this paper, we discuss the framework of constructing asymptotic confidence regions for the true solution of SVIP with a focus on stochastic dual average method. We first establish the asymptotic normality of the SA solutions both in ergodic sense and non-ergodic sense. Then the online methods of estimating the covariance matrices in the normal distributions are studied. Finally, practical procedures of building the asymptotic confidence regions of solutions to SVIP with numerical simulations are presented.

Low-Latency Collaborative Predictive Maintenance: Over-the-Air Federated Learning in Noisy Industrial Environments.

The emergence of Industry 4.0 has revolutionized the industrial sector, enabling the development of compact, precise, and interconnected assets. This transformation has not only generated vast amounts of data but also facilitated the migration of learning and optimization processes to edge devices. Consequently, modern industries can effectively leverage this paradigm through distributed learning to define product quality and implement predictive maintenance (PM) strategies. While computing speeds continue to advance rapidly, the latency in communication has emerged as a bottleneck for fast edge learning, particularly in time-sensitive applications such as PM. To address this issue, we explore Federated Learning (FL), a privacy-preserving framework. FL entails updating a global AI model on a parameter server (PS) through aggregation of locally trained models from edge devices. We propose an innovative approach: analog aggregation over-the-air of updates transmitted concurrently over wireless channels. This leverages the waveform-superposition property in multi-access channels, significantly reducing communication latency compared to conventional methods. However, it is vulnerable to performance degradation due to channel properties like noise and fading. In this study, we introduce a method to mitigate the impact of channel noise in FL over-the-air communication and computation (FLOACC). We integrate a novel tracking-based stochastic approximation scheme into a standard federated stochastic variance reduced gradient (FSVRG). This effectively averages out channel noise's influence, ensuring robust FLOACC performance without increasing transmission power gain. Numerical results confirm our approach's superior communication efficiency and scalability in various FL scenarios, especially when dealing with noisy channels. Simulation experiments also highlight significant enhancements in prediction accuracy and loss function reduction for analog aggregation in over-the-air FL scenarios.

Optimal Oracle Inequalities for Projected Fixed-Point Equations, with Applications to Policy Evaluation

Linear fixed-point equations in Hilbert spaces arise in a variety of settings, including reinforcement learning, and computational methods for solving differential and integral equations. We study methods that use a collection of random observations to compute approximate solutions by searching over a known low-dimensional subspace of the Hilbert space. First, we prove an instance-dependent upper bound on the mean-squared error for a linear stochastic approximation scheme that exploits Polyak–Ruppert averaging. This bound consists of two terms: an approximation error term with an instance-dependent approximation factor and a statistical error term that captures the instance-specific complexity of the noise when projected onto the low-dimensional subspace. Using information-theoretic methods, we also establish lower bounds showing that both of these terms cannot be improved, again in an instance-dependent sense. A concrete consequence of our characterization is that the optimal approximation factor in this problem can be much larger than a universal constant. We show how our results precisely characterize the error of a class of temporal difference learning methods for the policy evaluation problem with linear function approximation, establishing their optimality. Funding: This work was partially supported by grants from the Office of Naval Research [Grant DOD-ONRN00014-18-1-2640] and the National Science Foundation (NSF) [NSF-IIS Grant 1909365, NSF-DMS Grant 2015454, and NSF-CCF Grant 1955450] to M. J. Wainwright. Part of this work was performed when A. Pananjady was visiting the Simons Institute for the Theory of Computing, where he was supported by a Swiss Re Research Fellowship. Supplemental Material: The online supplementary file is available at https://doi.org/10.1287/moor.2022.1341 .

Probability maximization via Minkowski functionals: convex representations and tractable resolution

In this paper, we consider the maximizing of the probability $${\mathbb {P}}\left\{ \, \zeta \, \mid \, \zeta \, \in \, {\mathbf {K}}({\mathbf {x}}) \, \right\} $$ over a closed and convex set $${\mathcal {X}}$$ , a special case of the chance-constrained optimization problem. Suppose $${\mathbf {K}}({\mathbf {x}}) \, \triangleq \, \left\{ \, \zeta \, \in \, {\mathcal {K}}\, \mid \, c({\mathbf {x}},\zeta ) \, \ge \, 0 \right\} $$ , and $$\zeta $$ is uniformly distributed on a convex and compact set $${\mathcal {K}}$$ and $$c({\mathbf {x}},\zeta )$$ is defined as either $$c({\mathbf {x}},\zeta )\, \triangleq \, 1-\left| \zeta ^T{\mathbf {x}}\right| ^m$$ where $$m\ge 0$$ (Setting A) or $$c({\mathbf {x}},\zeta ) \, \triangleq \, T{\mathbf {x}}\, - \, \zeta $$ (Setting B). We show that in either setting, by leveraging recent findings in the context of non-Gaussian integrals of positively homogenous functions, $${\mathbb {P}}\left\{ \,\zeta \, \mid \, \zeta \, \in \, {\mathbf {K}}({\mathbf {x}}) \, \right\} $$ can be expressed as the expectation of a suitably defined continuous function $$F(\bullet ,\xi )$$ with respect to an appropriately defined Gaussian density (or its variant), i.e. $${\mathbb {E}}_{{{\tilde{p}}}} \left[ \, F({\mathbf {x}},\xi )\, \right] $$ . Aided by a recent observation in convex analysis, we then develop a convex representation of the original problem requiring the minimization of $$g\left( {\mathbb {E}}\left[ \, F(\bullet ,\xi )\, \right] \right) $$ over $${\mathcal {X}}$$ , where g is an appropriately defined smooth convex function. Traditional stochastic approximation schemes cannot contend with the minimization of $$g\left( {\mathbb {E}}\left[ F(\bullet ,\xi )\right] \right) $$ over $$\mathcal X$$ , since conditionally unbiased sampled gradients are unavailable. We then develop a regularized variance-reduced stochastic approximation (r-VRSA) scheme that obviates the need for such unbiasedness by combining iterative regularization with variance-reduction. Notably, (r-VRSA) is characterized by almost-sure convergence guarantees, a convergence rate of $$\mathcal {O}(1/k^{1/2-a})$$ in expected sub-optimality where $$a > 0$$ , and a sample complexity of $$\mathcal {O}(1/\epsilon ^{6+\delta })$$ where $$\delta > 0$$ . To the best of our knowledge, this may be the first such scheme for probability maximization problems with convergence and rate guarantees. Preliminary numerics on a portfolio selection problem (Setting A) and a set-covering problem (Setting B) suggest that the scheme competes well with naive mini-batch SA schemes as well as integer programming approximation methods.

Stochastic generalized Nash equilibrium seeking under partial-decision information

We consider for the first time a stochastic generalized Nash equilibrium problem, i.e., with expected-value cost functions and joint feasibility constraints, under partial-decision information, meaning that the agents communicate only with some trusted neighbors. We propose several distributed algorithms for network games and aggregative games that we show being special instances of a preconditioned forward–backward splitting method. We prove that the algorithms converge to a generalized Nash equilibrium when the forward operator is restricted cocoercive by using the stochastic approximation scheme with variance reduction to estimate the expected value of the pseudogradient.

A concentration bound for contractive stochastic approximation

We derive a ‘high probability’ concentration bound for stochastic approximation schemes for finding the fixed point of a contraction map, and illustrate its applications in reinforcement learning for approximate dynamic programming.

Efficient stochastic optimisation by unadjusted Langevin Monte Carlo

Stochastic approximation methods play a central role in maximum likelihood estimation problems involving intractable likelihood functions, such as marginal likelihoods arising in problems with missing or incomplete data, and in parametric empirical Bayesian estimation. Combined with Markov chain Monte Carlo algorithms, these stochastic optimisation methods have been successfully applied to a wide range of problems in science and industry. However, this strategy scales poorly to large problems because of methodological and theoretical difficulties related to using high-dimensional Markov chain Monte Carlo algorithms within a stochastic approximation scheme. This paper proposes to address these difficulties by using unadjusted Langevin algorithms to construct the stochastic approximation. This leads to a highly efficient stochastic optimisation methodology with favourable convergence properties that can be quantified explicitly and easily checked. The proposed methodology is demonstrated with three experiments, including a challenging application to statistical audio analysis and a sparse Bayesian logistic regression with random effects problem.

General multilevel adaptations for stochastic approximation algorithms II: CLTs

In this article we establish central limit theorems for multilevel Polyak–Ruppert averaged stochastic approximation schemes. We work under very mild technical assumptions and consider the slow regime in which typical errors decay like N−δ with δ∈(0,12) and the critical regime in which errors decay of order N−1∕2logN in the runtime N of the algorithm.

Stochastic Recursive Inclusions in Two Timescales with Nonadditive Iterate-Dependent Markov Noise

In this paper, we study the asymptotic behavior of a stochastic approximation scheme on two timescales with set-valued drift functions and in the presence of nonadditive iterate-dependent Markov noise. We show that the recursion on each timescale tracks the flow of a differential inclusion obtained by averaging the set-valued drift function in the recursion with respect to a set of measures accounting for both averaging with respect to the stationary distributions of the Markov noise terms and the interdependence between the two recursions on different timescales. The framework studied in this paper builds on a recent work by Ramaswamy and Bhatnagar, by allowing for the presence of nonadditive iterate-dependent Markov noise. As an application, we consider the problem of computing the optimum in a constrained convex optimization problem, where the objective function and the constraints are averaged with respect to the stationary distribution of an underlying Markov chain. Further, the proposed scheme neither requires the differentiability of the objective function nor the knowledge of the averaging measure.

Analysis of Stochastic Approximation Schemes With Set-Valued Maps in the Absence of a Stability Guarantee and Their Stabilization

In this paper, we analyze the behavior of stochastic approximation schemes with set-valued maps in the absence of a stability guarantee. We prove that after a large number of iterations, if the stochastic approximation process enters the domain of attraction of an attracting set, it gets locked into the attracting set with high probability. We demonstrate that the above-mentioned result is an effective instrument for analyzing stochastic approximation schemes in the absence of a stability guarantee, by using it to obtain an alternate criterion for convergence in the presence of a locally attracting set for the mean field and by using it to show that a feedback mechanism, which involves resetting the iterates at regular time intervals, stabilizes the scheme when the mean field possesses a globally attracting set, thereby guaranteeing convergence. The results in this paper build on the works of Borkar, Andrieu et al., and Chen et al. , by allowing for the presence of set-valued drift functions.

High-Dimensional Nonconvex Stochastic Optimization by Doubly Stochastic Successive Convex Approximation

In this paper, we consider supervised learning problems over training sets in which the number of training examples and the dimension of feature vectors are both large. We focus on the case where the loss function defining the quality of the parameter we wish to estimate may be non-convex, but also has a convex regularization. We propose a Doubly Stochastic Successive Convex approximation scheme (DSSC) able to handle non-convex regularized expected risk minimization. The method operates by decomposing the decision variable into blocks and operating on random subsets of blocks at each step (fusing the merits of stochastic approximation with block coordinate methods), and then implements successive convex approximation. In contrast to many stochastic convex methods whose almost sure behavior is not guaranteed in non-convex settings, DSSC attains almost sure convergence to a stationary solution of the problem. Moreover, we show that the proposed DSSC algorithm achieves stationarity at a rate of O((log t)/t <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/4</sup> ). Numerical experiments on a non-convex variant of a lasso regression problem show that DSSC performs favorably in this setting. We then apply this method to the task of dictionary learning from high-dimensional visual data collected from a ground robot, and observe reliable convergence behavior for a difficult non-convex stochastic program.

A robust structural parameter estimation method using seismic response measurements

Although successful applications of data assimilation techniques such as Kalman filter to structural parameter identification problems exist in literature, they have been explored mostly via simulations because real-world application poses challenges. Current practice is to set the process and measurement noise covariance matrices by manual tuning and keeping them constant over the estimation period. However, when the matrices are not correct, Kalman filters are prone to suboptimal performance and/ or even divergence. In this paper, use of a stochastic approximation scheme along with the unscented Kalman filter is explored. The sensitivity of the initial filter parameter selection has been investigated through both simulations and full-scale laboratory experiments. Results show that by automatically updating the filter statistics, system states, including the structural parameters, are successfully estimated with a fast convergence rate, good accuracy, and trackability in a stable and objective manner. The method is further applied to in situ seismic response records of a box-girder bridge. The stiffness values of the rubber bearings are successfully identified.

SI-ADMM: A Stochastic Inexact ADMM Framework for Stochastic Convex Programs

We consider the structured stochastic convex program requiring the minimization of $\mathbb{E}[\tilde f(x,\xi)]+\mathbb{E}[\tilde g(y,\xi)]$ subject to the constraint $Ax + By = b$. Motivated by the need for decentralized schemes and structure, we propose a stochastic inexact ADMM (SI-ADMM) framework where subproblems are solved inexactly via stochastic approximation schemes. Based on this framework, we prove the following: (i) under suitable assumptions on the associated batch-size of samples utilized at each iteration, the SI-ADMM scheme produces a sequence that converges to the unique solution almost surely; (ii) If the number of gradient steps (or equivalently, the number of sampled gradients) utilized for solving the subproblems in each iteration increases at a geometric rate, the mean-squared error diminishes to zero at a prescribed geometric rate; (iii) The overall iteration complexity in terms of gradient steps (or equivalently samples) is found to be consistent with the canonical level of $\mathcal{O}(1/\epsilon)$. Preliminary applications on LASSO and distributed regression suggest that the scheme performs well compared to its competitors.

On Synchronous, Asynchronous, and Randomized Best-Response Schemes for Stochastic Nash Games

In this paper, we consider a stochastic Nash game in which each player minimizes a parameterized expectation-valued convex objective function. In deterministic regimes, proximal best-response (BR) schemes have been shown to be convergent under a suitable spectral property associated with the proximal BR map. However, a direct application of this scheme to stochastic settings requires obtaining exact solutions to stochastic optimization problems at each iteration. Instead, we propose an inexact generalization of this scheme in which an inexact solution to the BR problem is computed in an expected-value sense via a stochastic approximation (SA) scheme. On the basis of this framework, we present three inexact BR schemes: (i) First, we propose a synchronous inexact BR scheme where all players simultaneously update their strategies. (ii) Second, we extend this to a randomized setting where a subset of players is randomly chosen to update their strategies while the other players keep their strategies invariant. (iii) Third, we propose an asynchronous scheme, where each player chooses its update frequency while using outdated rival-specific data in updating its strategy. Under a suitable contractive property on the proximal BR map, we proceed to derive almost sure convergence of the iterates to the Nash equilibrium (NE) for (i) and (ii) and mean convergence for (i)–(iii). In addition, we show that for (i)–(iii), the generated iterates converge to the unique equilibrium in mean at a linear rate with a prescribed constant rather than a sublinear rate. Finally, we establish the overall iteration complexity of the scheme in terms of projected stochastic gradient (SG) steps for computing an [Formula: see text]-NE2 (or [Formula: see text]-NE∞) and note that in all settings, the iteration complexity is [Formula: see text], where [Formula: see text] in the context of (i), and c &gt; 0 represents the positive cost of randomization in (ii) and asynchronicity and delay in (iii). Notably, in the synchronous regime, we achieve a near-optimal rate from the standpoint of solving stochastic convex optimization problems by SA schemes. The schemes are further extended to settings where players solve two-stage stochastic Nash games with linear and quadratic recourse. Finally, preliminary numerics developed on a multiportfolio investment problem and a two-stage capacity expansion game support the rate and complexity statements.

Optimal stochastic extragradient schemes for pseudomonotone stochastic variational inequality problems and their variants

We consider the stochastic variational inequality problem in which the map is expectation-valued in a component-wise sense. Much of the available convergence theory and rate statements for stochastic approximation schemes are limited to monotone maps. However, non-monotone stochastic variational inequality problems are not uncommon and are seen to arise from product pricing, fractional optimization problems, and subclasses of economic equilibrium problems. Motivated by the need to address a broader class of maps, we make the following contributions: (i) We present an extragradient-based stochastic approximation scheme and prove that the iterates converge to a solution of the original problem under either pseudomonotonicity requirements or a suitably defined acute angle condition. Such statements are shown to be generalizable to the stochastic mirror-prox framework; (ii) Under strong pseudomonotonicity, we show that the mean-squared error in the solution iterates produced by the extragradient SA scheme converges at the optimal rate of O(1/k) statements that were hitherto unavailable K in this regime. Notably, we optimize the initial steplength by obtaining an {\epsilon}-infimum of a discontinuous nonconvex function. Similar statements are derived for mirror-prox generalizations and can accommodate monotone SVIs under a weak-sharpness requirement. Finally, both the asymptotics and the empirical rates of the schemes are studied on a set of variational problems where it is seen that the theoretically specified initial steplength leads to significant performance benefits.

Stochastic Approximation on Riemannian Manifolds

The standard theory of stochastic approximation (SA) is extended to the case when the constraint set is a Riemannian manifold. Specifically, the standard ODE method for analyzing SA schemes is extended to iterations constrained to stay on a manifold using a retraction mapping. In addition, for submanifolds of a Euclidean space, a framework is developed for a projected SA scheme with approximate retractions. The framework is also extended to non-differentiable constraint sets.

Two-stage stochastic approximation for dynamic rebalancing of shared mobility systems

Mobility systems featuring shared vehicles are often unable to serve all potential customers, as the distribution of demand does not coincide with the positions of vehicles at any given time. System operators often choose to reposition these shared vehicles (such as bikes, cars, or scooters) actively during the course of the day to improve service rate. They face a complex dynamic optimization problem in which many integer-valued decisions must be made, using real-time state and forecast information, and within the tight computation time constraints inherent to real-time decision-making. We first present a novel nested-flow formulation of the problem, and demonstrate that its linear relaxation is significantly tighter than one from existing literature. We then adapt a two-stage stochastic approximation scheme from the generic SPAR algorithm due to Powell et al., in which rebalancing plans are optimized against a value function representing the expected cost (in terms of fulfilled and unfulfilled customer demand) of the future evolution of the system. The true value function is approximated by a separable function of contributions due to the rebalancing actions carried out at each station and each time step of the planning horizon. The new algorithm requires surprisingly few iterations to yield high-quality solutions, and is suited to real-time use as it can be terminated early if required. We provide insight into this good performance by examining the mathematical properties of our new flow formulation, and perform rigorous tests on standardized benchmark networks to explore the effect of system size. We then use data from Philadelphia’s public bike sharing scheme to demonstrate that the approach also yields performance gains for real systems.

Stochastic approximation schemes for economic capital and risk margin computations

We consider the problem of the numerical computation of its economic capital by an insurance or a bank, in the form of a value-at-risk or expected shortfall of its loss over a given time horizon. This loss includes the appreciation of the mark-to-model of the liabilities of the firm, which we account for by nested Monte Carlo à la Gordy and Juneja [17] or by regression à la Broadie, Du, and Moallemi [10]. Using a stochastic approximation point of view on value-at-risk and expected shortfall, we establish the convergence of the resulting economic capital simulation schemes, under mild assumptions that only bear on the theoretical limiting problem at hand, as opposed to assumptions on the approximating problems in [17] and [10]. Our economic capital estimates can then be made conditional in a Markov framework and integrated in an outer Monte Carlo simulation to yield the risk margin of the firm, corresponding to a market value margin (MVM) in insurance or to a capital valuation adjustment (KVA) in banking parlance. This is illustrated numerically by a KVA case study implemented on GPUs.

On fixed gain recursive estimators with discontinuity in the parameters

In this paper, we estimate the expected tracking error of a fixed gain stochastic approximation scheme. The underlying process is not assumed Markovian, a mixing condition is required instead. Furthermore, the updating function may be discontinuous in the parameter.