Stochastic Approximation Type Research Articles

Adaptive Importance Sampling for Efficient Stochastic Root Finding and Quantile Estimation

Stochastic root-finding problems are fundamental in the fields of operations research and data science. However, when the root-finding problem involves rare events, crude Monte Carlo can be prohibitively inefficient. Importance sampling (IS) is a commonly used approach, but selecting a good IS parameter requires knowledge of the problem’s solution, which creates a circular challenge. In “Adaptive Importance Sampling for Efficient Stochastic Root Finding and Quantile Estimation,” He, Jiang, Lam, and Fu propose an adaptive IS approach to untie this circularity. The adaptive IS simultaneously estimates the root and the IS parameters, and can be embedded in sample average approximation–type algorithms and stochastic approximation–type algorithms. They provide theoretical analysis on strong consistency and asymptotic normality of the resulting estimators, and show the benefit of adaptivity from a worst-case perspective. They also provide specialized analyses on extreme quantile estimation under milder conditions.

On stochastic accelerated gradient with convergence rate

Abstract This article studies the regression learning problem from given sample data by using stochastic approximation (SA) type algorithm, namely, the accelerated SA. We focus on problems without strong convexity, for which all well-known algorithms achieve a convergence rate for function values of O ( 1 / n ) O\left(1\hspace{0.1em}\text{/}\hspace{0.1em}n) . We consider and analyze accelerated SA algorithm that achieves a rate of O ( 1 / n ) O\left(1\hspace{0.1em}\text{/}\hspace{0.1em}n) for classical least-square regression and logistic regression problems, respectively. Comparing with the well-known results, we only need fewer conditions to obtain the tight convergence rate for least-square regression and logistic regression problems.

A Stochastic Approximation Method for Simulation-Based Quantile Optimization

We present a gradient-based algorithm for solving a class of simulation optimization problems in which the objective function is the quantile of a simulation output random variable. In contrast with existing quantile (quantile derivative) estimation techniques, which aim to eliminate the estimator bias by gradually increasing the simulation sample size, our algorithm incorporates a novel recursive procedure that only requires a single simulation sample at each step to simultaneously obtain quantile and quantile derivative estimators that are asymptotically unbiased. We show that these estimators, when coupled with the standard gradient descent method, lead to a multitime-scale stochastic approximation type of algorithm that converges to an optimal quantile value with probability one. In our numerical experiments, the proposed algorithm is applied to optimal investment portfolio problems, resulting in new solutions that complement those obtained under the classical Markowitz mean-variance framework. History: Accepted by Alice E. Smith, Editor-in-Chief; Bruno Tuffin, Area Editor for Simulation. Funding: The work of Y. Peng was supported in part by the National Natural Science Foundation of China (NSFC) [Grants 72022001, 92146003, and 71901003], and by the Key Research and Development Programof Beijing Municipal Science and Technology Commission. Supplemental Material: The e-companion is available at https://doi.org/10.1287/ijoc.2022.1214 .

A Stochastic Approximation Method for Simulation-Based Quantile Optimization

We present a gradient-based algorithm for solving a class of simulation optimization problems in which the objective function is the quantile of a simulation output random variable. In contrast with existing quantile (quantile derivative) estimation techniques, which aim to eliminate the estimator bias by gradually increasing the simulation sample size, our algorithm incorporates a novel recursive procedure that only requires a single simulation sample at each step to simultaneously obtain quantile and quantile derivative estimators that are asymptotically unbiased. We show that these estimators, when coupled with the standard gradient descent method, lead to a multi-time-scale stochastic approximation type of algorithm that converges to an optimal quantile value with probability one. In our numerical experiments, the proposed algorithm is applied to optimal investment portfolio problems, resulting in new solutions that complement those obtained under the classical Markowitz mean-variance framework.

On stochastic gradient Langevin dynamics with dependent data streams in the logconcave case

We study the problem of sampling from a probability distribution $\pi $ on $\mathbb{R}^{d}$ which has a density w.r.t. the Lebesgue measure known up to a normalization factor $x\mapsto \mathrm{e}^{-U(x)}/\int _{\mathbb{R}^{d}}\mathrm{e}^{-U(y)}\,\mathrm{d}y$. We analyze a sampling method based on the Euler discretization of the Langevin stochastic differential equations under the assumptions that the potential $U$ is continuously differentiable, $\nabla U$ is Lipschitz, and $U$ is strongly concave. We focus on the case where the gradient of the log-density cannot be directly computed but unbiased estimates of the gradient from possibly dependent observations are available. This setting can be seen as a combination of a stochastic approximation (here stochastic gradient) type algorithms with discretized Langevin dynamics. We obtain an upper bound of the Wasserstein-2 distance between the law of the iterates of this algorithm and the target distribution $\pi $ with constants depending explicitly on the Lipschitz and strong convexity constants of the potential and the dimension of the space. Finally, under weaker assumptions on $U$ and its gradient but in the presence of independent observations, we obtain analogous results in Wasserstein-2 distance.

Distributed non‐linear robust consensus‐based sensor calibration for networked control systems

In this study a new algorithm is proposed for distributed blind sensor macro-calibration in networked control systems robust to noise. The proposed distributed algorithm for estimation of gain and offset correction parameters is of stochastic approximation type, with local non-linear transformations of residuals. Convergence of the algorithm in mean-square and with probability one to consensus is proved for a large class of non-linear transformations, network properties and communication and measurement noise characteristics. The choice of the introduced non-linear transformations in accordance with the theory of robust statistics leads to the proposal of new calibration algorithms robustified w.r.t. noise. It is demonstrated by Monte Carlo simulation that the proposed algorithms are very efficient in the presence of large outliers from the point of view of both achievement of high convergence rate and adequate values of convergence points, outperforming the existing linear algorithms.

Dynamic stochastic approximation for multi-stage stochastic optimization

In this paper, we consider multi-stage stochastic optimization problems with convex objectives and conic constraints at each stage. We present a new stochastic first-order method, namely the dynamic stochastic approximation (DSA) algorithm, for solving these types of stochastic optimization problems. We show that DSA can achieve an optimal $${{\mathcal {O}}}(1/\epsilon ^4)$$ rate of convergence in terms of the total number of required scenarios when applied to a three-stage stochastic optimization problem. We further show that this rate of convergence can be improved to $${{\mathcal {O}}}(1/\epsilon ^2)$$ when the objective function is strongly convex. We also discuss variants of DSA for solving more general multi-stage stochastic optimization problems with the number of stages $$T > 3$$ . The developed DSA algorithms only need to go through the scenario tree once in order to compute an $$\epsilon $$ -solution of the multi-stage stochastic optimization problem. As a result, the memory required by DSA only grows linearly with respect to the number of stages. To the best of our knowledge, this is the first time that stochastic approximation type methods are generalized for multi-stage stochastic optimization with $$T \ge 3$$ .

Algorithms for stochastic optimization with function or expectation constraints

This paper considers the problem of minimizing an expectation function over a closed convex set, coupled with a function or expectation constraint on either decision variables or problem parameters. We first present a new stochastic approximation (SA) type algorithm, namely the cooperative SA (CSA), to handle problems with the constraint on devision variables. We show that this algorithm exhibits the optimal $${{{\mathcal {O}}}}(1/\epsilon ^2)$$ rate of convergence, in terms of both optimality gap and constraint violation, when the objective and constraint functions are generally convex, where $$\epsilon$$ denotes the optimality gap and infeasibility. Moreover, we show that this rate of convergence can be improved to $${{{\mathcal {O}}}}(1/\epsilon )$$ if the objective and constraint functions are strongly convex. We then present a variant of CSA, namely the cooperative stochastic parameter approximation (CSPA) algorithm, to deal with the situation when the constraint is defined over problem parameters and show that it exhibits similar optimal rate of convergence to CSA. It is worth noting that CSA and CSPA are primal methods which do not require the iterations on the dual space and/or the estimation on the size of the dual variables. To the best of our knowledge, this is the first time that such optimal SA methods for solving function or expectation constrained stochastic optimization are presented in the literature.

Simulation Optimization Using Multi-Time-Scale Adaptive Random Search

We propose a random search algorithm for seeking the global optimum of an objective function in a simulation setting. The algorithm can be viewed as an extension of the MARS algorithm proposed in Hu and Hu (2011) for deterministic optimization, which iteratively finds improved solutions by modifying and sampling from a parameterized probability distribution over the solution space. However, unlike MARS and many other algorithms in this class, which are often population-based, our method only requires a single candidate solution to be generated at each iteration. This is primarily achieved through an effective use of past sampling information by means of embedding multiple nested stochastic approximation type of recursions into the algorithm. We prove the global convergence of the algorithm under general conditions and discuss two special simulation noise cases of interest, in which we show that only one simulation replication run is needed for each sampled solution. A preliminary numerical study is also carried out to illustrate the algorithm.

Containment control for distributed networks subject to multiplicative and additive noises with stochastic approximation–type protocols

SummaryThis paper is concerned with the stochastic containment control problem for distributed network with multiplicative and additive noises, in which the stochastic approximation technique is utilized. A major challenge, compared with the existing results with multiplicative and additive noises, is that the involving leaders are dynamically changing. It is shown that finite boundedness for dynamical leaders is of great importance for addressing the stochastic containment control issue. A suitable approach is proposed to handle multiplicative and additive noises based on the stochastic approximation, and some necessary/sufficient conditions are derived for mean square convergence with mild requirements. Furthermore, the convergence rates are studied for both dynamical leaders and static leaders scenarios. We conclude this paper by using two illustrative examples to verify the effectiveness of the theoretical results.

Distributed Averaging With Random Network Graphs and Noises

We consider a discrete-time distributed averaging algorithm over multi-agent networks with measurement noises and time-varying random graphs. Each agent updates its state by a weighted sum of pairwise state differences between its neighbors and itself with both additive and multiplicative measurement noises. The network structure is modeled by a sequence of time-varying random digraphs, which may be spatially and temporally dependent. By stochastic Lyapunov method and the combination of algebraic graph theory and martingale convergence theory, we obtain sufficient conditions for stochastic approximation type algorithms to achieve mean square and almost sure average consensus. We prove that all states of the agents converge to a common random variable, whose mathematical expectation is the average of initial values, in mean square and almost surely if the sequence of digraphs is conditionally balanced and uniformly conditionally jointly connected. An upper bound of the variance of the limit random variable, that is, the mean square steady-state error for stochastic average consensus is given quantitatively related to the weights, the algorithm gain and the energy level of the noises.

Adjustment of Consensus Protocol Step-Size in a Network System with Different Task Priorities via SPSA-like Algorithm under the Cost Constraints

Abstract A multi-agent network system of different computing nodes processing tasks of different priority levels is considered. Agents exchange information about their states in presence of noise in communication channels in the system with switching topology. Load balancing problem in the network for each priority level is formulated as differentiated consensuses achievement problem and solved via local voting protocol. The stochastic approximation type algorithm is used to adjust step-size of proposed protocol for each priority level. Simulation example demonstrating step-size adjustment is provided.

Recurrent Algorithms of Structural Classification Analysis for Complex Organized Information

For the structural classification analysis of complex organized information, we propose to use recurrent algorithms of stochastic approximation type. We introduce classification quality functionals that depend on non-normalized and zero moments of probability distribution functions for the probability of sample objects appearing in the classes, as well as the type of optimal classification. We propose a new classification algorithm for this type of classification quality criteria and prove a theorem about its convergence that ensures the stationary value of the corresponding functional. We show that the proposed algorithm can be used to solve a wide class of problems in structural classification analysis.

Randomized Control Strategies Under Arbitrary External Noise

This technical note deals with the identification problem for a linear dynamic plant described by an autoregressive moving average model with additive external noise (exogenous disturbance). We use an approach which is based on randomization of control and allows to make minimal assumptions about the noise: randomized test perturbations in control and the external noise must be stochastically independent. In particular, any unknown but bounded deterministic real sequence is an example of such a noise. In the case of a finite set of observations, we propose two procedures for computing data-based confidence regions for unknown parameters of the plant. They could be used in adaptive control schemes. The first procedure is of the stochastic approximation type, while the second one is developed in the general framework of “counting of leave-out sign-dominant correlation regions” (LSCR), which returns confidence regions that are guaranteed to contain the true parameters with a prescribed probability. If the number of observations increases infinitely, we propose the combined procedure for computing confidence regions which shrink to the true parameters asymptotically. The theoretical results are illustrated via a simulation example with a nonminimum-phase second-order plant.

Consensus stabilization in stochastic multi-agent systems with Markovian switching topology, noises and delay

This paper investigates the consensus stabilization problem of stochastic multi-agent systems with noises, Markovian switching topology, and communication delays. To solve the consensus problem of the group of interacting agents, it is supposed to use the stochastic approximation type algorithm with the step-size non-decreasing to zero. For stochastic approximation based consensus algorithms with switching topologies, the existing convergence analysis methods may be difficult to guarantee for switching digraphs because these techniques heavily relies on quadratic Lyapunov functions. To overcome the inherent limitations of the existing methods, we develop a new ergodicity approach for backward products of degenerating stochastic matrices via a discrete time dynamical system approach to analyze the consensus stabilization problem of stochastic multi-agent systems with noises, Markovian switching topology, and communication delays; and we attain necessary and sufficient condition of the consensus stabilization. Our approach does not require the double stochasticity condition typically assumed for the existence of a quadratic Lyapunov function, and provides an effective tool for analyzing consensus problems via the existing theory of stochastic matrices and ergodicity of backward products. Finally, numerical simulation is shown to demonstrate theoretical results.

Using Stochastic Approximation Type Algorithm for Choice of Consensus Protocol Step-Size in Changing Conditions

In the paper a multi-agent network system of different computing nodes is considered. A problem of load balancing in the network is addressed. The problem is formulated as consensus achievement problem and solved via local voting protocol. Agents exchange information about their states in presence of noise in communication channels. At certain moment network system topology changes and new step size of control protocol is chosen to meet new conditions. Step size adjustment is done by stochastic approximation type algorithm. Analytically obtained optimal step size values are given. Simulation example demonstrating step size adjustment is provided.

Continuous-time stochastic consensus: Stochastic approximation and Kalman–Bucy filtering based protocols

This paper investigates the continuous-time multi-agent consensus with stochastic communication noises. Each agent can only use its own and neighbors’ information corrupted by random noises to design its control input. To attenuate the communication noises, we consider the stochastic approximation type and the Kalman–Bucy filtering based protocols. By using the tools of stochastic analysis and algebraic theory, the asymptotic properties of these two kinds of protocols are analyzed. Firstly, for the stochastic approximation type protocol, we clarify the relationship between the convergence rate of the consensus error and a representative class of consensus gains in both mean square and probability one. Secondly, we propose Kalman–Bucy filtering based consensus protocols. Each agent uses Kalman–Bucy filters to get asymptotically unbiased estimates of neighbors’ states and the control input is designed based on the protocol with precise communication and the certainty equivalence principle. The iterated logarithm law of estimation errors is developed. It is shown that if the communication graph has a spanning tree, then the consensus error is bounded above by O(t−1) in mean square and by O(t−1/2(loglogt)1/2) almost surely. Finally, the superiority of the Kalman–Bucy filtering based protocol over the stochastic approximate type protocol is studied both theoretically and numerically.

Recursive identification of FIR systems with binary-valued outputs and communication channels

This paper investigates the recursive identification of the finite impulse response systems whose outputs are subject to both the binary-valued quantization and the communication unreliability. The discrete memoryless channel with packet losses and the additive white Gaussian noise channel with time-delay are considered, a stochastic gradient type algorithm and a stochastic approximation type algorithm are respectively proposed to estimate the unknown system parameters. Under some mild conditions, the algorithms can be proved to be convergent in the almost sure and mean square senses. Also, the convergence rates are given. Numerical examples are included to illustrate the performance of the algorithms.

Convergence Rate of Discrete-Time Stochastic Approximation Type Consensus Algorithms

In this paper, we study the convergence rate of the distributed stochastic approximation (SA) type algorithm for the discrete-time multi-agent consensus with communication noises. Basic results of algebraic graph theory and probability limit theory are used to study the closed-form solution of the consensus error. Under mild conditions on the decreasing step size and the network topology, we give upper bounds for the mean square and almost sure convergence rates of the consensus errors. Furthermore, for the case with balanced graphs, the exact convergence rate is provided for the mean square of the consensus error.

Recursive algorithms for parameter estimation with adaptive quantizer

This paper studies a parameter estimation problem of networked linear systems with fixed-rate quantization. Under the minimum mean square error criterion, we propose a recursive estimator of stochastic approximation type, and derive a necessary and sufficient condition for its asymptotic unbiasedness. This motivates to design an adaptive quantizer for the estimator whose strong consistency, asymptotic unbiasedness, and asymptotic normality are rigorously proved. Using the Newton-based and averaging techniques, we obtain two accelerated recursive estimators with the fastest convergence speed of O(1/k), and exactly evaluate the quantization effect on the estimation accuracy. If the observation noise is Gaussian, an optimal quantizer and the accelerated estimators are co-designed to asymptotically approach the minimum Cramer–Rao lower bound. All the estimators share almost the same computational complexity as the gradient algorithms with un-quantized observations, and can be easily implemented. Finally, the theoretical results are validated by simulations.