Stochastic Approximation Theory Research Articles

Distributed entropy-regularized multi-agent reinforcement learning with policy consensus

Sample efficiency is a limiting factor for existing distributed multi-agent reinforcement learning (MARL) algorithms over networked multi-agent systems. In this paper, the sample efficiency problem is tackled by formally incorporating the entropy regularization into the distributed MARL algorithm design. Firstly, a new entropy-regularized MARL problem is formulated under the model of networked multi-agent Markov decision processes with observation-based policies and homogeneous agents, where the policy parameter sharing among the agents provably preserves the optimality. Secondly, an on-policy distributed actor–critic algorithm is proposed, where each agent shares its parameters of both the critic and actor for consensus update. Then, the convergence analysis of the proposed algorithm is provided based on the stochastic approximation theory under the assumption of linear function approximation of the critic. Furthermore, a practical off-policy version of the proposed algorithm is developed which possesses scalability, data efficiency and learning stability. Finally, the proposed distributed algorithm is compared against the solid baselines including two classic centralized training algorithms in the multi-agent particle environment, whose learning performance is empirically demonstrated through extensive simulation experiments.

A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDP

We study the risk-sensitive exponential cost Markov decision process (MDP) formulation and develop a trajectory-based gradient algorithm to find the stationary point of the cost associated with a set of parameterized policies. We derive a formula that can be used to compute the policy gradient from (state, action, cost) information collected from sample paths of the MDP for each fixed parameterized policy. Unlike the traditional average cost problem, standard stochastic approximation theory cannot be used to exploit this formula. To address the issue, we introduce a truncated and smooth version of the risk-sensitive cost and show that this new cost criterion can be used to approximate the risk-sensitive cost and its gradient uniformly under some mild assumptions. We then develop a trajectory-based gradient algorithm to minimize the smooth truncated estimation of the risk-sensitive cost and derive conditions under which a sequence of truncations can be used to solve the original, untruncated cost problem. Funding: This work was supported by the Office of Naval Research Global [Grant N0001419-1-2566], the Division of Computer and Network Systems [Grant 21-06801], the Army Research Office [Grant W911NF-19-1-0379], and the Division of Computing and Communication Foundations [Grants 17-04970 and 19-34986].

IMPLEMENTATION OF THE ADVANCED LEARNING IN THE PROCESS OF FUNDAMENTAL MATHEMATICAL TRAINING OF BACHELORS IN THE FIELD OF ELECTRONICS AND TELECOMMUNICATION

The article substantiates the expediency of the implementation of the advanced learning in the process of fundamental mathematical training of bachelors in the field of electronics and telecommunications. At the moment, the field requires mandatory (deep) knowledge of the main classical sections of mathematics, it is also important to acquaint students in higher mathematics with elements of modern mathematical theories, concepts that allow learners to better understand special courses in mathematics and relevant special disciplines. materials of mathematics, which are used in the modern mathematical models of technical developments. The aim of the article is to reveal the approaches to the concept of the advanced learning in the process of fundamental mathematical training of future bachelors in the field of electronics and telecommunications.The main methods that were implemented in the study of the problem of advanced learning were the analysis and synthesis of the scientific sources on the selected problem, observation, implementation of projects and evaluation of their results.
 It is offered (from the first semester) to introduce generalized concepts (norm, operator, etc.) in time, to acquaint with ideas of variational calculus, functional analysis, mathematical methods of research of linear equations with variable coefficients, theory of stochastic approximation, mathematical modelling. Then the future specialist will be able to comprehend the objects, which have been already developed by mathematicians but have not been used yet. Thus, a difficult topic or concept can be considered in advance in some connection with the currently studied material. For example, during the study of a function and the construction of its graph, along with the asymptote, students can be introduced to the approximation of the function, the concepts of interpolation and approximation. It is expedient to acquaint students with these problems more deeply during performance of independent tasks (consultations, the abstract, the report at conference, etc.).

Transition Based Discount Factor for Model Free Algorithms in Reinforcement Learning

Reinforcement Learning (RL) enables an agent to learn control policies for achieving its long-term goals. One key parameter of RL algorithms is a discount factor that scales down future cost in the state’s current value estimate. This study introduces and analyses a transition-based discount factor in two model-free reinforcement learning algorithms: Q-learning and SARSA, and shows their convergence using the theory of stochastic approximation for finite state and action spaces. This causes an asymmetric discounting, favouring some transitions over others, which allows (1) faster convergence than constant discount factor variant of these algorithms, which is demonstrated by experiments on the Taxi domain and MountainCar environments; (2) provides better control over the RL agents to learn risk-averse or risk-taking policy, as demonstrated in a Cliff Walking experiment.

Nonlinear Unbalanced Urn Models via Stochastic Approximation

This paper presents a link between unbalanced non-linear urn model (a two-colored urn model) and stochastic approximation theory. Findings of our study reveal a successful establishment of limit laws for the urn composition, obtained under a drawing rule reinforced by an $\mathbb {R}_{+}$ -valued concave function and a non-balanced replacement matrix.

Actor-Critic Learning Control With Regularization and Feature Selection in Policy Gradient Estimation.

Actor-critic (AC) learning control architecture has been regarded as an important framework for reinforcement learning (RL) with continuous states and actions. In order to improve learning efficiency and convergence property, previous works have been mainly devoted to solve regularization and feature learning problem in the policy evaluation. In this article, we propose a novel AC learning control method with regularization and feature selection for policy gradient estimation in the actor network. The main contribution is that l1 -regularization is used on the actor network to achieve the function of feature selection. In each iteration, policy parameters are updated by the regularized dual-averaging (RDA) technique, which solves a minimization problem that involves two terms: one is the running average of the past policy gradients and the other is the l1 -regularization term of policy parameters. Our algorithm can efficiently calculate the solution of the minimization problem, and we call the new adaptation of policy gradient RDA-policy gradient (RDA-PG). The proposed RDA-PG can learn stochastic and deterministic near-optimal policies. The convergence of the proposed algorithm is established based on the theory of two-timescale stochastic approximation. The simulation and experimental results show that RDA-PG performs feature selection successfully in the actor and learns sparse representations of the actor both in stochastic and deterministic cases. RDA-PG performs better than existing AC algorithms on standard RL benchmark problems with irrelevant features or redundant features.

Online algorithm for variance components estimation

In this study, we develop a new algorithm for online variance components estimation (Online-VCE) of geodetic data based on the batch expectation-maximization (EM) algorithm and stochastic approximation theory. The Online-VCE algorithm is then applied to the Kalman filter and least-squares method and validated using simulated kinematic precise point positioning (PPP) based on the global navigation satellite system as well as real-data PPP experiments. The Online-VCE algorithm is specifically designed to monitor and establish a stochastic model in real-time or high-rate data applications. Compared to other methods, the Online-VCE is faster and can estimate the stochastic model in real time because it does not need to store all data, but simply estimates the expected result and computes the gradient of the parameters using only one or a few observations. In future, the Online-VCE algorithm can be used to develop a real-time atmospheric stochastic model for PPP applications.

Multi-Time Scale Smoothed Functional With Nesterov’s Acceleration

Smoothed functional (SF) algorithm estimates the gradient of the stochastic optimization problem by convolution with a smoothening kernel. This process helps the algorithm to converge to a global minimum or a point close to it. We study a two-time scale SF based gradient search algorithm with Nesterov’s acceleration for stochastic optimization problems. The main contribution of our work is to prove the convergence of this algorithm using the stochastic approximation theory. We propose a novel Lyapunov function to show the associated second-order ordinary differential equations’ (o.d.e.) stability for a non-autonomous system. We compare our algorithm with other smoothed functional algorithms such as Quasi-Newton SF, Gradient SF and Jacobi Variant of Newton SF on two different optimization problems: first, on a simple stochastic function minimization problem, and second, on the problem of optimal routing in a queueing network. Additionally, we compared the algorithms on real weather data in a weather prediction task. Experimental results show that our algorithm performs significantly better than these baseline algorithms.

Decentralized Multiagent Actor‐Critic Algorithm Based on Message Diffusion

The exponential explosion of joint actions and massive data collection are two main challenges in multiagent reinforcement learning algorithms with centralized training. To overcome these problems, in this paper, we propose a model‐free and fully decentralized actor‐critic multiagent reinforcement learning algorithm based on message diffusion. To this end, the agents are assumed to be placed in a time‐varying communication network. Each agent makes limited observations regarding the global state and joint actions; therefore, it needs to obtain and share information with others over the network. In the proposed algorithm, agents hold local estimations of the global state and joint actions and update them with local observations and the messages received from neighbors. Under the hypothesis of the global value decomposition, the gradient of the global objective function to an individual agent is derived. The convergence of the proposed algorithm with linear function approximation is guaranteed according to the stochastic approximation theory. In the experiments, the proposed algorithm was applied to a passive location task multiagent environment and achieved superior performance compared to state‐of‐the‐art algorithms.

Iterative Learning Stochastic MPC with Adaptive Constraint Tightening for Building HVAC Systems

Most of the existing stochastic model predictive control (SMPC) algorithms for systems subject to random disturbance are designed offline using the distribution information of the uncertainties. In this paper, we propose an iterative learning based MPC for systems subject to time varying stochastic constraints on states. Different from those existing offline design approaches, except for the boundedness, this algorithm does not require to know the distributions or statistics such as the covariances of the uncertainties and the parameters of the controllers are adjusted online using the observations of past state trajectories. By making use of the iterative nature of the process, pointwise in time stochastic constraints are enforced so that it can handle time-varying constraints. Under some proper assumptions, this iterative procedure is shown to be equivalent to a root-searching problem and stochastic approximation theory is applied to show that the empirical average converges to the prescribed expectation in probability. The proposed algorithm is applied to an HVAC control problem to show its effectiveness.

Convergence of Stochastic Vector Quantization and Learning Vector Quantization with Bregman Divergences

Stochastic vector quantization methods have been extensively studied in supervised and unsupervised learning problems as online, data-driven, interpretable, robust, and fast to train and evaluate algorithms. Being prototype-based methods, they depend on a dissimilarity measure, which is both necessary and sufficient to belong to the family of Bregman divergences, if the mean value is used as the representative of the cluster. In this work, we investigate the convergence properties of stochastic vector quantization (VQ) and its supervised counterpart, Learning Vector Quantization (LVQ), using Bregman divergences. We employ the theory of stochastic approximation to study the conditions on the initialization and the Bregman divergence generating functions, under which, the algorithms converge to desired configurations. These results formally support the use of Bregman divergences, such as the Kullback-Leibler divergence, in vector quantization algorithms.

Technical Note—Consistency Analysis of Sequential Learning Under Approximate Bayesian Inference

“Bayesian learning works even with censored information” We often learn about a problem from information that is incomplete or censored: for example, a medical treatment may cause side effects with no indication of what the right dose should have been. Bayesian belief models are useful in such settings, but cannot be constructed using traditional methods; as a result, practitioners have developed ways of constructing them approximately. These approximations have been very successful in many application domains, yet until now have lacked theoretical support. The paper “Consistency analysis of sequential learning under approximate Bayesian inference,” by Chen and Ryzhov, links approximate Bayesian learning to stochastic approximation theory. Using this link, the authors prove – for the first time – the consistency of a suite of approximate Bayesian methods culled from the literature. One highlight is an entirely new consistency proof for Bayesian logistic regression, a well-established approximation technique that essentially treats logistic regression as if it were ordinary least squares.

Convergence of Markovian stochastic approximation for Markov random fields with hidden variables

This paper studies the convergence of the stochastic algorithm of the modified Robbins–Monro form for a Markov random field (MRF), in which some of the nodes are clamped to be observed variables while the others are hidden ones. Based on the theory of stochastic approximation, we propose proper assumptions to guarantee the Hölder regularity of both the update function and the solution of the Poisson equation. Under these assumptions, it is proved that the control parameter sequence is almost surely bounded and accordingly the algorithm converges to the stable point of the log-likelihood function with probability [Formula: see text].

Stochastic Approximation on Riemannian Manifolds

The standard theory of stochastic approximation (SA) is extended to the case when the constraint set is a Riemannian manifold. Specifically, the standard ODE method for analyzing SA schemes is extended to iterations constrained to stay on a manifold using a retraction mapping. In addition, for submanifolds of a Euclidean space, a framework is developed for a projected SA scheme with approximate retractions. The framework is also extended to non-differentiable constraint sets.

General multilevel adaptations for stochastic approximation algorithms of Robbins–Monro and Polyak–Ruppert type

In this article we present and analyse new multilevel adaptations of classical stochastic approximation algorithms for the computation of a zero of a function $$f:D \rightarrow {{\mathbb {R}}}^d$$ defined on a convex domain $$D\subset {{\mathbb {R}}}^d$$ , which is given as a parameterised family of expectations. The analysis of the error and the computational cost of our method is based on similar assumptions as used in Giles (Oper Res 56(3):607–617, 2008) for the computation of a single expectation. Additionally, we essentially only require that f satisfies a classical contraction property from stochastic approximation theory. Under these assumptions we establish error bounds in pth mean for our multilevel Robbins–Monro and Polyak–Ruppert schemes that decay in the computational time as fast as the classical error bounds for multilevel Monte Carlo approximations of single expectations known from Giles (Oper Res 56(3):607–617, 2008). Our approach is universal in the sense that having multilevel implementations for a particular application at hand it is straightforward to implement the corresponding stochastic approximation algorithm.

Convergence in Misspecified Learning Models with Endogenous Actions

We establish convergence of beliefs and actions in a class of one-dimensional learning settings in which the agent’s model is misspecified, she chooses actions endogenously, and the actions affect how she misinterprets information. The crucial assumptions of our model are that the state and action spaces are continuous, the state has a unidirectional effect on output, and the prior and noise are normal. These assumptions imply that the agent’s posterior admits a one-dimensional summary statistic, allowing us to apply tools from stochastic approximation theory to establish convergence. Applications of our framework include learning by a person who has an incorrect model of a technology she uses, is subject to confirmatory bias, conservatism, or base-rate neglect, or is overconfident about herself, learning by a representative agent who misunderstands macroeconomic outcomes, as well as learning by a firm that has an incorrect parametric model of demand.

Экономическая динамика при неоднородном адаптивном обучении: общие критерии и достаточные условия стабильности

The article extends the results of Honkapohja and Mitra (2006) and Kolyuzhnov (2011) and provides criteria and sufficient conditions for stability in a structurally heterogeneous economy under heterogeneous adaptive learning of agents. The criteria for stability under heterogeneous mixed RLS/SG learning for four classes of models – without lags and with lags of the endogenous variable and with t or t – 1 – dating of expectations – and sufficient conditions for stability for the cases of the diagonal structure of the shock process behavior or the heterogeneous RLS learning are presented in terms of the corresponding Jacobian matrices. In addition, the study presents a very useful criterion for the stability for all types of models under mixed RLS/SG learning with equal degrees of inertia for each type of learning algorithm in terms of stability of a suitably defined average economy with two agents. The derived criteria and sufficient conditions for stability are based on the results of the theory of stochastic approximation and are presented in terms of mixture of structural and learning heterogeneity, which are essential to get sufficient and necessary conditions for stability irrespective of heterogeneity in learning presented in terms of E-stability of suitably defined aggregate economies, the “same sign” conditions and the E-stability of a suitably defined average economy and its subeconomies. The fundamental nature of the approach adopted in the paper makes it possible to apply the results to a vast majority of the existing and prospective linear and linearized economic models (including estimated DSGE models) with adaptive learning of agents.

Stochastic learning in multi-agent optimization: Communication and payoff-based approaches

Game theory serves as a powerful tool for distributed optimization in multi-agent systems in different applications. In this paper we consider multi-agent systems that can be modeled by means of potential games whose potential function coincides with a global objective function to be maximized. In this approach, the agents correspond to the strategic decision makers and the optimization problem is equivalent to the problem of learning a potential function maximizer in the designed game. The paper deals with two different information settings in the system. Firstly, we consider systems, where agents have the access to the gradient of their utility functions. However, they do not possess the full information about the joint actions. Thus, to be able to move along the gradient toward a local optimum, they need to exchange the information with their neighbors by means of communication. The second setting refers to a payoff-based approach. Here, we assume that at each iteration agents can only observe their own played actions and experienced payoffs. In both cases, the paper studies unconstrained non-concave optimization with a differentiable objective function. To develop the corresponding algorithms guaranteeing convergence to a local maximum of the potential function in absence of saddle points, we utilize the idea of the well-known Robbins–Monro procedure based on the theory of stochastic approximation.

Adaptive Learning Algorithm Convergence in Passive and Reactive Environments.

Although the number of artificial neural network and machine learning architectures is growing at an exponential pace, more attention needs to be paid to theoretical guarantees of asymptotic convergence for novel, nonlinear, high-dimensional adaptive learning algorithms. When properly understood, such guarantees can guide the algorithm development and evaluation process and provide theoretical validation for a particular algorithm design. For many decades, the machine learning community has widely recognized the importance of stochastic approximation theory as a powerful tool for identifying explicit convergence conditions for adaptive learning machines. However, the verification of such conditions is challenging for multidisciplinary researchers not working in the area of stochastic approximation theory. For this reason, this letter presents a new stochastic approximation theorem for both passive and reactive learning environments with assumptions that are easily verifiable. The theorem is widely applicable to the analysis and design of important machine learning algorithms including deep learning algorithms with multiple strict local minimizers, Monte Carlo expectation-maximization algorithms, contrastive divergence learning in Markov fields, and policy gradient reinforcement learning.

Nonlinear randomized urn models: a stochastic approximation viewpoint

This paper extends the link between stochastic approximation (SA) theory and randomized urn models developed in Laruelle, Pages (2013), and their applications to clinical trials introduced in Bai, HU (1999,2005) and Bai, Hu, Shen (2002). We no longer assume that the drawing rule is uniform among the balls of the urn (which contains d colors), but can be reinforced by a function f. This is a way to model risk aversion. Firstly, by considering that f is concave or convex and by reformulating the dynamics of the urn composition as an SA algorithm with remainder, we derive the a.s. convergence and the asymptotic normality (Central Limit Theorem, CLT) of the normalized procedure by calling upon the so-called ODE and SDE methods. An in-depth analysis of the case d=2 exhibits two different behaviors: A single equilibrium point when f is concave, and when f is convex, a transition phase from a single attracting equilibrium to a system with two attracting and one repulsive equilibrium points. The last setting is solved using results on non-convergence toward noisy and noiseless ``traps in order to deduce the a.s. convergence toward one of the attracting points. Secondly, the special case of a Polya urn (when the addition rule is the identity matrix) is analyzed, still using result from SA theory about ``traps''. Finally, these results are applied to a function with regular variation and to an optimal asset allocation in Finance.