Two-timescale Stochastic Approximation Research Articles

Review of Two-timescale Stochastic Approximation Algorithms

Artificial Intelligence has gradually become an important force to drive human beings into the intelligent era, and machine learning has made great contributions to the rise and development of Artificial Intelligence. Stochastic approximation (SA) is a com-monly used optimization algorithm in machine learning, and with the complexity of practical problem scenarios, two-timescale SA have received extensive attention and research. In this paper, the basic idea and development process of SA are introduced firstly, followed by the description of several algorithmic frameworks for linear and nonlinear SA, and the specific applications of two-timescale SA in the fields of opti-mization and reinforcement learning are also introduced. Finally, the two-timescale SA is summarized and outlooked.

Blind Post-Decision State-Based Reinforcement Learning for Intelligent IoT

Recent years have witnessed a renewed interest in reinforcement learning (RL) due to the rapid growth of the Internet-of-Things (IoT) and their associated intelligent information processing and decision-making demands. As the slow learning speed is one of the major stumbling blocks of the classic RL algorithms, substantial efforts have been devoted to developing faster RL algorithms. Among them, post-decision state (PDS) learning is a prominent one, which can often improve the learning speed by orders of magnitude by exploiting the structural property of the underlying Markov decision processes (MDPs). However, conventional PDS learning requires prior information about the PDS transition probability, which may not be always available in practice. To lift this limitation, a novel blind PDS (b-PDS) learning algorithm is proposed in this work by leveraging the generic two-timescale stochastic approximation framework. By introducing an extra estimating procedure about the PDS transition probability, b-PDS learning can achieve a similar improvement of learning speed as conventional PDS learning while excluding the need for prior information. In addition, by analyzing the globally asymptotically stable equilibrium of the corresponding ordinary differential equation (o.d.e.), the convergence and optimality of b-PDS learning are established. Moreover, extensive simulation results are provided to validate the effectiveness of the proposed algorithm. Over the considered random MDPs, it has been observed that, to reach 90% of the best possible time average reward, the proposed b-PDS learning can reduce the learning time by 70% compared to Q-learning and 30% compared to Dyna.

A Two-Timescale Stochastic Algorithm Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic

This paper analyzes a two-timescale stochastic algorithm framework for bilevel optimization. Bilevel optimization is a class of problems which exhibits a two-level structure, and its goal is to minimize an outer objective function with variables which are constrained to be the optimal solution to an (inner) optimization problem. We consider the case when the inner problem is unconstrained and strongly convex, while the outer problem is constrained and has a smooth objective function. We propose a two-timescale stochastic approximation (TTSA) algorithm for tackling such a bilevel problem. In the algorithm, a stochastic gradient update with a larger step size is used for the inner problem, while a projected stochastic gradient update with a smaller step size is used for the outer problem. We analyze the convergence rates for the TTSA algorithm under various settings: when the outer problem is strongly convex (resp. weakly convex), the TTSA algorithm finds an -optimal (resp. -stationary) solution, where is the total iteration number. As an application, we show that a two-timescale natural actor-critic proximal policy optimization algorithm can be viewed as a special case of our TTSA framework. Importantly, the natural actor-critic algorithm is shown to converge at a rate of in terms of the gap in expected discounted reward compared to a global optimal policy.

Actor-Critic Learning Control With Regularization and Feature Selection in Policy Gradient Estimation.

Actor-critic (AC) learning control architecture has been regarded as an important framework for reinforcement learning (RL) with continuous states and actions. In order to improve learning efficiency and convergence property, previous works have been mainly devoted to solve regularization and feature learning problem in the policy evaluation. In this article, we propose a novel AC learning control method with regularization and feature selection for policy gradient estimation in the actor network. The main contribution is that l1 -regularization is used on the actor network to achieve the function of feature selection. In each iteration, policy parameters are updated by the regularized dual-averaging (RDA) technique, which solves a minimization problem that involves two terms: one is the running average of the past policy gradients and the other is the l1 -regularization term of policy parameters. Our algorithm can efficiently calculate the solution of the minimization problem, and we call the new adaptation of policy gradient RDA-policy gradient (RDA-PG). The proposed RDA-PG can learn stochastic and deterministic near-optimal policies. The convergence of the proposed algorithm is established based on the theory of two-timescale stochastic approximation. The simulation and experimental results show that RDA-PG performs feature selection successfully in the actor and learns sparse representations of the actor both in stochastic and deterministic cases. RDA-PG performs better than existing AC algorithms on standard RL benchmark problems with irrelevant features or redundant features.

Causal Reinforcement Learning: An Instrumental Variable Approach

In the standard data analysis framework, data is first collected (once for all), and then data analysis is carried out. Moreover, the data-generating process is typically assumed to be exogenous. This approach is natural when the data analyst has no impact on how the data is generated. The advancement of digital technology, however, has facilitated firms to learn from data and make decisions at the same time. As these decisions generate new data, the data analyst---a business manager or an algorithm---also becomes the data generator. In this article, we formulate the problem as a Markov decision process (MDP) and show that the interaction generates a new type of bias---reinforcement bias---that exacerbates the endogeneity problem in static data analysis. When the data are independent and identically distributed, we embed the instrumental variable (IV) approach in the stochastic gradient descent algorithm to correct for the bias. For general MDP problems, we propose a class of IV-based reinforcement learning (RL) algorithms to correct for the bias. We establish asymptotic properties of the algorithms by incorporating them into two-timescale stochastic approximation (SA). Our formulation requires unbounded state space and more importantly, Markovian noise. Therefore, standard techniques in RL and SA literature, which rely on boundedness of the state space and martingale-difference structure of the noise, do not apply. We develop new techniques to establish finite-time risk bounds, finite-time bounds for trajectory stability, and asymptotic distribution of a class of IV-RL algorithms.

A Tale of Two-Timescale Reinforcement Learning with the Tightest Finite-Time Bound

Policy evaluation in reinforcement learning is often conducted using two-timescale stochastic approximation, which results in various gradient temporal difference methods such as GTD(0), GTD2, and TDC. Here, we provide convergence rate bounds for this suite of algorithms. Algorithms such as these have two iterates, θn and wn, which are updated using two distinct stepsize sequences, αn and βn, respectively. Assuming αn = n−α and βn = n−β with 1 > α > β > 0, we show that, with high probability, the two iterates converge to their respective solutions θ* and w* at rates given by ∥θn - θ*∥ = Õ(n−α/2) and ∥wn - w*∥ = Õ(n−β/2); here, Õ hides logarithmic terms. Via comparable lower bounds, we show that these bounds are, in fact, tight. To the best of our knowledge, ours is the first finite-time analysis which achieves these rates. While it was known that the two timescale components decouple asymptotically, our results depict this phenomenon more explicitly by showing that it in fact happens from some finite time onwards. Lastly, compared to existing works, our result applies to a broader family of stepsizes, including non-square summable ones.

A stability criterion for two timescale stochastic approximation schemes

We present the first sufficient conditions that guarantee stability of two-timescale stochastic approximation schemes. Our analysis is based on the ordinary differential equation (ODE) method and is an extension of the results in Borkar and Meyn (2000) for single-timescale schemes. As an application of our result, we show the stability of iterates in a two-timescale stochastic approximation scheme arising in reinforcement learning.

Mixed-Strategy Learning With Continuous Action Sets

Motivated by the recent applications of game-theoretical learning to the design of distributed control systems, we study a class of control problems that can be formulated as potential games with continuous action sets. We propose an actor-critic reinforcement learning algorithm that adapts mixed strategies over continuous action spaces. To analyze the algorithm, we extend the theory of finite-dimensional two-timescale stochastic approximation to a Banach space setting, and prove that the continuous dynamics of the process converge to equilibrium in the case of potential games. These results combine to give a provably-convergent learning algorithm in which players do not need to keep track of the controls selected by other agents.

Optimized Policies for the Retransmission Probabilities in Slotted Aloha

In this paper, we study the behaviour of the slotted Aloha multiple access scheme with a finite number of users under different traffic loads and optimize the retransmission probability qr for various settings, cost objectives and policies. First, we formulate the problem as a parameter optimization problem and use certain efficient smoothed functional algorithms for finding the optimal retransmission probability parameter. Next, we propose two classes of multi-level closed-loop feedback policies (for finding in each case the retransmission probability qr that now depends on the current system state) and apply the above algorithms for finding an optimal policy within each class of policies. While one of the policy classes depends on the number of backlogged nodes in the system, the other depends on the number of time slots since the last successful transmission. The latter policies are more realistic as it is difficult to keep track of the number of backlogged nodes at each instant. We investigate the effect of increasing the number of levels in the feedback policies. We also investigate the effects of using different cost functions (with and without penalization) in our algorithms and the corresponding change in the throughput and delay using these. Both of our algorithms use two-timescale stochastic approximation. One of the algorithms uses one simulation while the other uses two simulations of the system. The two-simulation algorithm is seen to perform better than the other algorithm. Optimal multi-level closed-loop policies are seen to perform better than optimal open-loop policies. The performance further improves when more levels are used in the feedback policies.

New algorithms of the Q-learning type

We propose two algorithms for Q-learning that use the two-timescale stochastic approximation methodology. The first of these updates Q-values of all feasible state–action pairs at each instant while the second updates Q-values of states with actions chosen according to the ‘current’ randomized policy updates. A proof of convergence of the algorithms is shown. Finally, numerical experiments using the proposed algorithms on an application of routing in communication networks are presented on a few different settings.