Distributed Reinforcement Learning Algorithm Research Articles

Distributed Reinforcement Learning Algorithm for Dynamic Economic Dispatch With Unknown Generation Cost Functions

In this article, the dynamic economic dispatch (DED) problem for smart grid is solved under the assumption that no knowledge of the mathematical formulation of the actual generation cost functions is available. The objective of the DED problem is to find the optimal power output of each unit at each time so as to minimize the total generation cost. To address the lack of a priori knowledge, a new distributed reinforcement learning optimization algorithm is proposed. The algorithm combines the state-action-value function approximation with a distributed optimization based on multiplier splitting. Theoretical analysis of the proposed algorithm is provided to prove the feasibility of the algorithm, and several case studies are presented to demonstrate its effectiveness.

Smart fog based workflow for traffic control networks

In this paper, we propose a novel traffic control architecture which is based on fog computing paradigm and reinforcement leaning technologies. We firstly provide an overview of this framework and detail the components and workflows designed to relieve traffic congestion. These workflows, which are connecting traffic lights, vehicles, Fog nodes and traffic cloud, aim to generate traffic light control flow and communication flow for each intersection to avoid a traffic jam. In order to make the whole city’s traffic highly efficient, the fog computing paradigm and a distributed reinforcement learning algorithm is designed to overcome communication bandwidth limitation and local optimal traffic control flow, respectively. We also demonstrate that our framework outperforms traditional systems and provides high practicability in future research for building the intelligent transportation system.

When Social Network Meets Mobile Cloud: A Social Group Utility Approach for Optimizing Computation Offloading in Cloudlet

Cloudlet is a new paradigm in mobile cloud computing to provide resources to nearby mobile users via one-hop wireless connections. In this paper, we leverage the social tie structure among mobile users to achieve mutually beneficial computation offloading decision making, and hence, enhance the system-wide performance. Drawing on a social group utility maximization (SGUM) framework, we cast users’ decision making of whether to offload or not as a Socially aware computation offloading game (COG). We study the SGUM-based COG for both strong and weak information cases. For the strong information case, where each user has the knowledge of other users’ actions and the perfect observation of its achieved social group utility, we show that there exists a socially aware Nash equilibrium (SNE). We then design a distributed algorithm to achieve the SNE and quantify its performance gap with respect to the social optimal solution. For the weak information case, where each user does not have the knowledge of other users’ actions and observes a noise-corrupted social group utility, we develop a distributed reinforcement learning algorithm, which is shown to converge almost surely to an $\epsilon $ -SNE. The numerical results show that the computation offloading performance can be significantly enhanced by leveraging the social ties among the users.

Decentralized Multi-Agent Reinforcement Learning in Average-Reward Dynamic DCOPs

Researchers have introduced the Dynamic Distributed Constraint Optimization Problem (Dynamic DCOP) formulation to model dynamically changing multi-agent coordination problems, where a dynamic DCOP is a sequence of (static canonical) DCOPs, each partially different from the DCOP preceding it. Existing work typically assumes that the problem in each time step is decoupled from the problems in other time steps, which might not hold in some applications. Therefore, in this paper, we make the following contributions: (i) We introduce a new model, called Markovian Dynamic DCOPs (MD-DCOPs), where the DCOP in the next time step is a function of the value assignments in the current time step; (ii) We introduce two distributed reinforcement learning algorithms, the Distributed RVI Q-learning algorithm and the Distributed R-learning algorithm, that balance exploration and exploitation to solve MD-DCOPs in an online manner; and (iii) We empirically evaluate them against an existing multi-arm bandit DCOP algorithm on dynamic DCOPs.

A fuzzy reinforcement learning approach to power control in wireless transmitters

We address the issue of power-controlled shared channel access in wireless networks supporting packetized data traffic. We formulate this problem using the dynamic programming framework and present a new distributed fuzzy reinforcement learning algorithm (ACFRL-2) capable of adequately solving a class of problems to which the power control problem belongs. Our experimental results show that the algorithm converges almost deterministically to a neighborhood of optimal parameter values, as opposed to a very noisy stochastic convergence of earlier algorithms. The main tradeoff facing a transmitter is to balance its current power level with future backlog in the presence of stochastically changing interference. Simulation experiments demonstrate that the ACFRL-2 algorithm achieves significant performance gains over the standard power control approach used in CDMA2000. Such a large improvement is explained by the fact that ACFRL-2 allows transmitters to learn implicit coordination policies, which back off under stressful channel conditions as opposed to engaging in escalating "power wars."