Distributed Reinforcement Learning Algorithm Research Articles

Analysis of the joint trading of local green power certificates, carbon emissions rights, and electricity considering demand flexibility

Establishing synergy among distinct yet associated markets through market orientation is crucial because it can facilitate a clean and low-carbon energy transition. However, researchers have not yet developed a quantitative analytical dynamic model for studying market participants in the context of the joint electricity-carbon-green power certificates market. Moreover, the solution algorithm merits further exploration. This article proposes an analytical method for evaluating the joint trading of local green power certificates, carbon emission rights, and electricity, while taking into account the demand flexibility. First, a framework for the joint trading of local green power certificates, carbon emission rights, and electricity is designed. Second, market participants, such as traditional thermal power plants, renewable energy power plants, and load aggregators, are modelled via system dynamics theory. On this basis, a deep distributed reinforcement learning algorithm is applied to solve the problem. Finally, the effectiveness of the proposed theoretical approach is verified through real-life simulations, and the sensitivity of the key factors influencing joint trading is also analysed.

Multi-Agent Distributed Reinforcement Learning Algorithm for Free-Model Economic-Environmental Power and CHP Dispatch Problems

In conventional methods for Economic Dispatch Problem (EDP) and Economic-Environmental Dispatch Problem (EEDP), a full connection between the units and the control center is considered, while some faults in the communication system are possible in practical conditions. Hence, an intelligent and high-speed distributed method that is robust against disconnection is needed. In this paper, a Multi-Agent Distributed Reinforcement Learning (MADRL) algorithm based on consensus control for EDP and EEDP is presented. In this method, the incremental cost of units is optimized based on Lagrange method and the reinforcement learning algorithm. Thus, a performance index is defined for each agent in proposed algorithm to be independent of the unit model. The performance index of each agent is the sum of two terms, including i) the difference between the previous performance index value and local power and heat mismatch values and ii) the sum of the difference between the incremental cost of the unit with other neighboring units. In other words, optimizing the defined performance indexes of the agents eliminates the need for the system model. The MADRL method is tested on several grids and compared with other methods. The numerical results show an improvement in the algorithm speed and optimal point.

Distributed Multiagent Deep Reinforcement Learning for Multiline Dynamic Bus Timetable Optimization

As a primary countermeasure to mitigate traffic congestion and air pollution, promoting public transit has become a global census. Designing a robust and reliable bus timetable is a pivotal step to increase ridership and reduce operating cost for transit authorities. However, most previous studies on bus timetabling rely on historical passenger count and travel time data to generate static schedules, which often yield biased results in these uncertain scenarios, such as demand surge or adverse weather. In addition, acquiring real-time passenger origin/destination from a limited number of running buses is not feasible. This article considers the multiline dynamic bus timetable optimization problem as a Markov decision process model to address the aforementioned issues, and proposes a multiagent deep reinforcement learning framework to ensure effective learning from the imperfect-information game, where the passenger demand and traffic condition are not always known in advance. Moreover, a distributed reinforcement learning algorithm is applied to overcome the limitation of high computational cost and low efficiency. A case study of multiple bus lines in Beijing, China, confirms the effectiveness and efficiency of the proposed model. The results demonstrate that our method outperforms heuristic and state-of-the-art reinforcement learning algorithms by reducing 20.30% of operating and passenger costs compared with actual timetables.

A distributed reinforcement learning and model predictive control architecture for routing decisions in urban road networks

In this paper, platoons of autonomous vehicles in urban road networks are considered. From a methodological point of view, the problem consists in characterizing vehicle state trajectory tubes by means of routing decisions complying with traffic congestion criteria. To this end, a novel distributed control architecture is conceived by taking advantage of two methodologies: the routing decisions are obtained by using a distributed reinforcement learning algorithm that exploits available traffic data and a bank of model predictive controllers is in charge of computing the more adequate control action for each involved vehicle.

Cloud Mining Pool Aided Blockchain-Enabled Internet of Things: An Evolutionary Game Approach

The past few years have witnessed an exponential growth of diverse Internet of Things (IoT) devices as well as compelling applications ranging from industrial production to medical care. Dramatic advances in IoT technology not only brought enormous economic opportunities but also challenges (e.g., privacy and security vulnerabilities). Recently, with the appearance of blockchain technology, the integration of IoT and blockchain (BCoT) is considered a promising solution to address these issues. Blockchain provides a secure and scalable data management framework for IoT devices. However, the huge computation and energy cost of the consensus process in blockchain prevents it from being directly applied as a generic platform. To overcome this challenge, in this article, we propose a cloud mining pool-aided BCoT architecture, where the IoT devices can rent the computing resources from the cloud mining pools to offload the mining process. Based on this architecture, we study the mining pool selection problem and analyze the colony behaviors of IoT devices with different pooling strategies. We propose a centralized evolutionary game-based pool selection algorithm for the sake of maximizing the system utility. Considering the non-cooperative relationship among multiple miners, we also propose a lightweight distributed reinforcement learning algorithm, named the ‘WoLF-PHC’ algorithm.

A Survey on Distributed Reinforcement Learning

In many settings, reinforcement learning (RL) has proven to be an effective tool for tackling difficult decision-making challenges. Traditional RL algorithms, on the other hand, frequently hit walls when confronted with issues of a sufficiently great scale or complexity. Distributed reinforcement learning (DRL) is a new area of study that hopes to circumvent these restrictions by dividing the learning workload among several computers. In this work, we offer a thorough overview of DRL, discussing its history, difficulties, applications, evaluation, scalability, and outstanding issues. We classify DRL approaches and frameworks and examine their similarities and differences. We also highlight the difficulties and restrictions of using DRL in real-world circumstances and examine its practical applicability in a variety of fields. We also describe current trends and future directions for evaluating DRL algorithms, and we analyse the performance of DRL algorithms on benchmark tasks. We also go through various options for distributed computing in DRL, as well as other methods for increasing the scalability and efficiency of DRL algorithms. We conclude by outlining key concerns and obstacles in DRL study, and by making suggestions for moving the subject forward. Overall, the goal of this survey is to give readers a picture of where things stand in terms of DRL study and application at the moment.

Intelligent Decision Making Based on the Combination of Deep Reinforcement Learning and an Influence Map

Almost all recent deep reinforcement learning algorithms use four consecutive frames as the state space to retain the dynamic information. If the training state data constitute an image, the state space is used as the input of the neural network for training. As an AI-assisted decision-making technology, a dynamic influence map can describe dynamic information. In this paper, we propose the use of a frame image superimposed with an influence map as the state space to express dynamic information. Herein, we optimize Ape-x as a distributed reinforcement learning algorithm. Sparse reward is an issue that must be solved in refined intelligent decision making. The use of an influence map is proposed to generate the intrinsic reward when there is no external reward. The experiments conducted in this study prove that the combination of a dynamic influence map and deep reinforcement learning is effective. Compared with the traditional method that uses four consecutive frames to represent dynamic information, the score of the proposed method is increased by 11–13%, the training speed is increased by 59%, the video memory consumption is reduced by 30%, and the memory consumption is reduced by 50%. The proposed method is compared with the Ape-x algorithm without an influence map, DQN, N-Step DQN, QR-DQN, Dueling DQN, and C51. The experimental results show that the final score of the proposed method is higher than that of the compared baseline methods. In addition, the influence map is used to generate an intrinsic reward to effectively resolve the sparse reward problem.

Financial Accounting Information Data Analysis System Based on Internet of Things

In order to meet the intelligent demand of modern financial data analysis, this paper proposes a financial accounting information data analysis system based on the Internet of things. Based on the central reinforcement learning architecture, the model uses multiple execution modules to enhance the computing and generalization ability of the single-agent reinforcement learning algorithm. In the selection of reinforcement learning algorithm, the instantaneous time difference algorithm is introduced. The algorithm can synchronize the experience of the previous iteration state in the learning process and does not depend on the final prediction value, which greatly saves the storage cost. In the establishment of the financial data analysis index system, the paper comprehensively considers the enterprise’s operation, development, debt repayment, and other capabilities, ensuring the integrity and rationality of the index system. In order to evaluate the performance of the algorithm, this paper takes the real financial data as the sample and uses BP neural network to conduct a comparative experiment. The experimental results show that the recognition accuracy of the model is better than that of the BP neural network in each experimental scenario, and the recognition accuracy of Experiment 3 is improved by 4.6%. Conclusion. The performance of the distributed reinforcement learning algorithm is better than that of the common back-propagation neural network in the real data set scenario.

An Efficient Resource Management Optimization Scheme for Internet of Vehicles in Edge Computing Environment.

The contradiction between limited network resources and a large number of user demands in vehicle environment will cause a lot of system delay and energy consumption. To solve the problem, this paper proposes an efficient resource management optimization scheme for Internet of Vehicles in edge computing environment. Firstly, we give a detailed formulation description of communication and computing cost incurred in the resource optimization process. Then, the optimization objective of this paper is clarified by considering the constraints of computing resources, and system delay and energy consumption are considered comprehensively. Secondly, considering dynamic, random, and time-varying characteristics of vehicle network, the optimal resource management scheme of Internet of Vehicles is given by using distributed reinforcement learning algorithm to optimize total system overhead to the greatest extent. Finally, experiments show that when bandwidth = 40 MHz, the total system cost of the proposed algorithm is only 3.502, while that of comparison algorithms is 4.732 and 4.251, respectively. It is proved that the proposed method can effectively reduce the total system overhead.

Guest Editorial: Enhancing hosting capability for renewable energy generation in active distribution networks

Guest Editorial: Enhancing hosting capability for renewable energy generation in active distribution networks

Communication-Efficient and Federated Multi-Agent Reinforcement Learning

In this paper, we consider a distributed reinforcement learning setting where agents are communicating with a central entity in a shared environment to maximize a global reward. A main challenge in this setting is that the randomness of the wireless channel perturbs each agent&#x2019;s model update while multiple agents&#x2019; updates may cause interference when communicating under limited bandwidth. To address this issue, we propose a novel distributed reinforcement learning algorithm based on the alternating direction method of multipliers (ADMM) and &#x201C;<i>over air aggregation</i>&#x201D; using analog transmission scheme, referred to as A-RLADMM. Our algorithm incorporates the wireless channel into the formulation of the ADMM method, which enables agents to transmit each element of their updated models over the same channel using analog communication. Numerical experiments on a multi-agent collaborative navigation task show that our proposed algorithm significantly outperforms the digital communication baseline of A-RLADMM (D-RLADMM), the lazily aggregated policy gradient (RL-LAPG), as well as the analog and the digital communication versions of the vanilla FL, (A-FRL) and (D-FRL) respectively.

Blockchain-Based Task Offloading for Edge Computing on Low-Quality Data via Distributed Learning in the Internet of Energy

With the development of the Internet of energy, more and more participants share data by different types of edge devices. However, such multi-source heterogenous data typically contain low-quality data, e.g., missing values, which may result in potential risks. Besides, resource-constrained devices incur large latency in edge computing networks. To alleviate such latency, distributed task offloading schemes are designed to share the computation burden between edge nodes and nearby servers. However, there are three main drawbacks of such schemes. First, low-quality data are not carefully evaluated by constraints under scenarios, which may result in slow convergence in distributed computation. Second, multi-source data including sensitive information are computed and shared among edge nodes without privacy protection. Third, distributed tasks on low-quality data may result in low-quality results even with an optimal offloading scheme. To address the problems above, a task offloading framework for edge computing based on consortium blockchain and distributed reinforcement learning is proposed in this paper, which can provide high-quality task offloading policies with data privacy protected. This framework consists of three key components: data quality evaluation (DQ) with multiple data quality dimensions, data repairing (DR) with a repairing algorithm based on a novel repairing consensus mechanism and distributed reinforcement learning for task arrangement (DELTA) with a distributed reinforcement learning algorithm based on a novel low-quality data distributing strategy. Numeric results are presented to illustrate the effectiveness and efficiency of the proposed task offloading framework for edge computing on low-quality data in the IoE.

Distributed Reinforcement Learning Algorithm for Multi-Wave Fire Fighting Scheduling Problem

This paper studies the distribution of FFEs (fire fighting equipments) carried by UAVs (unmanned aerial vehicles) from FFUs (fire fighting units) under the background of multi-wave forest fire. The objective is to allocate the FFEs of each FFU to minimize the sum of the probabilities of each fire site’s unsuccessful extinguishment. In order to solve the multi-wave equipment distribution problem of the FFUs, a distributed reinforcement learning algorithm is designed in this paper. In the algorithm, agents cooperate to find the optimal distribution of FFEs based on information exchange, and a local Q-function is established for each agent to find the optimal FFE distribution combination. Simulation results demonstrate the effectiveness of the algorithm.

Modern Value Based Reinforcement Learning: A Chronological Review

Investigation of value based Reinforcement Learning algorithms exhibited a resurgence into mainstream research in 2015 following demonstration of super-human performance when applied to Atari 2600 games. Since then, significant media attention and hype have accompanied this area, and the field of Artificial Intelligence generally, spread across distinct categories. This review paper is focused exclusively on the progression of value based Reinforcement Learning in the last five years. We aim to distill the incremental improvements to stability and performance in this period, highlighting the minimal changes to the base algorithm over this time. This holds true with all but the one exception of the Recurrent Experience Replay in Distributed Reinforcement Learning algorithm, representing a fundamental shift and increase in agent performance through an advanced memory representation. We suggest a new focus area for value based Reinforcement Learning research.

Blockchain Enabled Intelligence of Federated Systems (BELIEFS): An attack-tolerant trustable distributed intelligence paradigm

In this article, a Blockchain Enabled Intelligence of Federated Systems (BELIEFS) is proposed to conduct cooperative control for the multi-regional large-scale power system with a multi-agents system (MAS). By establishing a two levels blockchain, each regional AI agent can simultaneously manage intra-regional controllers and cooperate with other AI agents. Under the consensus mechanism, the agents, which respectively conducted distributed deep reinforcement learning (DDRL) algorithm in multi-regions, can have the tolerant capability of malicious attacks in their training process. The demonstration of the proposed approach is within a multi-regional large-scale interconnected power system. Under the mode of “centralized dispatching and hierarchical management”, this article aims to definite a mathematical model to deal with the control problem of the power systems. With the comparison experiments, the effectiveness and efficiency of our proposed method in the training process are verified. In addition, malicious attacks are set on the main chain and shard chains to verify the attack-tolerant capability. We expect that such approach and results can suggest a new paradigm of attack-tolerant trustable distributed AI deployment.

SDP-Based Bounds for the Quadratic Cycle Cover Problem via Cutting-Plane Augmented Lagrangian Methods and Reinforcement Learning

We study the quadratic cycle cover problem (QCCP), which aims to find a node-disjoint cycle cover in a directed graph with minimum interaction cost between successive arcs. We derive several semidefinite programming (SDP) relaxations and use facial reduction to make these strictly feasible. We investigate a nontrivial relationship between the transformation matrix used in the reduction and the structure of the graph, which is exploited in an efficient algorithm that constructs this matrix for any instance of the problem. To solve our relaxations, we propose an algorithm that incorporates an augmented Lagrangian method into a cutting-plane framework by utilizing Dykstra’s projection algorithm. Our algorithm is suitable for solving SDP relaxations with a large number of cutting-planes. Computational results show that our SDP bounds and efficient cutting-plane algorithm outperform other QCCP bounding approaches from the literature. Finally, we provide several SDP-based upper bounding techniques, among which is a sequential Q-learning method that exploits a solution of our SDP relaxation within a reinforcement learning environment. Summary of Contribution: The quadratic cycle cover problem (QCCP) is the problem of finding a set of node-disjoint cycles covering all the nodes in a graph such that the total interaction cost between successive arcs is minimized. The QCCP has applications in many fields, among which are robotics, transportation, energy distribution networks, and automatic inspection. Besides this, the problem has a high theoretical relevance because of its close connection to the quadratic traveling salesman problem (QTSP). The QTSP has several applications, for example, in bioinformatics, and is considered to be among the most difficult combinatorial optimization problems nowadays. After removing the subtour elimination constraints, the QTSP boils down to the QCCP. Hence, an in-depth study of the QCCP also contributes to the construction of strong bounds for the QTSP. In this paper, we study the application of semidefinite programming (SDP) to obtain strong bounds for the QCCP. Our strongest SDP relaxation is very hard to solve by any SDP solver because of the large number of involved cutting-planes. Because of that, we propose a new approach in which an augmented Lagrangian method is incorporated into a cutting-plane framework by utilizing Dykstra’s projection algorithm. We emphasize an efficient implementation of the method and perform an extensive computational study. This study shows that our method is able to handle a large number of cuts and that the resulting bounds are currently the best QCCP bounds in the literature. We also introduce several upper bounding techniques, among which is a distributed reinforcement learning algorithm that exploits our SDP relaxations.

MLDRL: A multi-layer distributed reinforcement learning framework with multiple trainers

Synchronous distributed sampling is widely used in various distributed reinforcement learning algorithms. However, it is difficult to scale to a large case due to significant synchronization and communication costs. In this paper, we propose a scalable layered distributed reinforcement learning framework by introducing multiple trainers. We use a parameter processor to coordinate parameters obtained from multiple trainers and introduce worker nodes to reduce the communication cost among the parameter processor and the trainers. We do experiments in a multi-agent environment, and the results show that the proposed framework significantly accelerates the training efficiency of the reinforcement learning algorithm.

Computation Offloading in Beyond 5G Networks: A Distributed Learning Framework and Applications

Facing the trend of merging wireless communications and multi-access edge computing (MEC), this article studies computation offloading in beyond fifth generation networks. To address the technical challenges originating from the uncertainties and the sharing of limited resource in an MEC system, we formulate the computation offloading problem as a multi-agent Markov decision process, for which a distributed learning framework is proposed. We present a case study on resource orchestration in computation offloading to showcase the potential of an online distributed reinforcement learning algorithm developed under the proposed framework. Experimental results demonstrate that our learning algorithm outperforms the benchmark resource orchestration algorithms. Furthermore, we outline the research directions worth in-depth investigation to minimize the time cost, which is one of the main practical issues that prevent the implementation of the proposed distributed learning framework.

Distributed reinforcement learning algorithm of operator service slice competition prediction based on zero-sum markov game

As a key enabling technology in the emerging network, network slicing can dynamically provide on-demand service with distinct logical slice instance. While most related studies have mainly focused on resource management, this study targets solving business competition between two operator slices using artificial intelligence. In this competition, each operator slice tries to maximize its own payoff, meanwhile its opponent strives to minimize it. Moreover, two operators update their marketing strategies over time. Therefore, predicting its result is a challenge. After the zero-sum Markov game is modeled for the research problem, we present the min–max Q learning algorithm. In each market state, each slice attains its temporary optimal strategy using the min–max algorithm. In the Markov decision process, Q value is dynamically modified under different market states, and the final Q value presents predictive result for this competition. Finally, a mass of numerical results prove that the min–max Q learning algorithm outperforms the repeated game, in which market state is invariable over time.

Distributed Reinforcement Learning for Cyber-Physical System With Multiple Remote State Estimation Under DoS Attacker

In this paper, we consider cyber-physical system (CPS) with multiple remote state estimation under denial-of-service (DoS) attack in infinite time-horizon. The sensors monitor the system and send their local state estimate to remote estimators by choosing the local channels in “State 0” or “State 1”. The aim of sensors is to find policies for choosing local channel in a specific state to transmit message to minimize the total estimation error covariance on account of energy-saving in an infinite time-horizon. The DoS attacker aims to achieve the opposite goal by choosing channels to attack or not. The games between sensors and DoS attacker under two different structures of public information are investigated, that is the open-loop case (where sensors and attacker cannot observe others' behaviors) and the closed-loop case (where sensors and attacker can observe the others' behaviors causally). For the open-loop case with assumption that the DoS attacker can get the information from the remote estimators to the sensors, the distributed reinforcement learning algorithms for sensors and attacker based on local information are proposed to find their Nash equilibrium policies, respectively. Further, we consider in closed loop case that the DoS attacker cannot get the information from the remote estimators to the sensors which leads to asymmetric information between the sensors and attacker. To derive Nash equilibrium policies for sensors and attacker, we convert the original game into a belief-based continuous-state stochastic game. The convergence of distributed reinforcement learning method is proved. Some simulations are presented to demonstrate its effectiveness.