Distributed Deep Reinforcement Learning Research Articles

Deep Reinforcement Learning techniques for dynamic task offloading in the 5G edge-cloud continuum

The integration of new Internet of Things (IoT) applications and services heavily relies on task offloading to external devices due to the constrained computing and battery resources of IoT devices. Up to now, Cloud Computing (CC) paradigm has been a good approach for tasks where latency is not critical, but it is not useful when latency matters, so Multi-access Edge Computing (MEC) can be of use. In this work, we propose a distributed Deep Reinforcement Learning (DRL) tool to optimize the binary task offloading decision, this is, the independent decision of where to execute each computing task, depending on many factors. The optimization goal in this work is to maximize the Quality-of-Experience (QoE) when performing tasks, which is defined as a metric related to the battery level of the UE, but subject to satisfying tasks’ latency requirements. This distributed DRL approach, specifically an Actor-Critic (AC) algorithm running on each User Equipment (UE), is evaluated through the simulation of two distinct scenarios and outperforms other analyzed baselines in terms of QoE values and/or energy consumption in dynamic environments, also demonstrating that decisions need to be adapted to the environment’s evolution.

Joint mode selection and resource allocation for cellular V2X communication using distributed deep reinforcement learning under 5G and beyond networks

Vehicle-to-everything (V2X) communication via cellular networks is a promising technique for 5G and beyond networks. The cars interact directly with one another, as well as with the infrastructure and various vehicles on the road, in this mode. It enables the interchange of time-sensitive and safety-critical data. Despite these benefits, unstable vehicle-to-vehicle (V2V) communications, insufficient channel status information, high transmission overhead, and the considerable communication cost of centralized resource allocation systems all pose challenges for defense applications. To address these difficulties, this study proposes a combined mode selection and resource allocation system based on distributed deep reinforcement learning (DRL) to optimize the overall network sum rate while maintaining the reliability and latency requirements of V2V pairs and the data rate of V2R connections. Because the optimization issue is non-convex and NP-hard, it cannot be solved directly. To tackle this problem, the defined problem is first translated into machine learning form using the Markov decision process (MDP) to construct the reward function and decide whether agent would conduct the action. Following that, the distributed coordinated duelling deep Q-network (DDQN) method based on prioritized sampling is employed to improve mode selection and resource allocation. This approach learns the action-value distribution by estimating both the state-value and action advantage functions using duelling deep networks. The results of the simulation show that the suggested scheme outperforms state-of-the-art decentralized systems in terms of sum rate and QoS satisfaction probability.

Cheaper and Faster: Distributed Deep Reinforcement Learning with Serverless Computing

Deep reinforcement learning (DRL) has gained immense success in many applications, including gaming AI, robotics, and system scheduling. Distributed algorithms and architectures have been vastly proposed (e.g., actor-learner architecture) to accelerate DRL training with large-scale server-based clusters. However, training on-policy algorithms with the actor-learner architecture unavoidably induces resource wasting due to synchronization between learners and actors, thus resulting in significantly extra billing. As a promising alternative, serverless computing naturally fits on-policy synchronization and alleviates resource wasting in distributed DRL training with pay-as-you-go pricing. Yet, none has leveraged serverless computing to facilitate DRL training. This paper proposes MinionsRL, the first serverless distributed DRL training framework that aims to accelerate DRL training- and cost-efficiency with dynamic actor scaling. We prototype MinionsRL on top of Microsoft Azure Container Instances and evaluate it with popular DRL tasks from OpenAI Gym. Extensive experiments show that MinionsRL reduces total training time by up to 52% and training cost by 86% compared to latest solutions.

Distributed Deep Reinforcement Learning With Prioritized Replay for Power Allocation in Underwater Acoustic Communication Networks

Distributed Deep Reinforcement Learning With Prioritized Replay for Power Allocation in Underwater Acoustic Communication Networks

Resilient Optimal Defensive Strategy of Micro-Grids System via Distributed Deep Reinforcement Learning Approach Against FDI Attack.

The ever-increasing false data injection (FDI) attack on the demand side brings great challenges to the energy management of interconnected microgrids. To address those aspects, this article proposes a resilient optimal defensive strategy with the distributed deep reinforcement learning (DRL) approach. To evaluate the FDI attack on demand response (DR), an online evaluation approach with the recursive least-square (RLS) method is proposed to evaluate the extent of supply security or voltage stability of the microgrids system is affected by the FDI attack. On the basis of evaluated security confidence, a distributed actor network learning approach is proposed to deduce optimal network weight, which can generate an optimal defensive scheme to ensure the economic and security issue of the microgrids system. From the methodology's view, it can also enhance the autonomy of each microgrid as well as accelerate DRL efficiency. According to those simulation results, it can reveal that the proposed method can evaluate FDI attack impact well and an improved distributed DRL approach can be a viable and promising way for the optimal defense of microgrids against the FDI attack on the demand side.

Towards Efficient Task Offloading with Dependency Guarantees in Vehicular Edge Networks through Distributed Deep Reinforcement Learning

Towards Efficient Task Offloading with Dependency Guarantees in Vehicular Edge Networks through Distributed Deep Reinforcement Learning

Adaptive Static Equivalences for Active Distribution Networks With Massive Renewable Energy Integration: A Distributed Deep Reinforcement Learning Approach

Active distribution networks (ADNs) with 100% renewable energy sources (RESs) penetration are essential in the net-zeros emission power sector. As an efficient analysis tool, the equivalent model for ADNs (EMADNs) can be leveraged to expedite the lengthy analysis and preserve data privacy. Nevertheless, volatile RESs, along with their active management techniques, e.g., voltage regulation schemes (VRSs), deteriorate the fidelity of the developed EMADNs derived from historical data. This paper proposes an adaptive EMADN with tunable network scales and adaptive parameters to address the above challenges. A leaves-trimming network topological reduction algorithm is utilized to preserve the radial topology, the nodes of interest, and VRSs. Upon the derived network topology and components, a distributed Proximal Policy Optimization (DPPO)-based deep reinforcement learning agent is employed to adjust the parameters periodically to maintain the fidelity of EMADNs over time. The distributed training scheme extends the ordinary PPO to boost exploration and training efficiency by exploiting the multi-processors in a synchronous way. Case studies on the IEEE-33 bus and IEEE 123-bus ADNs with massive photovoltaic and wind generation demonstrate the superior accuracy and efficiency of the proposed agent-based EMADNs in various scenarios, especially when the production of RESs exceeds the load amount.

: A QoS-Aware Distributed Deep Reinforcement Learning Technique for Service Offloading in Fog Computing Environments

: A QoS-Aware Distributed Deep Reinforcement Learning Technique for Service Offloading in Fog Computing Environments

Digital-Twin-Assisted Resource Allocation for Network Slicing in Industry 4.0 and Beyond Using Distributed Deep Reinforcement Learning

Personalization is one of the primary emerging trends in Industry 4.0 and Beyond. Highly personalized services will present a significant challenge to the existing algorithms for Network Slicing (NS) and resource allocation, leading to issues such as nonequilibratory resource allocation in which some services are sacrificed for the maximum total reward of the algorithm, excessive cost and slow algorithm convergence. Digital Twin Network (DTN) is offered as a novel solution to the challenges listed above. By integrating the DTN and IIoT network slicing, we propose a Digital Twin Network Assisted Industry Internet of Things Network Slicing (DTN-IIoT NS) Architecture for Personalized IIoT services in Industry 4.0 and Beyond. The DTN-IIoT NS Architecture consists of three layers, three modules and two closed loops. On the basis of the aforementioned architecture, we focus on the resource allocation process in DTN-IIoT NS, model the DT-assisted resource allocation for highly personalized IIoT services, propose the service equilibrium rate, and formulate the optimization problem aiming at maximizing the equilibrium rate weighted net profit of network providers. Then, we propose a Dual-Channel Weighted (DCW) Critic network for service equilibrium in DTN-IIoT NS resource allocation and the matching Improved Prioritized Experience Replay (PER) to enhance convergent speed. In addition, we present a distributed DT-assisted DCW-PER Multiagent Deep Deterministic Policy Gradient (PER-DCW MADDPG) algorithm for the resource allocation process in DTN-IIoT NS. Simulation results indicate that the PER-DCW MADDPG algorithm can produce a better service equilibrium and accelerate the convergence speed of the algorithm.

Spectral Efficiency Improvement in Downlink Fog Radio Access Network With Deep-Reinforcement-Learning-Enabled Power Control

Fog radio access network (F-RAN) is a promising architecture that leverages edge computing and caching to improve devices’ latency and quality of service. However, interference, which arises when multiple devices are concurrently scheduled on the same radio resource block (RRB), limits the performance of a dense F-RAN. This paper considers a multi-cell F-RAN in which the devices of each small cell receive data from the associated fog access point (F-AP) over the same RRB(s). The F-APs transmit data to the associated devices using rate-splitting multiple access (RSMA) schemes to manage co-channel interference within the small cells efficiently. A transmit power control scheme is proposed to maximize the network’s spectral efficiency (SE) while considering the devices’ hardware impairments (HWIs). The considered transmit power control scheme is an NP-hard problem, which is highly challenging to solve using the legacy optimization approach. To address this challenge, we propose a distributed deep reinforcement learning (DRL)-based power allocation (DDPA) scheme that takes the time-varying dynamics of the network and the HWIs of devices into account. Each F-AP in the proposed framework is equipped with a DRL agent that collects signal-to-interference-plus-noise ratio and channel state information from connected devices and adapts the transmit power allocation each scheduling interval. In addition, the ensemble learning framework is exploited to further improve the proposed DDPA scheme’s performance. We use extensive simulations to demonstrate that the DDPA scheme achieves greater SE than contemporary transmit power control schemes. In particular, the proposed DDPA scheme is especially suited to scenarios with non-negligible HWIs-induced distortion.

Distributed Deep Reinforcement Learning Assisted Resource Allocation Algorithm for Space-Air-Ground Integrated Networks

Distributed Deep Reinforcement Learning Assisted Resource Allocation Algorithm for Space-Air-Ground Integrated Networks

Learning-Based UAV Coverage-Aware Path Planning in Large-scale Urban Environments

In our research, the coverage-aware path planning problem of unmanned aerial vehicles (UAVs) in large-scale urban environments is studied. The goal of our research is to train the intelligent UAV to autonomously cover the area with satisfactory communication quality and finally arrive at the objective place without colliding with numerous skyscrapers. Such a multi-objective path planning problem is regularly non-convex and thus hard to converge. To this end, a distributed deep reinforcement learning (DRL)-based approach called Distributed-RPPO is developed to address the problem. To be specific, firstly we formulate our proposed problem as a Markov decision process (MDP). Since communication connectivity is the core constraint, a proximal policy optimization (PPO)-based SIR prediction network is designed for guiding the UAV to cover the area with superior communication connectivity, using only the UAV’s current location. Then we combine the PPO algorithm with the Long short-term memory (LSTM) structure to solve the basic navigation problem while only the historical interactive data is exploited for updating the parameter vector of the entire network. Simulation results demonstrate that our approach is capable of allowing the UAV to execute coverage-aware path planning tasks in large-scale urban environments. Furthermore, the learning efficiency is greatly improved compared to existing DRL-based approaches.

Edge Computing Resource Allocation Algorithm for NB-IoT Based on Deep Reinforcement Learning

Mobile edge computing (MEC) technology guarantees the privacy and security of large-scale data in the Narrowband-IoT (NB-IoT) by deploying MEC servers near base stations to provide sufficient computing, storage, and data processing capacity to meet the delay and energy consumption requirements of NB-IoT terminal equipment. For the NB-IoT MEC system, this paper proposes a resource allocation algorithm based on deep reinforcement learning to optimize the total cost of task offloading and execution. Since the formulated problem is a mixed-integer non-linear programming (MINLP), we cast our problem as a multi-agent distributed deep reinforcement learning (DRL) problem and address it using dueling Q-learning network algorithm. Simulation results show that compared with the deep Q-learning network and the all-local cost and all-offload cost algorithms, the proposed algorithm can effectively guarantee the success rates of task offloading and execution. In addition, when the execution task volume is 200KBit, the total system cost of the proposed algorithm can be reduced by at least 1.3%,and when the execution task volume is 600KBit, the total cost of system execution tasks can be reduced by 16.7% at most.

A Distributed Deep Reinforcement Learning Technique for Application Placement in Edge and Fog Computing Environments

Fog/Edge computing is a novel computing paradigm supporting resource-constrained Internet of Things (IoT) devices by placement of their tasks on edge and/or cloud servers. Recently, several Deep Reinforcement Learning (DRL)-based placement techniques have been proposed in fog/edge computing environments, which are only suitable for centralized setups. The training of well-performed DRL agents requires manifold training data while obtaining training data is costly. Hence, these centralized DRL-based techniques lack generalizability and quick adaptability, thus failing to efficiently tackle application placement problems. Moreover, many IoT applications are modeled as Directed Acyclic Graphs (DAGs) with diverse topologies. Satisfying dependencies of DAG-based IoT applications incur additional constraints and increase the complexity of placement problem. To overcome these challenges, we propose an actor-critic-based distributed application placement technique, working based on the IMPortance weighted Actor-Learner Architectures (IMPALA). IMPALA is known for efficient distributed experience trajectory generation that significantly reduces exploration costs of agents. Besides, it uses an adaptive off-policy correction method for faster convergence to optimal solutions. Our technique uses recurrent layers to capture temporal behaviors of input data and a replay buffer to improve the sample efficiency. The performance results, obtained from simulation and testbed experiments, demonstrate that our technique significantly improves execution cost of IoT applications up to 30\% compared to its counterparts.

Deep Recurrent Reinforcement Learning-Based Distributed Dynamic Spectrum Access in Multichannel Wireless Networks With Imperfect Feedback

This paper investigates Dynamic Spectrum Access (DSA) paradigm with imperfect feedback for multiuser wireless network. Each user selects an orthogonal channel in particular time slot to transmit packet with a certain transmission probability. In next time slot, the user who has transmitted a packet receives an ACK signal based on local observation. Bearing in mind the dynamic nature of wireless networks, it is appealing to develop a blended strategy to perform effective DSA. This paper aims to design a distributed Deep Reinforcement Learning (DRL) based scheme with an objective of maximizing network utility. In conventional DRL framework, it is assumed that the feedback received (ACK packet) is always correct but in wireless networks, it may be lost or corrupted due to noise. Furthermore, it is challenging to promise that the multiple agents will cooperate to make coherent decisions in order to accomplish the same objective, particularly under imperfect feedback. To tackle these challenges, this work proposes (i) Deep Recurrent Reinforcement Learning network with integrated GRU layer to optimize the network utility function and (ii) a feedback recovery mechanism using complete and incomplete replay buffers. Extensive simulations corroborate the success of proposed scheme in complex multiuser scenario and exhibit robustness against the detrimental effects of the imperfect feedback.

Channel assignment and power allocation for throughput improvement with PPO in B5G heterogeneous edge networks

In Beyond the Fifth Generation (B5G) heterogeneous edge networks, numerous users are multiplexed on a channel or served on the same frequency resource block, in which case the transmitter applies coding and the receiver uses interference cancellation. Unfortunately, uncoordinated radio resource allocation can reduce system throughput and lead to user inequity, for this reason, in this paper, channel allocation and power allocation problems are formulated to maximize the system sum rate and minimum user achievable rate. Since the construction model is non-convex and the response variables are high-dimensional, a distributed Deep Reinforcement Learning (DRL) framework called distributed Proximal Policy Optimization (PPO) is proposed to allocate or assign resources. Specifically, several simulated agents are trained in a heterogeneous environment to find robust behaviors that perform well in channel assignment and power allocation. Moreover, agents in the collection stage slow down, which hinders the learning of other agents. Therefore, a preemption strategy is further proposed in this paper to optimize the distributed PPO, form DP-PPO and successfully mitigate the straggler problem. The experimental results show that our mechanism named DP-PPO improves the performance over other DRL methods.

Straight-Path Following and Formation Control of USVs Using Distributed Deep Reinforcement Learning and Adaptive Neural Network

Dear Editor, This letter presents a distributed deep reinforcement learning (DRL) based approach to deal with the path following and formation control problems for underactuated unmanned surface vehicles (USVs). By constructing two independent actor-critic architectures, the deep deterministic policy gradient (DDPG) method is proposed to determine the desired heading and speed command for each USV. We consider the realistic dynamical model and the input saturation problem. The radial basis function neural networks (RBF NNs) are employed to approximate the hydrodynamics and unknown external disturbances of USVs. Simulation results show that our proposed method can achieve high-level tracking control accuracy while keeping a desired stable formation.

5G Multi-RAT URLLC and eMBB Dynamic Task Offloading With MEC Resource Allocation Using Distributed Deep Reinforcement Learning

In this article, a deep reinforcement learning (DRL) control scheme is proposed to satisfy the strict Quality-of-Service (QoS) requirements of ultrareliability low-latency communication (URLLC) and enhanced mobile broadband (eMBB) using 5G multiple radio access technology (RAT)-based partial offloading and multiaccess edge-computing (MEC) resource allocation. In the proposed scheme, the user equipment (UE) makes optimal offloading decisions while the MEC server dynamically adjusts the server resources based on offloading requests from multiple UEs using DRL technology. The aim of the proposed scheme is to minimize the energy consumption of the UEs while maximizing the system utility (SU) performance, which is composed of the spectral efficiency (SE) and offloading success rate (OSR) of the MEC server. In addition, multiagent distributed learning technology and best experience push (BEP) techniques are used to enhance the learning efficiency of the DRL framework. The simulation result shows that the proposed scheme provides an improved SU and energy consumption performance compared to the benchmark offloading schemes.

Distributed Beamforming Techniques for Cell-Free Wireless Networks Using Deep Reinforcement Learning

In a cell-free network, a large number of mobile devices are served simultaneously by several base stations (BSs)/access points(APs) using the same time/frequency resources. However, this creates high signal processing demands (e.g., for beamforming) at the transmitters and receivers. In this work, we develop centralized and distributed deep reinforcement learning (DRL)-based methods to optimize beamforming at the uplink of a cell-free network. First, we propose a fully centralized uplink beamforming method (i.e., centralized learning) that uses the Deep Deterministic Policy Gradient algorithm (DDPG) for an offline-trained DRL model. We then enhance this method, in terms of convergence and performance, by using distributed experiences collected from different APs based on the Distributed Distributional Deterministic Policy Gradients algorithm (D4PG) in which the APs represent the distributed agents of the DRL model. To reduce the complexity of signal processing at the central processing unit (CPU), we propose a fully distributed DRL-based uplink beamforming scheme. This scheme divides the beamforming computations among distributed APs. The proposed schemes are then benchmarked against two common linear beamforming schemes, namely, minimum mean square estimation (MMSE) and the simplified conjugate symmetric schemes. The results show that the D4PG scheme with distributed experience achieves the best performance irrespective of the network size. Furthermore, although the proposed distributed beamforming technique reduces the complexity of centralized learning in the DDPG algorithm, it performs better than the DDPG algorithm only for small-scale networks. The performance superiority of the fully centralized DDPG model becomes more evident as the number of APs and/or UEs increases. The codes for all of our DRL implementations are available at <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><uri>https://github.com/RayRedd/Distributed_beamforming_rl</uri></monospace> .

Distributed Resource Scheduling for Large-Scale MEC Systems: A Multiagent Ensemble Deep Reinforcement Learning With Imitation Acceleration

In large-scale mobile edge computing (MEC) systems, the task latency, and energy consumption are important for massive resource-consuming and delay-sensitive Internet of Things Devices (IoTDs). Against this background, we propose a distributed intelligent resource scheduling (DIRS) framework to minimize the sum of task latency and energy consumption for all IoTDs, which can be formulated as a mixed-integer nonlinear programming. The DIRS framework includes centralized training relying on the global information and distributed decision making by each agent deployed in each MEC server. Specifically, we first introduce a novel multiagent ensemble-assisted distributed deep reinforcement learning (DRL) architecture, which can simplify the overall neural network structure of each agent by partitioning the state space and also improve the performance of a single agent by combining decisions of all the agents. Second, we apply action refinement to enhance the exploration ability of the proposed DIRS framework, where the near-optimal state-action pairs are obtained by a novel Levy flight search. Finally, an imitation acceleration scheme is presented to pretrain all the agents, which can significantly accelerate the learning process of the proposed framework through learning the professional experience from a small amount of demonstration data. The simulation results in three typical scenarios demonstrate that the proposed DIRS framework is efficient and outperforms the existing benchmark schemes.