Deep Reinforcement Learning-Based Resource Research Articles

Deep Reinforcement Learning-Based Resource Management in Maritime Communication Systems.

With the growing maritime economy, ensuring the quality of communication for maritime users has become imperative. The maritime communication system based on nearshore base stations enhances the communication rate of maritime users through dynamic resource allocation. A virtual queue-based deep reinforcement learning beam allocation scheme is proposed in this paper, aiming to maximize the communication rate. More particularly, to reduce the complexity of resource management, we employ a grid-based method to discretize the maritime environment. For the combinatorial optimization problem of grid and beam allocation under unknown channel state information, we model it as a sequential decision process of resource allocation. The nearshore base station is modeled as a learning agent, continuously interacting with the environment to optimize beam allocation schemes using deep reinforcement learning techniques. Furthermore, we guarantee that grids with poor channel state information can be serviced through the virtual queue method. Finally, the simulation results provided show that our proposed beam allocation scheme is beneficial in terms of increasing the communication rate.

Deep Reinforcement Learning-Based Resource Allocation for RSMA in LEO Satellite-Terrestrial Networks

Deep Reinforcement Learning-Based Resource Allocation for RSMA in LEO Satellite-Terrestrial Networks

Deep Reinforcement Learning-Based Resource Allocation for Multi-UAV-Assisted Full-Duplex Wireless-Powered IoT Networks

Deep Reinforcement Learning-Based Resource Allocation for Multi-UAV-Assisted Full-Duplex Wireless-Powered IoT Networks

Deep Reinforcement Learning-Based Resource Allocation for UAV-Enabled Federated Edge Learning

Deep Reinforcement Learning-Based Resource Allocation for UAV-Enabled Federated Edge Learning

A Deep Reinforcement Learning-Based Resource Scheduler for Massive MIMO Networks

A Deep Reinforcement Learning-Based Resource Scheduler for Massive MIMO Networks

Differential Grey Wolf Load-Balanced Stochastic Bellman Deep Reinforced Resource Allocation in Fog Environment

Fog computing is becoming a dynamic and sought-after computing prototype for Internet of Things (IoT) application deployments. It works in conjunction with the cloud computing environment. Load balancing, which is employed by IoT applications when deciding, which fog or cloud computing nodes to use, is one of the most critical components for enhancing resource efficiency and avoiding problems like overloading or underloading. However, for IoT applications, ensuring that all CPU nodes are evenly distributed in terms of latency and energy utilization remains a challenge. To solve these issues, this work introduces Differential Grey Wolf (DGW) load balancing with stochastic Bellman deep reinforced resource optimization (DGW-SBDR) in fog situations. A Differential Evolution-based Grey Wolf Optimization algorithm based on load balancing has been developed for optimal resource management. The Grey Wolf Optimization algorithm, which employs differential evolution, assigns jobs to virtual machines based on user demands (VMs). In the event of an overutilized VM pool, a grey wolf optimization strategy based on differential evolution can detect both under and overutilized VMs, allowing for smooth transit between fogs. This step disables a number of virtual machines in order to reduce latency. In a Stochastic Gradient and Deep Reinforcement Learning-based Resource Allocation Model, a stochastic gradient bellman optimality function and Deep Reinforcement Learning are integrated for optimal resource allocation. According to the proposed method, QoS may be supplied to end-users by reducing energy consumption and better managing cache resources utilizing stochastic gradient bellman optimality.

Deep Reinforcement Learning-based Resource Allocation for 5G Machine-type Communication in Active Distribution Networks with Time-varying Interference

Deep Reinforcement Learning-based Resource Allocation for 5G Machine-type Communication in Active Distribution Networks with Time-varying Interference

DRAM: A DRL-based resource allocation scheme for MAR in MEC

Mobile Edge Computing (MEC) and 5G technology allow clients to access computing resources at the network frontier, which paves the way for applying Mobile Augmented Reality (MAR) applications. Under the MEC paradigm, MAR clients can offload complex tasks to the MEC server and enhance the human perception of the world by merging the received virtual information with the real environment. However, the resource allocation problem arises as a critical challenge in circumstances where several MAR clients compete for limited resources at the network frontier. In this paper, we aim to design an online resource allocation scheme on the MEC server that takes both high quality of experience and good fairness performance for MAR clients into consideration. We first formulate this problem as a Markov decision process and tackle the challenge of applying the deep reinforcement learning paradigm. Then, we propose DRAM, a Deep reinforcement learning-based Resource allocation scheme for mobile Augmented reality service in MEC. We also propose a self-adaptive algorithm on the MAR client that is derived based on the analysis of the MAR service to tackle client adaptation problems. The simulation results demonstrated that DRAM can provide high quality of experience and simultaneously achieve good fairness performance by coordinating with clients’ adaptation algorithms.

Deep Reinforcement Learning-Based Resource Allocation for Satellite Internet of Things with Diverse QoS Guarantee.

Large-scale terminals’ various QoS requirements are key challenges confronting the resource allocation of Satellite Internet of Things (S-IoT). This paper presents a deep reinforcement learning-based online channel allocation and power control algorithm in an S-IoT uplink scenario. The intelligent agent determines the transmission channel and power simultaneously based on contextual information. Furthermore, the weighted normalized reward concerning success rate, power efficiency, and QoS requirement is adopted to balance the performance between increasing resource efficiency and meeting QoS requirements. Finally, a practical deployment mechanism based on transfer learning is proposed to promote onboard training efficiency and to reduce computation consumption of the training process. The simulation demonstrates that the proposed method can balance the success rate and power efficiency with QoS requirement guaranteed. For S-IoT’s normal operation condition, the proposed method can improve the power efficiency by 60.91% and 144.44% compared with GA and DRL_RA, while its power efficiency is only 4.55% lower than that of DRL-EERA. In addition, this method can be transferred and deployed to a space environment by merely 100 onboard training steps.

Deep Reinforcement Learning-Based Resource Allocation for Cellular Vehicular Network Mode 3 with Underlay Approach.

Vehicle-to-vehicle (V2V) communication has attracted increasing attention since it can improve road safety and traffic efficiency. In the underlay approach of mode 3, the V2V links need to reuse the spectrum resources preoccupied with vehicle-to-infrastructure (V2I) links, which will interfere with the V2I links. Therefore, how to allocate wireless resources flexibly and improve the throughput of the V2I links while meeting the low latency requirements of the V2V links needs to be determined. This paper proposes a V2V resource allocation framework based on deep reinforcement learning. The base station (BS) uses a double deep Q network to allocate resources intelligently. In particular, to reduce the signaling overhead for the BS to acquire channel state information (CSI) in mode 3, the BS optimizes the resource allocation strategy based on partial CSI in the framework of this article. The simulation results indicate that the proposed scheme can meet the low latency requirements of V2V links while increasing the capacity of the V2I links compared with the other methods. In addition, the proposed partial CSI design has comparable performance to complete CSI.

Multiagent Deep-Reinforcement-Learning-Based Resource Allocation for Heterogeneous QoS Guarantees for Vehicular Networks

Vehicle-to-vehicle communications can offer direct information interaction, including security-centered information and entertainment information. However, the rapid proliferation of vehicles and the diversity of communications services demand for a more intelligent and efficient resource allocation framework to enhance network performance. In this article, a multi-agent deep reinforcement learning-based resource allocation framework is developed to jointly optimize the channel allocation and power control to satisfy the heterogeneous Quality-of-Service (QoS) requirements in heterogeneous vehicular networks. In the proposed framework, the utility maximization problem is formulated by considering two types of traffics, i.e., the strict ultrareliable and low-latency requirements for safety-centric applications and the high-capacity requirements for entertainment applications. The utility of each vehicular users is formulated as a multicriterion objective function by taking into account the heterogeneous traffic requirements. To overcome the drawbacks of the traditional totally centralized and distributed deep reinforcement learning-based resource allocation approaches, we propose a multi-agent deep deterministic policy gradient algorithm with centralized learning and decentralized execution to solve the formulated optimization problem. The normalization of the input states and reward functions is introduced to speed up the training and learning progress of the proposed algorithm. Simulation results show the superiority of the proposed algorithm in terms of the convergence and system performance through the comparison with the other methods and schemes for the delay-sensitive applications and delay-tolerant applications.

Multi-Agent Deep Reinforcement Learning-Based Resource Allocation in HPC/AI Converged Cluster

As the complexity of deep learning (DL) networks and training data grows enormously, methods that scale with computation are becoming the future of artificial intelligence (AI) development. In this regard, the interplay between machine learning (ML) and high-performance computing (HPC) is an innovative paradigm to speed up the efficiency of AI research and development. However, building and operating an HPC/AI converged system require broad knowledge to leverage the latest computing, networking, and storage technologies. Moreover, an HPC-based AI computing environment needs an appropriate resource allocation and monitoring strategy to efficiently utilize the system resources. In this regard, we introduce a technique for building and operating a high-performance AI-computing environment with the latest technologies. Specifically, an HPC/AI converged system is configured inside Gwangju Institute of Science and Technology (GIST), called GIST AI-X computing cluster, which is built by leveraging the latest Nvidia DGX servers, high-performance storage and networking devices, and various open source tools. Therefore, it can be a good reference for building a small or middle-sized HPC/AI converged system for research and educational institutes. In addition, we propose a resource allocation method for DL jobs to efficiently utilize the computing resources with multi-agent deep reinforcement learning (mDRL). Through extensive simulations and experiments, we validate that the proposed mDRL algorithm can help the HPC/AI converged cluster to achieve both system utilization and power consumption improvement. By deploying the proposed resource allocation method to the system, total job completion time is reduced by around 20% and inefficient power consumption is reduced by around 40%.

Deep Reinforcement Learning-Based Resource Allocation in Cooperative UAV-Assisted Wireless Networks

We consider the downlink of an unmanned aerial vehicle (UAV) assisted cellular network consisting of multiple cooperative UAVs, whose operations are coordinated by a central ground controller using wireless fronthaul links, to serve multiple ground user equipments (UEs). A problem of jointly designing UAVs’ positions, transmit beamforming, as well as UAV-UE association is formulated in the form of mixed integer nonlinear programming (MINLP) to maximize the sum UEs’ achievable rate subject to limited fronthaul capacity constraints. Solving the considered problem is hard owing to its non-convexity and the unavailability of channel state information (CSI) due to the movement of UAVs. To tackle these effects, we propose a novel algorithm comprising of two distinguishing features: (i) exploiting a deep Q-learning approach to tackle the issue of CSI unavailability for determining UAVs’ positions, (ii) developing a difference of convex algorithm (DCA) to efficiently solve for the UAV’s transmit beamforming and UAV-UE association. The proposed algorithm recursively solves the problem of interest until convergence, where each recursion executes two steps. In the first step, the deep Q-learning (DQL) algorithm allows UAVs to learn the overall network state and account for the joint movement of all UAVs to adapt their locations. In the second step, given the determined UAVs’ positions from the DQL algorithm, the DCA iteratively solves a convex approximate subproblem of the original non-convex MINLP problem with the updated parameters, where the problem’s variables are transmit beamforming and UAV-UE association. Numerical results show that our design outperforms the existing algorithms in terms of algorithmic convergence and network performance with a gain of up to 70%.

Deep reinforcement learning-based resource allocation for D2D communications in heterogeneous cellular networks

Device-to-Device (D2D) communication-enabled Heterogeneous Cellular Networks (HCNs) have been a promising technology for satisfying the growing demands of smart mobile devices in fifth-generation mobile networks. The introduction of Millimeter Wave (mm-wave) communications into D2D-enabled HCNs allows higher system capacity and user data rates to be achieved. However, interference among cellular and D2D links remains severe due to spectrum sharing. In this paper, to guarantee user Quality of Service (QoS) requirements and effectively manage the interference among users, we focus on investigating the joint optimization problem of mode selection and channel allocation in D2D-enabled HCNs with mm-wave and cellular bands. The optimization problem is formulated as the maximization of the system sum-rate under QoS constraints of both cellular and D2D users in HCNs. To solve it, a distributed multiagent deep Q-network algorithm is proposed, where the reward function is redefined according to the optimization objective. In addition, to reduce signaling overhead, a partial information sharing strategy that does not observe global information is proposed for D2D agents to select the optimal mode and channel through learning. Simulation results illustrate that the proposed joint optimization algorithm possesses good convergence and achieves better system performance compared with other existing schemes.

Deep reinforcement learning-based resource allocation and seamless handover in multi-access edge computing based on SDN

Deep reinforcement learning-based resource allocation and seamless handover in multi-access edge computing based on SDN

ADRL: A Hybrid Anomaly-Aware Deep Reinforcement Learning-Based Resource Scaling in Clouds

The virtualization concept and elasticity feature of cloud computing enable users to request resources on-demand and in the pay-as-you-go model. However, the high flexibility of the model makes the on-time resource scaling problem more complex. A variety of techniques such as threshold-based rules, time series analysis, or control theory are utilized to increase the efficiency of dynamic scaling of resources. However, the inherent dynamicity of cloud-hosted applications requires autonomic and adaptable systems that learn from the environment in real-time. Reinforcement Learning (RL) is a paradigm that requires some agents to monitor the surroundings and regularly perform an action based on the observed states. RL has a weakness to handle high dimensional state space problems. Deep-RL models are a recent breakthrough for modeling and learning in complex state space problems. In this article, we propose a Hybrid Anomaly-aware Deep Reinforcement Learning-based Resource Scaling (ADRL) for dynamic scaling of resources in the cloud. ADRL takes advantage of anomaly detection techniques to increase the stability of decision-makers by triggering actions in response to the identified anomalous states in the system. Two levels of global and local decision-makers are introduced to handle the required scaling actions. An extensive set of experiments for different types of anomaly problems shows that ADRL can significantly improve the quality of service with less number of actions and increased stability of the system.

DESOLATER: Deep Reinforcement Learning-Based Resource Allocation and Moving Target Defense Deployment Framework

The recent development of autonomous driving technologies has led to the proliferation of research on sensors and electronic equipment inside a vehicle. To deal with security concerns of in-vehicle networks, various deep learning (DL) and reinforcement learning (RL) have been developed to enhance in-vehicle security. However, the DL/RL agents are vulnerable to adversarial perturbation, where an attacker can perform a manipulation attack to interfere with the agent’s operation. In this work, we aim to develop two key mechanisms to build secure in-vehicle networks: (1) RL-based proactive defense mechanism to achieve multiple objectives of minimizing system security vulnerabilities while maximizing service availability; and (2) a resilient RL method that allows an agent to operate in the presence of adversarial disturbances that neutralize the system security. To this end, we propose, DESOLATER ( D rl-based r ESO urce a L location A nd m T d d E ployment f R amework), which is a multi-agent deep reinforcement learning (mDRL)-based network slicing technique that can help determine two key network management decisions: (1) link bandwidth allocation to meet quality-of-service (QoS) requirements; and (2) the frequency of triggering IP shuffling as a proactive defense mechanism not to hinder service availability by maintaining normal system operations. We also introduce an anomaly detection mechanism with a memory-based RL technique to enhance the resiliency of the RL agents in a partially observable environment under the situation that adversarial attackers manipulating observation information. Through extensive simulation experiments, we validate that the proposed robust mDRL algorithm can help the deployed proactive security mechanism achieve both security and network performance improvement in the presence of adversarial attacks.

Deep Reinforcement Learning-Based Resource Scheduling Strategy for Reliability-Oriented Wireless Body Area Networks

Reliability is a critical factor in designing of wireless body area networks. In this letter, we propose a resource scheduling strategy and solving an optimization problem to maximize the reliability of the transmission of emergency-critical sensory data. We jointly consider transmission mode, relay selection, time slot allocation, and transmit power of each body sensor and formulating the scheduling problem to be a Markov decision process. In this strategy, the scheduling decision is made by each body sensor that do not have complete and global network information. Owning to the formulated problem is nonconvex and the high computation complexity, we propose a deep reinforcement learning algorithm to solve the problem. Numerical results reveal that the proposed strategy is capacity of guaranteeing the reliability of transmission with an acceptable convergence speed.

Deep Reinforcement Learning-Based Resource Allocation and Power Control in Small Cells With Limited Information Exchange

In multi-user downlink small cell networks, cooperative resource allocation (RA) within a small cell cluster is a key technique to enhance network capacity. However, capacity-maximizing RA in frequency-selective fading channels requires global channel state information (CSI) of users within a small cell cluster, which makes it infeasible in practical networks with limited direct link capacity. To circumvent this global CSI assumption, most of the existing studies on RA have been based on several CSI assumptions such as local CSI and local CSI at the transmitters (CSIT). Nevertheless, cost functions with local CSI or local CSIT in the literature rely on heuristic formulations, because the sum-rate cannot be computed if without global CSI. In this paper, we propose a deep reinforcement learning-based RA algorithm to maximize the sum-rate for any given limited information on instantaneous CSI or sum-rate at the previous period. The proposed scheme is not restricted to certain CSI assumptions, but attempts to find the best RA for any given information such as quantized local CSI and quantized local CSIT; thus, it is applicable to any given direct link capacity. The proposed algorithm is self-adaptive in time-varying channels, since it is not divided into training and test phases. We modify the target neural network (TNN) scheme to enhance the sum-rate and the convergence speed. Numerical simulations confirm that: i) the proposed algorithm outperforms the conventional algorithms even under the same CSI assumption such as local CSI and local CSIT; ii) a flexible trade-off between the amount of CSI and the sum-rate is realizable in practical systems.

Deep Reinforcement Learning-based resource allocation strategy for Energy Harvesting-Powered Cognitive Machine-to-Machine Networks

Machine-to-Machine (M2M) communication is a promising technology that may realize the Internet of Things (IoTs) in future networks. However, due to the features of massive devices and concurrent access requirement, it will cause performance degradation and enormous energy consumption. Energy Harvesting-Powered Cognitive M2M Networks (EH-CMNs) as an attractive solution is capable of alleviating the escalating spectrum deficient to guarantee the Quality of Service (QoS) meanwhile decreasing the energy consumption to achieve Green Communication (GC) became an important research topic. In this paper, we investigate the resource allocation problem for EH-CMNs underlaying cellular uplinks. We aim to maximize the energy efficiency of EH-CMNs with consideration of the QoS of Human-to-Human (H2H) networks and the available energy in EH-devices. In view of the characteristic of EH-CMNs, we formulate the problem to be a decentralized Discrete-time and Finite-state Markov Decision Process (DFMDP), in which each device acts as agent and effectively learns from the environment to make allocation decision without the complete and global network information. Owing to the complexity of the problem, we propose a Deep Reinforcement Learning (DRL)-based algorithm to solve the problem. Numerical results validate that the proposed scheme outperforms other schemes in terms of average energy efficiency with an acceptable convergence speed.