Reinforcement Learning-based Resource Management Research Articles

Reinforcement Learning Based Resource Management for 6G-Enabled mIoT With Hypergraph Interference Model

Reinforcement Learning Based Resource Management for 6G-Enabled mIoT With Hypergraph Interference Model

Reinforcement Learning Based Resource Management for CAR T-Cell Therapies

This paper focuses on optimizing resource management strategies in chimeric antigen receptor (CAR) T-cell therapies using reinforcement learning (RL). CAR T-cell therapy is an innovative and promising treatment within the field of immunotherapy, which is based on the isolation of patients’ own T-cells, genetically modifying these cells to express a CAR for tumor recognition, cultivating and expanding these T cells and infusing them back to the patient. These therapies require several reusable but scarce resources, including special equipment, such as the bioreactor that is used to expand CAR T cells, and medical resources, such as the hospital staff, e.g., doctors and nurses. Considering the stochastic nature of medical procedures and the production of CAR T cells, the scheduling of the therapies and the efficient allocation of the required resources pose a significant challenge inside the hospital environment. Here, we propose a derivative-free policy gradient algorithm that utilizes a simulation model of the therapy, built using real-world data, to obtain efficient control policies for the resource management problem. The proposed method is designed to minimize the expected number of deviations from the therapy protocol as well as the expected overall completion time of the therapies for every patient. The effectiveness of the proposed resource management approach is demonstrated via simulation experiments.

Multi-agent deep reinforcement learning based resource management in SWIPT enabled cellular networks with H2H/M2M co-existence

Machine-to-Machine (M2M) communication is crucial in developing Internet of Things (IoT). As it is well known that cellular networks have been considered as the primary infrastructure for M2M communications, there are several key issues to be addressed in order to deploy M2M communications over cellular networks. Notably, the rapid growth of M2M traffic dramatically increases energy consumption, as well as degrades the performance of existing Human-to-Human (H2H) traffic. Sustainable operation technology and resource management are efficacious ways for solving these issues. In this paper, we investigate a resource management problem in cellular networks with H2H/M2M coexistence. First, considering the energy-constrained nature of machine type communication devices (MTCDs), we propose a novel network model enabled by simultaneous wireless information and power transfer (SWIPT), which empowers MTCDs with the ability to simultaneously perform energy harvesting (EH) and information decoding. Given the diverse characteristics of IoT devices, we subdivide MTCDs into critical and tolerable types, further formulating the resource management problem as an energy efficiency (EE) maximization problem under divers Quality-of-Service (QoS) constraints. Then, we develop a multi-agent deep reinforcement learning (DRL) based scheme to solve this problem. It provides optimal spectrum, transmit power and power splitting (PS) ratio allocation policies, along with efficient model training under designed behavior-tracking based state space and common reward function. Finally, we verify that with a reasonable training mechanism, multiple M2M agents successfully work cooperatively in a distributed way, resulting in network performance that outperforms other intelligence approaches in terms of convergence speed and meeting the EE and QoS requirements.

Multi-agent deep reinforcement learning based resource management in heterogeneous V2X networks

In Heterogeneous Vehicle-to-Everything Networks (HVNs), multiple users such as vehicles and handheld devices and infrastructure can communicate with each other to obtain more advanced services. However, the increasing number of entities accessing HVNs presents a huge technical challenge to allocate the limited wireless resources. Traditional model-driven resource allocation approaches are no longer applicable because of rich data and the interference problem of multiple communication modes reusing resources in HVNs. In this paper, we investigate a wireless resource allocation scheme including power control and spectrum allocation based on the resource block reuse strategy. To meet the high capacity of cellular users and the high reliability of Vehicle-to-Vehicle (V2V) user pairs, we propose a data-driven Multi-Agent Deep Reinforcement Learning (MADRL) resource allocation scheme for the HVN. Simulation results demonstrate that compared to existing algorithms, the proposed MADRL-based scheme achieves a high sum capacity and probability of successful V2V transmission, while providing close-to-limit performance.

A Deep Reinforcement Learning-Based Resource Management Game in Vehicular Edge Computing

Vehicular Edge Computing (VEC) is a promising paradigm that leverages the vehicles to offload computation tasks to the nearby VEC server with the aim of supporting the low latency vehicular application scenarios. Incentivizing VEC servers to participate in computation offloading activities and make full use of computation resources is of great importance to the success of intelligent transportation services. In this paper, we formulate the competitive interactions between the VEC servers and vehicles as a two-stage Stackelberg game with the VEC servers as the leader players and the vehicles as the followers. After obtaining the full information of vehicles, the VEC server calculates the unit price of computation resource. Given the unit prices announced by VEC server, the vehicles determine the amount of computation resource to purchase from VEC server. In the scenario that vehicles do not want to share their computation demands, a deep reinforcement learning based resource management scheme is proposed to maximize the profits of vehicles and VEC server. The extensive experimental results have demonstrated the effectiveness of our proposed resource management scheme based on Stackelberg game and deep reinforcement learning.

Reinforcement learning based resource management for fog computing environment: Literature review, challenges, and open issues

In the IoT-based systems, the fog computing allows the fog nodes to offload and process tasks requested from IoT-enabled devices in a distributed manner instead of the centralized cloud servers to reduce the response delay. However, achieving such a benefit is still challenging in the systems with high rate of requests, which imply long queues of tasks in the fog nodes, thus exposing probably an inefficiency in terms of latency to offload the tasks. In addition, a complicated heterogeneous degree in the fog environment introduces an additional issue that many of single fogs can not process heavy tasks due to lack of available resources or limited computing capabilities. Reinforcement learning is a rising component of machine learning, which provides intelligent decision making for agents to response effectively to the dynamics of environment. This vision implies a great potential of application of RL in the concept of fog computing regarding resource allocation for task offloading and execution to achieve the improved performance. This work presents an overview of RL applications to solve the resource allocation related problems in the fog computing environment. The open issues and challenges are explored and discussed for further study.

Deep Reinforcement Learning Based Resource Management for DNN Inference in Industrial IoT

Performing deep neural network (DNN) inference in real time requires excessive network resources, which poses a big challenge to the resource-limited industrial Internet of things (IIoT) networks. To address the challenge, in this paper, we introduce an end-edge-cloud orchestration architecture, in which the inference task assignment and DNN model placement are flexibly coordinated. Specifically, the DNN models, trained and pre-stored in the cloud, are properly placed at the end and edge to perform DNN inference. To achieve efficient DNN inference, a multi-dimensional resource management problem is formulated to maximize the average inference accuracy while satisfying the strict delay requirements of inference tasks. Due to the mix-integer decision variables, it is difficult to solve the formulated problem directly. Thus, we transform the formulated problem into a Markov decision process which can be solved efficiently. Furthermore, a deep reinforcement learning based resource management scheme is proposed to make real-time optimal resource allocation decisions. Simulation results are provided to demonstrate that the proposed scheme can efficiently allocate the available spectrum, caching, and computing resources, and improve average inference accuracy by 31.4$\%$ compared with the deep deterministic policy gradient benchmark.

Deep Reinforcement Learning Based Resource Management in UAV-Assisted IoT Networks

The resource management in wireless networks with massive Internet of Things (IoT) users is one of the most crucial issues for the advancement of fifth-generation networks. The main objective of this study is to optimize the usage of resources for IoT networks. Firstly, the unmanned aerial vehicle is considered to be a base station for air-to-ground communications. Secondly, according to the distribution and fluctuation of signals; the IoT devices are categorized into urban and suburban clusters. This clustering helps to manage the environment easily. Thirdly, real data collection and preprocessing tasks are carried out. Fourthly, the deep reinforcement learning approach is proposed as a main system development scheme for resource management. Fifthly, K-means and round-robin scheduling algorithms are applied for clustering and managing the users’ resource requests, respectively. Then, the TensorFlow (python) programming tool is used to test the overall capability of the proposed method. Finally, this paper evaluates the proposed approach with related works based on different scenarios. According to the experimental findings, our proposed scheme shows promising outcomes. Moreover, on the evaluation tasks, the outcomes show rapid convergence, suitable for heterogeneous IoT networks, and low complexity.

Multi-Agent Reinforcement Learning Based Resource Management in MEC- and UAV-Assisted Vehicular Networks

In this paper, we investigate multi-dimensional resource management for unmanned aerial vehicles (UAVs) assisted vehicular networks. To efficiently provide on-demand resource access, the macro eNodeB and UAV, both mounted with multi-access edge computing (MEC) servers, cooperatively make association decisions and allocate proper amounts of resources to vehicles. Since there is no central controller, we formulate the resource allocation at the MEC servers as a distributive optimization problem to maximize the number of offloaded tasks while satisfying their heterogeneous quality-of-service (QoS) requirements, and then solve it with a multi-agent deep deterministic policy gradient (MADDPG)-based method. Through centrally training the MADDPG model offline, the MEC servers, acting as learning agents, then can rapidly make vehicle association and resource allocation decisions during the online execution stage. From our simulation results, the MADDPG-based method can converge within 200 training episodes, comparable to the single-agent DDPG (SADDPG)-based one. Moreover, the proposed MADDPG-based resource management scheme can achieve higher delay/QoS satisfaction ratios than the SADDPG-based and random schemes.

Reinforcement Learning Based Resource Management for Network Slicing

Network slicing to create multiple virtual networks, called network slice, is a promising technology to enable networking resource sharing among multiple tenants for the 5th generation (5G) networks. By offering a network slice to slice tenants, network slicing supports parallel services to meet the service level agreement (SLA). In legacy networks, every tenant pays a fixed and roughly estimated monthly or annual fee for shared resources according to a contract signed with a provider. However, such a fixed resource allocation mechanism may result in low resource utilization or violation of user quality of service (QoS) due to fluctuations in the network demand. To address this issue, we introduce a resource management system for network slicing and propose a dynamic resource adjustment algorithm based on reinforcement learning approach from each tenant’s point of view. First, the resource management for network slicing is modeled as a Markov Decision Process (MDP) with the state space, action space, and reward function. Then, we propose a Q-learning-based dynamic resource adjustment algorithm that aims at maximizing the profit of tenants while ensuring the QoS requirements of end-users. The numerical simulation results demonstrate that the proposed algorithm can significantly increase the profit of tenants compared to existing fixed resource allocation methods while satisfying the QoS requirements of end-users.