Reinforcement Learning Based Resource Allocation Research Articles

Mobile Edge Computing Against Smart Attacks with Deep Reinforcement Learning in Cognitive MIMO IoT Systems

In wireless Internet of Things (IoT) systems, the multi-input multi-output (MIMO) and cognitive radio (CR) techniques are usually involved into the mobile edge computing (MEC) structure to improve the spectrum efficiency and transmission reliability. However, such a CR based MIMO IoT system will suffer from a variety of smart attacks from wireless environments, even the MEC servers in IoT systems are not secure enough and vulnerable to these attacks. In this paper, we investigate a secure communication problem in a cognitive MIMO IoT system comprising of a primary user (PU), a secondary user (SU), a smart attacker and several MEC servers. The target of our system design is to optimize utility of the SU, including its efficiency and security. The SU will choose an idle MEC server that is not occupied by the PU in the CR scenario, and allocates a proper offloading rate of its computation tasks to the server, by unloading such tasks with proper transmit power. In such a CR IoT system, the attacker will select one type of smart attacks. Then two deep reinforcement learning based resource allocation strategies are proposed to find an optimal policy of maximal utility without channel state information(CSI), one of which is the Dyna architecture and Prioritized sweeping based Edge Server Selection (DPESS) strategy, and the other is the Deep Q-network based Edge Server Selection (DESS) strategy. Specifically, the convergence speed of the DESS scheme is significantly improved due to the trained convolutional neural network (CNN) by utilizing the experience replay technique and stochastic gradient descent (SGD). In addition, the Nash equilibrium and existence conditions of the proposed two schemes are theoretically deduced for the modeled MEC game against smart attacks. Compared with the traditional Q-learning algorithm, the average utility and secrecy capacity of the SU can be improved by the proposed DPESS and DESS schemes. Numerical simulations are also presented to verify the better performance of our proposals in terms of efficiency and security, including the higher convergence speed of the DESS strategy.

Deep Reinforcement Learning-based resource allocation strategy for Energy Harvesting-Powered Cognitive Machine-to-Machine Networks

Machine-to-Machine (M2M) communication is a promising technology that may realize the Internet of Things (IoTs) in future networks. However, due to the features of massive devices and concurrent access requirement, it will cause performance degradation and enormous energy consumption. Energy Harvesting-Powered Cognitive M2M Networks (EH-CMNs) as an attractive solution is capable of alleviating the escalating spectrum deficient to guarantee the Quality of Service (QoS) meanwhile decreasing the energy consumption to achieve Green Communication (GC) became an important research topic. In this paper, we investigate the resource allocation problem for EH-CMNs underlaying cellular uplinks. We aim to maximize the energy efficiency of EH-CMNs with consideration of the QoS of Human-to-Human (H2H) networks and the available energy in EH-devices. In view of the characteristic of EH-CMNs, we formulate the problem to be a decentralized Discrete-time and Finite-state Markov Decision Process (DFMDP), in which each device acts as agent and effectively learns from the environment to make allocation decision without the complete and global network information. Owing to the complexity of the problem, we propose a Deep Reinforcement Learning (DRL)-based algorithm to solve the problem. Numerical results validate that the proposed scheme outperforms other schemes in terms of average energy efficiency with an acceptable convergence speed.

Resource Allocation for Cloud-Based Software Services Using Prediction-Enabled Feedback Control With Reinforcement Learning

With time-varying workloads and service requests, cloud-based software services necessitate adaptive resource allocation for guaranteeing Quality-of-Service (QoS) and reducing resource costs. However, due to the ever-changing system states, resource allocation for cloud-based software services faces huge challenges in dynamics and complexity. The traditional approaches mostly rely on expert knowledge or numerous iterations, which might lead to weak adaptiveness and extra costs. Moreover, existing RL-based methods target the environment with the fixed workload, and thus they are unable to effectively fit in the real-world scenarios with variable workloads. To address these important challenges, we propose a Prediction-enabled feedback Control with Reinforcement learning based resource Allocation (PCRA) method. First, a novel Q-value prediction model is designed to predict the values of management operations (by Q-values) at different system states. The model uses multiple prediction learners for making accurate Q-value prediction by integrating the Q-learning algorithm. Next, the objective resource allocation plans can be found by using a new feedback-control based decision-making algorithm. Using the RUBiS benchmark, simulation results demonstrate that the PCRA chooses the management operations of resource allocation with 93.7 percent correctness. Moreover, the PCRA achieves optimal/near-optimal performance, and it outperforms the classic ML-based and rule-based methods by 5<inline-formula><tex-math notation="LaTeX">$\sim$</tex-math><alternatives><mml:math><mml:mo>&#x223C;</mml:mo></mml:math><inline-graphic xlink:href="chen-ieq1-2992537.gif"/></alternatives></inline-formula>7&#x0025; and 10<inline-formula><tex-math notation="LaTeX">$\sim$</tex-math><alternatives><mml:math><mml:mo>&#x223C;</mml:mo></mml:math><inline-graphic xlink:href="chen-ieq2-2992537.gif"/></alternatives></inline-formula>13&#x0025;, respectively.

Deep Reinforcement Learning Based Resource Allocation For Narrowband Cognitive Radio-IoT Systems

Abstract Narrowband Internet-of-Things (NB-IoT) is a low-power wide area (LPWA) technology developed by the Third-generation Partnership Project (3GPP) with objective to enable a wide range of IoT devices, low cost device and low power in the 5G era. As the number of IoT devices continue to increase, the demand for the spectrum allocation grows proportionately. The NB-IoT spectrum allocation is limited from 180 KHz to 200 KHz and is not sufficient to accomodate the exponential surge in the size of the NB-IoT devices.Thus, the need to efficiently allocate the available spectrum to the NB-IoT devices. Furthermore, in an attempt to enhance the coverage in NB-IoT network, recent relevant studies (3GPP release 13) have introduced the concept of repeated transmission. Since repeated transmissions ensure coverage enhancement but cause spectrum wastage, the traditional resource allocation is not appropriate for NB-IoT network. Motivated by this research gap we propose a NB-Cognitive Radio-IoT (NB-CR-IoT) technique which integrates Cognitive Radio (CR) techniques into the operation of the conventional NB-IoT. The resulting architecture seeks to foster an efficient opportunistic spectrum access in distributed heterogeneous networks.We further formulate the resource allocation problem as a deep Q-learning solved by reducing the number of repeated transmissions and allocating more IoT devices in NB-IoT network. The results in this contribution indicate that DQN outperforms the traditional Q-learning algorithm.

Multi-Agent Reinforcement Learning-Based Resource Allocation for UAV Networks

Unmanned aerial vehicles (UAVs) are capable of serving as aerial base stations (BSs) for providing both cost-effective and on-demand wireless communications. This article investigates dynamic resource allocation of multiple UAVs enabled communication networks with the goal of maximizing long-term rewards. More particularly, each UAV communicates with a ground user by automatically selecting its communicating users, power levels and subchannels without any information exchange among UAVs. To model the uncertainty of environments, we formulate the long-term resource allocation problem as a stochastic game for maximizing the expected rewards, where each UAV becomes a learning agent and each resource allocation solution corresponds to an action taken by the UAVs. Afterwards, we develop a multi-agent reinforcement learning (MARL) framework that each agent discovers its best strategy according to its local observations using learning. More specifically, we propose an agent-independent method, for which all agents conduct a decision algorithm independently but share a common structure based on Q-learning. Finally, simulation results reveal that: 1) appropriate parameters for exploitation and exploration are capable of enhancing the performance of the proposed MARL based resource allocation algorithm; 2) the proposed MARL algorithm provides acceptable performance compared to the case with complete information exchanges among UAVs. By doing so, it strikes a good tradeoff between performance gains and information exchange overheads.

Deep Reinforcement Learning Based Resource Allocation for V2V Communications

In this paper, we develop a novel decentralized resource allocation mechanism for vehicle-to-vehicle (V2V) communications based on deep reinforcement learning, which can be applied to both unicast and broadcast scenarios. According to the decentralized resource allocation mechanism, an autonomous “agent,” a V2V link or a vehicle, makes its decisions to find the optimal sub-band and power level for transmission without requiring or having to wait for global information. Since the proposed method is decentralized, it incurs only limited transmission overhead. From the simulation results, each agent can effectively learn to satisfy the stringent latency constraints on V2V links while minimizing the interference to vehicle-to-infrastructure communications.

Reinforcement Learning-based Content-Centric Services in Mobile Sensing

The recent remarkable advancement of smart devices is enabling a higher-level flexibility of mobile sensing. Along with the rapid development of mobile devices and applications, a challenging issue is becoming more serious than ever before. A large number of mobility-based services have brought heavy workloads to mobile devices. Resource outsourcing via resource allocations is a type of method to mitigate local workloads. However, most current solutions are restricted by two issues, namely, the variety of inputs and the contradiction between optimal outputs and latency. In this article, we utilize the mechanism of Reinforcement Learning (RL) and propose a novel approach, named Smart Reinforcement Learning-based Resource Allocation (SRL-RA), to achieve optimal allocation through a self-learning process.

A Reinforcement Learning-Based Resource Allocation Scheme for Cloud Robotics

In recent years, robotic systems combined with cloud computing capability have become an emerging topic of discussion in academic fields. The concept of cloud robotics allows the system to offload computing-intensive tasks from the robots to the cloud. An appropriate resource allocation scheme is necessary for the cloud computing service platform to efficiently allocate its computing resources, when the robots send requests asking for computing service. This paper proposes a resource allocation scheme based on reinforcement learning (RL), which can make the cloud to decide whether a request should be accepted and how many resources are supposed to be allocated. The scheme realizes an autonomous management of computing resources through online learning, reduces human participation in scheme planning, and improves the overall utility of the system in the long run. Numerical results demonstrate that the proposed RL-based computing resource allocation scheme has better performances than the greedy allocation scheme.

Reinforcement learning based resource allocation in business process management

Efficient resource allocation is a complex and dynamic task in business process management. Although a wide variety of mechanisms are emerging to support resource allocation in business process execution, these approaches do not consider performance optimization. This paper introduces a mechanism in which the resource allocation optimization problem is modeled as Markov decision processes and solved using reinforcement learning. The proposed mechanism observes its environment to learn appropriate policies which optimize resource allocation in business process execution. The experimental results indicate that the proposed approach outperforms well known heuristic or hand-coded strategies, and may improve the current state of business process management.