Markov Decision Problem Research Articles

Online Minimax Q Network Learning for Two-Player Zero-Sum Markov Games.

The Nash equilibrium is an important concept in game theory. It describes the least exploitability of one player from any opponents. We combine game theory, dynamic programming, and recent deep reinforcement learning (DRL) techniques to online learn the Nash equilibrium policy for two-player zero-sum Markov games (TZMGs). The problem is first formulated as a Bellman minimax equation, and generalized policy iteration (GPI) provides a double-loop iterative way to find the equilibrium. Then, neural networks are introduced to approximate Q functions for large-scale problems. An online minimax Q network learning algorithm is proposed to train the network with observations. Experience replay, dueling network, and double Q-learning are applied to improve the learning process. The contributions are twofold: 1) DRL techniques are combined with GPI to find the TZMG Nash equilibrium for the first time and 2) the convergence of the online learning algorithm with a lookup table and experience replay is proven, whose proof is not only useful for TZMGs but also instructive for single-agent Markov decision problems. Experiments on different examples validate the effectiveness of the proposed algorithm on TZMG problems.

Transfer reinforcement learning via meta-knowledge extraction using auto-pruned decision trees

Transfer reinforcement learning (RL) has recently received increasing attention to make RL agents have better learning performance in target Markov decision problems (MDPs) by using the knowledge learned in source MDPs. However, it is still an open and challenging problem to improve the transfer capability and interpretability of RL algorithms. In this paper, we propose a novel transfer reinforcement learning approach via meta-knowledge extraction using auto-pruned decision trees. In source MDPs, pre-trained policies are firstly learned via RL algorithms using general function approximators. Then, a meta-knowledge extraction algorithm is designed with an auto-pruned decision tree model, where the meta-knowledge is learned by re-training the auto-pruned decision tree based on the data samples generated from the pre-trained policies. The state spaces of meta-knowledge are determined by estimating the uncertainty of state–action pairs in pre-trained policies based on the entropy value of leaf nodes. In target MDPs, according to whether the state is in the state set of meta-knowledge, a hybrid policy is generated by integrating the meta-knowledge and the policies learned on the target MDPs. Based on the proposed transfer RL approach, two meta-knowledge-based transfer reinforcement learning (MKRL) algorithms are developed for MDPs with discrete action spaces and continuous action spaces, respectively. Experimental results in several benchmark tasks show that the MKRL algorithm outperforms other baselines in terms of learning efficiency and interpretability in the target MDPs with generic cases of task similarity.

Optimal pricing in multi server systems

We study optimal service pricing in server farms where customers arrive according to a renewal process and have independent and identical (i.i.d.) exponential service times and i.i.d. valuations of the service. The service provider charges a time varying service fee aiming at maximizing its revenue rate. The customers that find free servers and service fees lesser than their valuation join for the service else they leave without waiting. We consider both finite server and infinite server farms. We solve the optimal pricing problems using the framework of Markov decision problems. We show that the optimal prices depend on the number of free servers. We propose algorithms to compute the optimal prices. We also establish several properties of the optimal prices and the corresponding revenue rates in the case of Poisson customer arrivals. We illustrate all our findings via numerical evaluation.

Fitted Value Iteration in Continuous MDPs With State Dependent Action Sets

In this letter, we establish the convergence of fitted value iteration and fitted Q-value iteration for continuous-state continuous-action Markov decision problems (MDPs) with state-dependent action sets. We further extend the algorithm and the convergence result to the case of monotone MDPs.

A Deep Learning Approach for Task Offloading in Multi-UAV Aided Mobile Edge Computing

Computation offloading has proven to be an effective method for facilitating resource-intensive tasks on IoT mobile edge nodes with limited processing capabilities. Additionally, in context of Mobile Edge Computing (MEC) systems, edge nodes can offload its computation-intensive tasks to a suitable edge server. Hence, they can reduce energy cost and speed up processing. Despite the numerous accomplished efforts in task offloading problems on the Internet of Things (IoT), this problem remains a research gap mainly because of its NP-hardness in addition to the unrealistic assumptions in many proposed solutions. In order to accurately extract information from raw sensor data from IoT devices deployed in complicated contexts, deep learning (DL) is a potential method. Therefore, in this paper, an approach based on deep reinforcement learning (DRL) will just be presented for optimize the offloading process for IoT in MEC environments. This approach can achieve the optimal offloading decision. A Markov Decision Problem (MDP) is used to formulate the offloading problem. Delay time and consumed energy are the main optimization targets in this work. The proposed approach has been verified using extensive simulations. Simulation results demonstrate that the proposed model effectively improves the MEC system latency, energy consumption, and significantly outperforms the Deep Q Networks (DQNs) and Actor Critic (AC) approaches.

Intelligent Drone-Base-Station Placement for Improved Revenue in B5G/6G Systems Under Uncertain Fluctuated Demands

In this paper, the drone base-station (DBS) dispatching problem in a multi-cell B5G/6G network is investigated. The main objective is to achieve the highest system profit by serving the largest possible number of users with the least possible cost while considering the uncertain time-dependent fluctuated user’s (service) demand in the different cells, the cost of dispatched drones, and the possible profit loss due to un-served users. The problem is formulated as a profit-maximization discount return problem. Due to the uncertainty in the demand (users) in each cell, the problem cannot be solved using conventional optimization methods. Hence, the problem is reformulated as a Markov decision problem (MDP). Due to the exponential complexity of finding the solution and the unavailability of statistical knowledge about user availability (demand) in the considered regions for such optimization, we adopt a reinforcement learning (RL) approach based on the state-action-reward-state-action (SARSA) algorithm to efficiently solve the MDP. Simulation results reveal that our RL-based approach significantly increases the overall operator profit by continuously adapting its DBS dispatching strategy based on the learned users’ behavior in the network, which enables serving a larger number of users (highest revenue) with least number of DBSs (least cost).

An autonomous agent for negotiation with multiple communication channels using parametrized deep Q-network

<abstract><p>Agent-based negotiation aims at automating the negotiation process on behalf of humans to save time and effort. While successful, the current research considers communication between negotiation agents through offer exchange. In addition to the simple manner, many real-world settings tend to involve linguistic channels with which negotiators can express intentions, ask questions, and discuss plans. The information bandwidth of traditional negotiation is therefore restricted and grounded in the action space. Against this background, a negotiation agent called MCAN (multiple channel automated negotiation) is described that models the negotiation with multiple communication channels problem as a Markov decision problem with a hybrid action space. The agent employs a novel deep reinforcement learning technique to generate an efficient strategy, which can interact with different opponents, i.e., other negotiation agents or human players. Specifically, the agent leverages parametrized deep Q-networks (P-DQNs) that provides solutions for a hybrid discrete-continuous action space, thereby learning a comprehensive negotiation strategy that integrates linguistic communication skills and bidding strategies. The extensive experimental results show that the MCAN agent outperforms other agents as well as human players in terms of averaged utility. A high human perception evaluation is also reported based on a user study. Moreover, a comparative experiment shows how the P-DQNs algorithm promotes the performance of the MCAN agent.</p></abstract>

Energy efficient management for distributed state estimation under DoS attacks

AbstractIn this article, we consider the issue of energy efficient management for distributed state estimation over unknown packet‐dropping communication channels which are exposed to Denial‐of‐Service (DoS) attacks. After a packet is transmitted, the sensor determines whether to retransmit and the retransmission power levels based on the feedback information from the receiving sensor. We formulate the energy efficient management problem as a modified Markov decision problem (MMDP), and proposed an algorithm to seek a retransmission power schedule that achieves optimal tradeoff between energy consumption and communication rate for given channel noise power. In the case of unknown channel state, we design a statistical learning method to estimate channel noise power. Besides, inspired by the hypothesis testing, we design an anomaly detector to guarantee the instantaneity of channel noise power estimation and efficiency of energy management. Furthermore, we derive the optimal gain for each estimator, and provide a sufficient condition for the stability of estimator under the energy management. Finally, we verify the main results by numerical examples.

Multi-objective vehicle rebalancing for ridehailing system using a reinforcement learning approach

The problem of designing a rebalancing algorithm for a large-scale ridehailing system with asymmetric (unbalanced) demand is considered here. We pose the rebalancing problem within a semi Markov decision problem (SMDP) framework with closed queues of vehicles serving stationary, but asymmetric demand, over a large city with multiple stations (representing neighborhoods). We assume that the passengers queue up at every station until they are matched with a vehicle. The goal of the SMDP is to minimize a convex combination of the waiting time of the passengers and the total empty vehicle miles traveled. The resulting SMDP appears to be difficult to solve yielding closed-form expression for the optimal rebalancing strategy. Consequently, we use a deep reinforcement learning algorithm to determine the approximately optimal solution to the SMDP. We show through extensive Monte Carlo simulations that the trained policy outperforms other well-known state-dependent rebalancing strategies.

Inventory rationing and replenishment for an omni-channel retailer

The growth of omni-channel retailing resulted in many new challenges for retailers, especially in relation to the replenishment and allocation of inventories for the different channels. In this paper we consider a brick-and-mortar store that uses its inventory to fulfil both in-store demand as well as online orders. In addition to deciding on replenishment quantities, such a retailer also has to decide how to ration its inventory across the channels. Practically, rationing inventory relates to storing part of the inventory in the backroom to satisfy online demand. The rationing process occurs regularly (e.g. daily) whereas inventory replenishment typically occurs less regular. To analyse this decision process, we model the rationing and ordering decisions as a Markov Decision Problem that maximises the expected profit. Based on the structure of the optimal policies, we determine heuristics that near-optimal results and scale well to retailers with many products.

Decentralized Decision for Multi-Band Sensing: A Deep Reinforcement Learning Approach

This letter focuses on seeking a robust decentralized solution of multi-band sensing-decision-making (MBSDM) for cognitive wireless networks (CWN). As the MBSDM process of each agent in CWN can be regarded as a Partially Observable Markov Decision Problem (POMDP), we propose a distributed MBSDM algorithm based on distributed reinforcement learning with Multi-Agent Deep Deterministic Policy Gradient (MADDPG) strategy to overcome the partial observability and prohibitive computation. The MADDPG is implemented with offline centralized training and online decentralized execution. For the centralized training, the network of each agent is trained offline with the mixed actions and state information in experience pool. While in decentralized implementation, each agent calculates the local observation based on its belief state and takes action by the well-trained network independently. Comparing with existing algorithms, simulation results showcase the effectiveness and robustness of the proposed MADDPG based decentralized MBSDM algorithm.

Dynamic Optimal Travel Strategies in Intelligent Stochastic Transit Networks

This paper addresses the search for a run-based dynamic optimal travel strategy, to be supplied through mobile devices (apps) to travelers on a stochastic multiservice transit network, which includes a system forecasting of bus travel times and bus arrival times at stops. The run-based optimal strategy is obtained as a heuristic solution to a Markovian decision problem. The hallmarks of this paper are the proposals to use only traveler state spaces and estimates of dispersion of forecast bus arrival times at stops in order to determine transition probabilities. The first part of the paper analyses some existing line-based and run-based optimal strategy search methods. In the second part, some aspects of dynamic transition probability computation in intelligent transit systems are presented, and a new method for dynamic run-based optimal strategy search is proposed and applied.

Wireless Body Area Network Control Policies for Energy-Efficient Health Monitoring.

Wireless body area networks (WBANs) have strong potential in the field of health monitoring. However, the energy consumption required for accurate monitoring determines the time between battery charges of the wearable sensors, which is a key performance factor (and can be critical in the case of implantable devices). In this paper, we study the inherent trade-off between the power consumption of the sensors and the probability of misclassifying a patient’s health state. We formulate this trade-off as a dynamic problem, in which at each step, we can choose to activate a subset of sensors that provide noisy measurements of the patient’s health state. We assume that the (unknown) health state follows a Markov chain, so our problem is formulated as a partially observable Markov decision problem (POMDP). We show that all the past measurements can be summarized as a belief state on the true health state of the patient, which allows tackling the POMDP problem as an MDP on the belief state. Then, we empirically study the performance of a greedy one-step look-ahead policy compared to the optimal policy obtained by solving the dynamic program. For that purpose, we use an open-source Continuous Glucose Monitoring (CGM) dataset of 232 patients over six months and extract the transition matrix and sensor accuracies from the data. We find that the greedy policy saves ≈50% of the energy costs while reducing the misclassification costs by less than 2% compared to the most accurate policy possible that always activates all sensors. Our sensitivity analysis reveals that the greedy policy remains nearly optimal across different cost parameters and a varying number of sensors. The results also have practical importance, because while the optimal policy is too complicated, a greedy one-step look-ahead policy can be easily implemented in WBAN systems.

Energy-Efficient UAV-Enabled Data Collection via Wireless Charging: A Reinforcement Learning Approach

In this article, we study the application of unmanned aerial vehicle (UAV) for data collection with wireless charging, which is crucial for providing seamless coverage and improving system performance in the next-generation wireless networks. To this end, we propose a reinforcement learning-based approach to plan the route of UAV to collect sensor data from sensor devices scattered in the physical environment. Specifically, the physical environment is divided into multiple grids, where one spot for UAV hovering as well as the wireless charging of UAV is located at the center of each grid. Each grid has a spot for the UAV to hover, and moreover, there is a wireless charger at the center of each grid, which can provide wireless charging to UAV when it is hovering in the grid. When the UAV lacks energy, it can be charged by the wireless charger at the spot. By taking into account the collected data amount as well as the energy consumption, we formulate the problem of data collection with UAV as a Markov decision problem, and exploit $Q$ -learning to find the optimal policy. In particular, we design the reward function considering the energy efficiency of UAV flight and data collection, based on which $Q$ -table is updated for guiding the route of UAV. Through extensive simulation results, we verify that our proposed reward function can achieve a better performance in terms of the average throughput, delay of data collection, as well as the energy efficiency of UAV, in comparison with the conventional capacity-based reward function.

Efficient Batch-Mode Reinforcement Learning Using Extreme Learning Machines

As a class of batch-mode reinforcement learning (RL) methods for Markov decision problems with large or continuous state spaces, approximate policy iteration (API) has received increasing attention in the past decades. One open problem in the design of API algorithms is how to construct the basis functions or features for value function approximation (VFA). In this paper, we propose a novel batch-mode RL approach with randomly projected features for VFA. The proposed approach can be viewed as an extension of extreme learning machines (ELMs) to RL problems so it can be called ELM-API. The ELMs have been popularly studied in supervised learning problems, but there is not much work on the extension of ELMs to learning control problems. The proposed approach has advantages over the previous API algorithms in that the features for VFA can be quickly generated without complex parameter selection and the performance will be adaptive to different sample sets in batch-mode RL. In particular, the ELM-API approach can realize fast and efficient feature reconstruction when training sample sets are relatively small. Comprehensive simulation studies on two benchmark learning control problems were carried out to test the performance of API algorithms with different feature construction methods. It is shown that the ELM-API algorithm can obtain comparable or better performance than the previous API approaches. To further show the effectiveness of ELM-API in real-world applications, the simulation results on a more challenging high-dimensional lane-changing decision problem in dynamic traffic environment are also reported, which show the capability of the ELM-API algorithm in learning satisfactory lane-changing policies with high data efficiency.

Detecting an Odd Restless Markov Arm With a Trembling Hand

In this paper, we consider a multi-armed bandit in which each arm is a Markov process evolving on a finite state space. The state space is common across the arms, and the arms are independent of each other. The transition probability matrix of one of the arms (the odd arm) is different from the common transition probability matrix of all the other arms. A decision maker, who knows these transition probability matrices, wishes to identify the odd arm as quickly as possible, while keeping the probability of decision error small. To do so, the decision maker collects observations from the arms by pulling the arms in a sequential manner, one at each discrete time instant. However, the decision maker has a trembling hand, and the arm that is actually pulled at any given time differs, with a small probability, from the one he intended to pull. The observation at any given time is the arm that is actually pulled and its current state. The Markov processes of the unobserved arms continue to evolve. This makes the arms restless. For the above setting, we derive the first known asymptotic lower bound on the expected time required to identify the odd arm, where the asymptotics is of vanishing error probability. The continued evolution of each arm adds a new dimension to the problem, leading to a family of Markov decision problems (MDPs) on a countable state space. We then stitch together certain parameterised solutions to these MDPs and obtain a sequence of strategies whose expected times to identify the odd arm come arbitrarily close to the lower bound in the regime of vanishing error probability. Prior works dealt with independent and identically distributed (across time) arms and rested Markov arms, whereas our work deals with restless Markov arms.

Distributed Deep Learning for Power Control in D2D Networks With Outdated Information

Traditional D2D power control methods require instantaneous interference information and so are difficult to implement in a real network due to the backhaul delay and high computational requirements. To overcome this challenge, we propose a distributed power allocation algorithm called interference feature extractor aided recurrent neural network (IFE-RNN). The core design ideas are described as follows. First, we design linear filters with various sizes termed IFEs to extract the different local interference patterns from outdated channel information. This feature extraction process enables our network to precisely learn the interference patterns around D2D links, so as to provide more effective power allocation strategies. Second, we propose to predict the real-time interference pattern based on the outputs of the IFEs and further make power decision. The prediction and decision can be modelled as a Markov decision problem (MDP) and solved by using a recurrent neural network. Third, an input reduction process is also designed to reduce the input size from O(N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) to O(1), which speeds up the operation time and reduces the system overhead. Finally, extensive simulation results show that the proposed algorithm achieves an encouraging performance compared to the state-of-the-art power allocation algorithm.

UAV-to-Device Underlay Communications: Age of Information Minimization by Multi-Agent Deep Reinforcement Learning

In recent years, unmanned aerial vehicles (UAVs) have unlocked numerous sensing applications, which are expected to add billions of dollars to the world economy in the next decade. To further improve the Quality-of-Service in these applications, the 3rd Generation Partnership Project has considered the use of terrestrial cellular networks to support UAV sensing services, also known as the cellular Internet of UAVs. In this paper, we consider a cellular Internet of UAVs, where the sensory data can be transmitted either to the base station via cellular links, or to the mobile devices by underlay UAV-to-Device (U2D) communications. To evaluate the freshness of the sensory data, the concept of age of information (AoI) is adopted, in which a lower AoI implies fresher data. Since UAVs' AoIs are determined by their trajectories during sensing and transmission, we investigate the AoI minimization problem for UAVs by designing their trajectories. This problem is a Markov decision problem with an infinite state-action space, and thus we utilize multi-agent deep reinforcement learning to approximate the state-action space. Then, we propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our proposed algorithm can achieve a lower AoI than a greedy algorithm, policy gradient algorithm, and overlay U2D scheme.

Task Offloading for Wireless VR-Enabled Medical Treatment With Blockchain Security Using Collective Reinforcement Learning

Wireless virtual reality (VR)-enabled medical treatment (WVMT) system, integrating the VR technology and the platform of the Internet of Medical Things (IoMT), is a promising application in future medical industries. Multiaccess edge computing (MEC) is an effective approach to support the ubiquitous applications of WVMT systems. Due to the high requirements of medical services, the computation efficiency and security are two issues in WVMT systems. In this article, we propose a blockchain-enabled task offloading scheme, where the viewport rendering tasks of VR devices (VDs) can be offloaded to edge access points (EAPs). The blockchain is integrated into the system to reach the consensus of the global information of task offloading and data processing to resist malicious attacks. To reduce VDs’ computation load under the promise of high VR QoE, we formulate the computation offloading and resource allocation to be a Markov decision problem, considering block consensus, content correlation, and fluctuating channel conditions. Then, a novel collective reinforcement learning (CRL) algorithm is proposed to adaptively allocate resources based on the requirements of viewport rendering, block consensus, and content transmission. In the simulations, the convergence rate and the performance in terms of energy consumption and stalling rate are evaluated. simulation results demonstrate the effectiveness of the proposed scheme.

Anticipatory Troubleshooting

Troubleshooting is the process of diagnosing and repairing a system that is behaving abnormally. It involves performing various diagnostic and repair actions. Performing these actions may incur costs, and traditional troubleshooting algorithms aim to minimize the costs incurred until the system is fixed. Prognosis deals with predicting future failures. We propose to incorporate prognosis and diagnosis techniques to solve troubleshooting problems. This integration enables (1) better fault isolation and (2) more intelligent decision making with respect to the repair actions to employ to minimize troubleshooting costs over time. In particular, we consider an anticipatory troubleshooting challenge in which we aim to minimize the costs incurred to fix the system over time, while reasoning about both current and future failures. Anticipatory troubleshooting raises two main dilemmas: the fix–replace dilemma and the replace-healthy dilemma. The fix–replace dilemma is the question of how to repair a faulty component: fixing it or replacing it with a new one. The replace-healthy dilemma is the question of whether a healthy component should be replaced with a new one in order to prevent it from failing in the future. We propose to solve these dilemmas by modeling them as a Markov decision problem and reasoning about future failures using techniques from the survival analysis literature. The resulting algorithm was evaluated experimentally, showing that the proposed anticipatory troubleshooting algorithms yield lower overall costs compared to troubleshooting algorithms that do not reason about future faults.