Markov Decision Process Problem Research Articles

Competitive Pricing for Resource Trading in Sliced Mobile Networks: A Multi-Agent Reinforcement Learning Approach

The emergence of network slicing as a flagship technology in 5G networks has not only enhanced network expansion and flexibility in resource management for service continuity, but also provided an avenue for establishing a viable market for resource sharing. To optimize the network's resource usage, stakeholders are encouraged to take pragmatic steps toward dynamic resource sharing. This paper designs a techno-economic model for the strategic interactions among multiple competing mobile virtual network operators (MVNOs) and their users in a trading marketplace. We formulate the dynamic pricing problem as a two-stage Stackelberg game, where the MVNOs are leaders, and the users are followers. In the first stage, the MVNOs compete to set their differentiated unit prices using a negotiation mechanism while considering system-level network load. Then, the users decide their purchasing volumes to match the prices of the MVNOs. We transform the game-based optimization problem into a stochastic Markov decision process (MDP) problem and propose a multi-agent deep Q-network (MADQN) method that obtains an optimal solution for the formulated game. Simulation results and analysis reveal that the proposed algorithm achieves convergence under the competitive pricing scheme (CPS) and independent pricing scheme (IPS) while enhancing MVNOs and users' utilities at acceptable levels.

Green Data Center Cooling Control via Physics-guided Safe Reinforcement Learning

Deep reinforcement learning (DRL) has shown good performance in tackling Markov decision process (MDP) problems. As DRL optimizes a long-term reward, it is a promising approach to improving the energy efficiency of data-center cooling. However, enforcement of thermal safety constraints during DRL’s state exploration is a main challenge. The widely adopted reward-shaping approach adds negative reward when the exploratory action results in unsafety. Thus, it needs to experience sufficient unsafe states before it learns how to prevent unsafety. In this article, we propose a safety-aware DRL framework for data-center cooling control. It applies offline imitation learning and online post-hoc rectification to holistically prevent thermal unsafety during online DRL. In particular, the post-hoc rectification searches for the minimum modification to the DRL-recommended action such that the rectified action will not result in unsafety. The rectification is designed based on a thermal state transition model that is fitted using historical safe operation traces and able to extrapolate the transitions to unsafe states explored by DRL. Extensive evaluation for chilled water and direct expansion-cooled data centers in two climate conditions show that our approach saves 18% to 26.6% of total data-center power compared with conventional control and reduces safety violations by 94.5% to 99% compared with reward shaping. We also extend the proposed framework to address data centers with non-uniform temperature distributions for detailed safety considerations. The evaluation shows that our approach saves 14% power usage compared with the PID control while addressing safety compliance during the training.

Minimizing Task Age upon Decision for Low-Latency MEC Networks Task Offloading with Action-Masked Deep Reinforcement Learning.

In this paper, we consider a low-latency Mobile Edge Computing (MEC) network where multiple User Equipment (UE) wirelessly reports to a decision-making edge server. At the same time, the transmissions are operated with Finite Blocklength (FBL) codes to achieve low-latency transmission. We introduce the task of Age upon Decision (AuD) aimed at the timeliness of tasks used for decision-making, which highlights the timeliness of the information at decision-making moments. For the case in which dynamic task generation and random fading channels are considered, we provide a task AuD minimization design by jointly selecting UE and allocating blocklength. In particular, to solve the task AuD minimization problem, we transform the optimization problem to a Markov Decision Process problem and propose an Error Probability-Controlled Action-Masked Proximal Policy Optimization (EMPPO) algorithm. Via simulation, we show that the proposed design achieves a lower AuD than baseline methods across various network conditions, especially in scenarios with significant channel Signal-to-Noise Ratio (SNR) differences and low average SNR, which shows the robustness of EMPPO and its potential for real-time applications.

SMITS: Social and Mobility aware Intelligent Task Scheduling in Vehicular Fog Computing — A Federated DRL Approach

Vehicular fog computing (VFC) has become an enticing research hot spot to provide resources for the Internet of Vehicles (IoV) application requests. Moreover, with the massive expansion of services in vehicular fog networks such as augmented reality, 3-D gaming, and autonomous driving, the allocation of fog computing resources to IoV applications constraining the deadline and energy consumption of the vehicles is becoming challenging. In this paper, we investigate a task scheduling problem in vehicular fog networks to jointly optimize the success rate of vehicular tasks and energy consumption of vehicles by considering the moving paths of the vehicles and the social relationships among the vehicles. In this problem, we model the social relationship among the vehicles based on their communication patterns and social characteristics to improve the success rate of the tasks. We also incorporate the mobility of vehicles using the Markov renewal process (MRP) technique. Initially, we formulate a mixed integer nonlinear programming (MINLP) problem for the proposed problem and further prove it to be an NP-hard problem. To solve the proposed problem, we design a federated deep reinforcement learning (FDRL) mechanism by converting the problem into a Markov decision process (MDP) problem. We compare our proposed algorithm with existing scheduling approaches, and simulation results show that the proposed algorithm performs efficiently for task scheduling problems. The proposed algorithm achieved a substantial improvement of 12% in the maximization of the success rate and 36% in the minimization of the energy consumption compared to the existing algorithms.

Autonomous Vehicle Traffic Accident Prevention using Mobile-Integrated Deep Reinforcement Learning Technique

When it concerns autonomous traffic management, the most effective decision-making reinforcement learning methods are often utilized for vehicle control. Surprisingly demanding circumstances, however, aggravate the collisions and, as a consequence, the chain collisions. In order to potentially offer guidance on eliminating and decreasing the danger of chain collision malfunctions, we first evaluate the main types of chain collisions and the chain events typically proceed. In an emergency, this study proposes mobile-integrated deep reinforcement learning (DRL) for autonomous vehicles to control collisions. Three essential influencing substances are completely taken into consideration and ultimately achieved by the offered strategy: accuracy, efficiency, and passenger comfort. Following this, we investigate the safety performance currently employed in security-driving solutions by interpreting the chain collision avoidance problem as a Markov Decision Process problem and offering a decision-making strategy based on mobile-integrated reinforcement learning. All of the analysis's findings have the objective of aid academics and policymakers to appreciate the positive aspects of a more reliable autonomous traffic infrastructure and to smooth out the way for the actual adoption of a driverless traffic scenario.

Action Candidate Driven Clipped Double Q-Learning for Discrete and Continuous Action Tasks.

Double Q-learning is a popular reinforcement learning algorithm in Markov decision process (MDP) problems. Clipped double Q-learning, as an effective variant of double Q-learning, employs the clipped double estimator to approximate the maximum expected action value. Due to the underestimation bias of the clipped double estimator, the performance of clipped Double Q-learning may be degraded in some stochastic environments. In this article, in order to reduce the underestimation bias, we propose an action candidate-based clipped double estimator (AC-CDE) for Double Q-learning. Specifically, we first select a set of elite action candidates with high action values from one set of estimators. Then, among these candidates, we choose the highest valued action from the other set of estimators. Finally, we use the maximum value in the second set of estimators to clip the action value of the chosen action in the first set of estimators and the clipped value is used for approximating the maximum expected action value. Theoretically, the underestimation bias in our clipped Double Q-learning decays monotonically as the number of action candidates decreases. Moreover, the number of action candidates controls the tradeoff between the overestimation and underestimation biases. In addition, we also extend our clipped Double Q-learning to continuous action tasks via approximating the elite continuous action candidates. We empirically verify that our algorithm can more accurately estimate the maximum expected action value on some toy environments and yield good performance on several benchmark problems. Code is available at https://github.com/Jiang-HB/ac_CDQ.

Quantifying the Benefits of Increasing Decision-Making Frequency for Health Applications with Regular Decision Epochs

Sequential decision-making problems in the context of uncertainty naturally arise in healthcare settings. In general, the frequency at which decisions can be made or changed is determined by physical limitations, such as the frequency of doctor’s visits or transplantation offers. Quantifying the benefits of increasing the frequency of decision-making allows us to quantify the value of changing these physical constraints and thus improve the quality of care. In this paper, we study the value provided by having additional decision-making opportunities in each epoch. We model this problem using a Markov decision process (MDP) framework. We provide structural properties of the optimal policies and quantify the difference in optimal values between MDP problems of different decision-making frequencies. We analyze numerical examples using liver transplantation in high-risk patients and treatment initiation in chronic kidney disease to illustrate our findings.

Rule-based shields embedded safe reinforcement learning approach for electric vehicle charging control

The optimal and secure electric vehicles charging control can be formulated as a large-scale partially-observable constrained Markov decision process (PO-CMDP) problem with high levels of security risks and uncertainties. Such a problem is very difficult to be handled by optimization-theory-based methods. Data-driven methods, especially reinforcement learning (RL), are suitable for handling uncertainties but are poor in ensuring safety, which is unacceptable in power systems. Motivated by the fact that humans could be supervised by an expert when learning a new skill involving risk, this paper proposes a safe RL framework that embeds rule-based local and global shields in the loop of RL for supervising the actions of agents. The proposed framework not only strictly guarantees local and global security in the training and execution phases, but also helps the agent to find a more near-optimal policy. The effectiveness and efficiency are demonstrated by comparison with multiple baseline methods in the IEEE-33 node system.

Optimal economic dispatch of a virtual power plant based on gated recurrent unit proximal policy optimization

The intermittent renewable energy in a virtual power plant (VPP) brings generation uncertainties, which prevents the VPP from providing a reliable and user-friendly power supply. To address this issue, this paper proposes a gated recurrent unit proximal policy optimization (GRUPPO)-based optimal VPP economic dispatch method. First, electrical generation, storage, and consumption are established to form a VPP framework by considering the accessibility of VPP state information. The optimal VPP economic dispatch can then be expressed as a partially observable Markov decision process (POMDP) problem. A novel deep reinforcement learning method called GRUPPO is further developed based on VPP time series characteristics. Finally, case studies are conducted over a 24-h period based on the actual historical data. The test results illustrate that the proposed economic dispatch can achieve a maximum operation cost reduction of 6.5% and effectively smooth the supply–demand uncertainties.

Electric vehicle charging design: The factored action based reinforcement learning approach

Charging optimization design for Electric Vehicles (EV) is challenging because it must account for various uncertainties and design aspects such as random EVs’ arrivals and departures, battery degradation, and transformer Loss of Life (LoL). Model-free reinforcement learning (RL) can be employed to tackle such the EV charging design where it does not require to explicitly model the environment dynamics and accurately predict relevant system parameters. However, the high complexity involved in conventional RL-based approaches usually limits its application to only small-scale EV charging settings, which is impractical. To overcome this limitation, we employ the factored action based RL method to transform the formulated Markov Decision Process (MDP). Then, we propose novel reward shaping and hybrid learning methods combining the Convolutional Neural Network (CNN) and Proximal Policy Optimization (PPO) algorithm to extract relevant features from high-dimension state space and efficiently solve the transformed MDP problem. Extensive numerical studies demonstrate that the proposed design can be used to control a charging station (CS) supporting a large number of EVs. Moreover, we show that the proposed framework greatly outperforms other baselines including single-agent and multi-agent RL based strategies and a heuristic power scheduling algorithm in terms of the achieved reward.

Spectrum utilization improvement for multi‐channel EH‐CRN with spectrum sensing

AbstractDue to the ever‐growing applications and services of the Internet of Things (IoT), designing energy‐efficient and spectral‐efficient transmission schemes to support IoT devices for the 6G space–air–ground integrated networks becomes much more challenging. Fortunately, energy harvesting (EH) and cognitive radio (CR) technologies have been proposed to alleviate these challenges. Inspired by this fact, this paper studies the issue of spectrum reuse in terms of spectrum utilization efficiency (SUE) in the energy harvesting cognitive radio network (EH‐CRN), where multiple primary transceiver pairs, one multi‐antenna secondary transmitter (ST), and one secondary base station (SBS) coexist. To characterize the impact of small‐scale fading and improve the SUE of the EH‐CRN with perfect spectrum sensing (SS), an adaptive scheme concerning SS, channel selection, EH, and data transmission (SCED) scheme are proposed, where the ST selects the channels for SS based on the residual energy, and adjusts the duration of EH and data transmission with respect to the sensing results. Then the Markov decision process problem of SUE is formulated, which is challenging due to the infinite system space and action space. To tackle the Markov decision process problem, the system space and action space are discreted, and divide the ST into the energy‐limited case and energy‐sufficient case according to specific energy condition. Moreover, theoretical results are extended to the EH‐CRN with imperfect SS. Numerical results show that the SUE under the SCED scheme in perfect SS and imperfect SS scenarios is better than that under other schemes.

Deep reinforcement learning architectures for automatic organ segmentation

In the automated segmentation field, the most promising results were obtained using Deep learning (DL) architectures that employ a mix of “Fully convolutional neural networks” (FCN), “Generative adversarial networks” (GAN), or “Recurrent neural networks” (RNN). In our paper we aim to challenge the status-quo of using supervised learning algorithms by proposing and implementing different Deep Reinforcement learning (DRL) architectures that execute the same task – human organ automatic segmentation. We present a completely new approach that defines the segmentation as a Markov Decision Process (MDP) problem where an agent navigates in the environment (CTs) and learns a policy to delineate the contour of the human organ. For solving this problem, we utilized different architectures built on DRL agents. The first architecture employs the Deep Q-Network (DQN) algorithm and uses discrete actions to navigate through CTs while executing the automatic segmentation for the heart. The second architecture implements the Proximal Policy Optimization (PPO) algorithm and utilizes a continuous action space in order to execute the same segmentation process. We tested these two architectures in different setups and using varied reward systems and our results prove that DRL can be used successfully for automated medical segmentation. We also proved that both off-policy, discrete action algorithms or on-policy, continuous action algorithms converge to generalized results.

Multi-agent reinforcement learning based optimal energy sensing threshold control in distributed cognitive radio networks with directional antenna

In CRNs, it is crucial to develop an efficient and reliable spectrum detector that consistently provides accurate information about the channel state. In this work, we investigate a CSS in a fully-distributed environment where all secondary users (SUs) are equipped with directional antennas and make decisions based solely on their local knowledge without information sharing between SUs. First, we establish a stochastic sequential optimization problem, which is an NP-hard, that maximizes the SU’s detection accuracy by the dynamic and optimal control of the energy sensing/detection threshold. It can enable SUs to select an available channel and sector without causing interference to the primary network. To address it in a distributed environment, the problem is transformed into a decentralized partially observed Markov decision process (Dec-POMDP) problem. Second, in order to determine the best control for the Dec-POMDP in a practical environment without any prior knowledge of state–action transition probabilities, we develop a multi-agent deep deterministic policy gradient (MADDPG)-based algorithm, which is referred to as MA-DCSS. This algorithm adopts the centralized training and decentralized execution (CTDE) architecture. Third, we analyzed its computational complexity and showed the proposed approach’s scalability by the polynomial computational complexity, in terms of the number of channels, sectors, and SUs. Lastly, the simulation confirms that the proposed scheme provides enhanced performance in terms of convergence speed, accurate detection, and false alarm probabilities when it is compared to baseline algorithms.

A DDPG-based energy efficient federated learning algorithm with SWIPT and MC-NOMA

Federated learning (FL) has emerged as a promising distributed machine learning technique. It has the potential to play a key role in future Internet of Things (IoT) networks by ensuring the security and privacy of user data combined with efficient utilization of communication resources. This paper addresses the challenge of maximizing energy efficiency in FL systems. We employed simultaneous wireless information and power transfer (SWIPT) and multi-carrier non-orthogonal multiple access (MC-NOMA) techniques. Also, we jointly optimized power allocation and central processing unit (CPU) resource allocation to minimize latency-constrained energy consumption. We formulated an optimization problem using a Markov decision process (MDP) and utilized a deep deterministic policy gradient (DDPG) reinforcement learning algorithm to solve our MDP problem. We tested the proposed algorithm through extensive simulations and confirmed it converges in a stable manner and provides enhanced energy efficiency compared to conventional schemes.

On Distributed Detection in EH-WSNs With Finite-State Markov Channel and Limited Feedback

We consider a network of N sensors, tasked with solving binary distributed detection, a fusion center (FC), and a feedback channel from the FC to sensors. Each sensor is capable of harvesting energy and is equipped with a finite-size battery to store randomly arrived energy. Sensors process their observations and transmit their symbols to the FC over orthogonal and Markovian time correlated fading channels. The FC fuses the received symbols and makes a global binary decision. We aim at developing adaptive channel-dependent transmit power control policies such that J-divergence based detection metric is maximized at the FC, subject to total transmit power constraint. Modeling quantized fading channel, energy arrival, and battery dynamics as time-homogeneous finite-state Markov chains, and the network lifetime as a geometric random variable, we formulate our power control optimization problem as a discounted infinite-horizon constrained Markov decision process (MDP) problem, where sensors’ transmit powers are functions of the battery states, quantized channel gains, and the arrived energies. We utilize stochastic dynamic programming and Lagrangian approach to find the optimal and sub-optimal power control policies. We demonstrate that our sub-optimal policy provides a close-to-optimal performance with a reduced computational complexity and without imposing signaling overhead on sensors.

Federated double DQN based multi-energy microgrid energy management strategy considering carbon emissions

Multi-energy microgrids (MEMG) play an important role in promoting carbon neutrality and achieving sustainable development. This study investigates an effective energy management strategy (EMS) for MEMG. First, an energy management system model that allows for intra-microgrid energy conversion is developed, and the corresponding Markov decision process (MDP) problem is formulated. Subsequently, an improved double deep Q network (iDDQN) algorithm is proposed to enhance the exploration ability by modifying the calculation of the Q value, and a prioritized experience replay (PER) is introduced into the iDDQN to improve the training speed and effectiveness. Finally, taking advantage of the federated learning (FL) and iDDQN algorithms, a federated iDDQN is proposed to design an MEMG energy management strategy to enable each microgrid to share its experiences in the form of local neural network (NN) parameters with the federation layer, thus ensuring the privacy and security of data. The simulation results validate the superior performance of the proposed energy management strategy in minimizing the economic costs of the MEMG while reducing CO2 emissions and protecting data privacy.

Battery State-of-Health Aware Path Planning for a Mars Rover

A rover mission consists of visiting waypoints to gather scientific samples based on set requirements. However, rovers face operational uncertainties during the mission, affecting the performance of its electrical and mechanical components and overall mission success. Hence, it is critical to have a decision-making framework that is aware of the health state of the components when planning the path of the vehicle. In particular, battery degradation, and consequently the battery State of Health (SOH), can affect the optimality of decisions made by the autonomous system in the long term. This paper presents a decision-making system that incorporates information on the energy drawn from the battery (based on the velocity of the vehicle), terrain conditions, and model-based prognostic modules to assess impact on the battery state of charge (SoC). The decision-making system was formulated as a Markov Decision Process (MDP) to reach the goal destination by sending commands in a determined amount of time, while maintaining the battery SoC within the policy stated. The MDP problem was programmed using the open-source framework POMDPs.jl, which has a variety of online and offline solvers. To solve the MDP problem online, we used Monte Carlo Tree Search (MCTS). Results from simulations demonstrate the effect that battery degradation and charging plans have on decision-making.

Energy-Efficient Power Scheduling Policy for Cloud-Assisted Distributed PV System: A TD3 Approach

To cope with climate change and other environmental problems, countries and regions around the world have begun to pay attention to the development of renewable energy under the drive of achieving the global carbon emission peak and carbon neutrality goal. The distributed photovoltaic (PV) power grid is an effective solution that can utilize solar energy resources to provide clean a energy supply. However, with the continuous grid connection of distributed energy, it poses great challenges to the power supply stability and security of the grid. Therefore, it is particularly important to promote the local consumption of distributed energy and the construction of the energy internet. This paper aims to study the cooperative operation and energy optimization scheduling problem among distributed PV power grids, and proposes a new scheme to reduce the electricity cost under the constraint of power supply and demand balance. The optimization problem is modeled as a Markov decision process (MDP), and a Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is used to solve the MDP problem. Simulation results show that the proposed algorithm outperforms other benchmark algorithms in terms of reducing electricity cost, convergence and stability, and verifies its effectiveness.

Adaptively optimal energy management for integrated hydrogen energy systems

AbstractIntegrated hydrogen energy systems (IHESs) have become attractive alternatives to cope with the depletion of fossil fuels and increasingly severe climate change problems. This paper proposes an adaptively optimal energy scheduling method based on deep deterministic policy gradient (DDPG) to improve the operational efficiency of IHES. The optimal scheduling problem is formulated as a Markov decision process problem with action space, environmental states, and action‐value function. The DDPG‐based optimal energy management algorithm with actor‐critic structure is proposed based on policy gradients and neural networks. Through actor‐critic network training and policy iteration, the energy management scheme can be adaptively optimized according to the dynamic responses of IHES. The benefits of the proposed algorithm are analysed through time‐domain simulations, and the scheduling robustness under different uncertain conditions is verified.

Simulating human walking: a model-based reinforcement learning approach with musculoskeletal modeling.

Recent advancements in reinforcement learning algorithms have accelerated the development of control models with high-dimensional inputs and outputs that can reproduce human movement. However, the produced motion tends to be less human-like if algorithms do not involve a biomechanical human model that accounts for skeletal and muscle-tendon properties and geometry. In this study, we have integrated a reinforcement learning algorithm and a musculoskeletal model including trunk, pelvis, and leg segments to develop control modes that drive the model to walk. We simulated human walking first without imposing target walking speed, in which the model was allowed to settle on a stable walking speed itself, which was 1.45 m/s. A range of other speeds were imposed for the simulation based on the previous self-developed walking speed. All simulations were generated by solving the Markov decision process problem with covariance matrix adaptation evolution strategy, without any reference motion data. Simulated hip and knee kinematics agreed well with those in experimental observations, but ankle kinematics were less well-predicted. We finally demonstrated that our reinforcement learning framework also has the potential to model and predict pathological gait that can result from muscle weakness.