Markov Decision Process Framework Research Articles

Condition-based maintenance policy for systems under dynamic environment

This paper addresses the challenge of optimizing maintenance for systems with varying degradation characteristics in dynamic environment. The focus is on repairable systems, where degradation is characterized by a Wiener process that incorporates the dynamic influence of the environment. We investigate a condition-based maintenance model aiming at minimizing the expected cost over a finite and infinite horizon. Firstly, the model considers the evolution of environment as a Markov process, which influences the drift parameter in the Wiener process. The choice between a corrective and preventive replacement rests on the periodic inspections on the system and environment. Subsequently, the maintenance model is formulated in the framework of Markov decision process. Meanwhile, the backward dynamic programming algorithm and value iteration algorithm are employed to gain the optimal maintenance policies over the finite and the infinite planning horizons, respectively. Finally, the proposed model is further explained with concrete examples and a comprehensive sensitivity analysis. Numerical results clearly demonstrate notable variations in the optimal maintenance strategies in different periods and underscore the importance of taking proactive maintenance measures in harsh environmental conditions, even if the degradation level of the system is relatively low.

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.

Reinforcement learning based bilevel real-time pricing strategy for a smart grid with distributed energy resources

The integration of flexible loads, distributed energy resources, and other technologies is becoming common in advance power and energy systems. However, the integration also presents significant challenges due to the increasing complexity and uncertainty. To effectively manage these resources, an adaptive pricing mechanism is needed that can account for their variable availability. Based on this, we propose a new bilevel real-time pricing model that considers different distributed energy resources, uncertainty of renewable energy generation, carbon trading mechanisms, and grid fluctuations. Specifically, the upper-level optimization problem aims to maximize the total profit of the supplier that applies Q-learning algorithm. The lower-level optimization problem addresses the need for each user to make optimal power consumption decisions by constructing an individual Markov Decision Process framework for each user. The bilevel model achieves an effective balance of interests between the supplier and users by simultaneously considering both the upper-level and lower-level optimization problems. Additionally, our model can be efficiently solved using the distributed algorithm without the need to acquire transition probabilities. Simulation results show that the method is highly effective in balancing power supply and demand between the supplier and users, reducing carbon emissions, and mitigating power fluctuations.

Leader–follower UAVs formation control based on a deep Q-network collaborative framework

This study examines a collaborative framework that utilizes an intelligent deep Q-network to regulate the formation of leader–follower Unmanned Aerial Vehicles (UAVs). The aim is to tackle the challenges posed by the highly dynamic and uncertain flight environment of UAVs. In the context of UAVs, we have developed a dynamic model that captures the collective state of the system. This model encompasses variables like as the relative positions, heading angle, rolling angle, and velocity of different nodes in the formation. In the subsequent section, we elucidate the operational procedure of UAVs in a collaborative manner, employing the conceptual framework of Markov Decision Process (MDP). Furthermore, we employ the Reinforcement Learning (RL) to facilitate this process. In light of this premise, a fundamental framework is presented for addressing the control problem of UAVs utilizing the DQN scheme. This framework encompasses a technique for action selection known as ε\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varepsilon$$\\end{document}-imitation, as well as algorithmic specifics. Finally, the efficacy and portability of the DQN-based approach are substantiated by numerical simulation validation. The average reward curve demonstrates a satisfactory level of convergence, and kinematic link between the nodes inside the formation satisfies the essential requirements for the creation of a controller.

Modelling tactical changes in association football using a Markov game

Abstract Accepted by: Phil Scarf We model tactical changes in association football as a Markov game. The pitch is discretized into nine zones and the states of the Markov game are defined according to the zone in which the ball is located in-play, the team in possession and the score. We first model tactical changes in a Markov decision process framework, wherein one team maximizes their probability of winning. Then, we model tactical changes in a two-person zero-sum Markov game framework, wherein both teams maximize their probability of winning. Fundamental to our modelling is the notion that tactical changes impact upon transition rates. We verify the models using data from matches in a season of the Japan Professional Football League. We define a change in transition rates that can be realized by changes in tactics, and illustrate an example of optimal tactical changes when both teams can vary their tactics. The models we develop in the paper can support managers who are considering important decisions about substitutions and changes to formation, for example, when a match is in-play.

Quantifying the Benefits of Increasing Decision-Making Frequency for Health Applications with Regular Decision Epochs

Sequential decision-making problems in the context of uncertainty naturally arise in healthcare settings. In general, the frequency at which decisions can be made or changed is determined by physical limitations, such as the frequency of doctor’s visits or transplantation offers. Quantifying the benefits of increasing the frequency of decision-making allows us to quantify the value of changing these physical constraints and thus improve the quality of care. In this paper, we study the value provided by having additional decision-making opportunities in each epoch. We model this problem using a Markov decision process (MDP) framework. We provide structural properties of the optimal policies and quantify the difference in optimal values between MDP problems of different decision-making frequencies. We analyze numerical examples using liver transplantation in high-risk patients and treatment initiation in chronic kidney disease to illustrate our findings.

Delighting Palates with AI: Reinforcement Learning's Triumph in Crafting Personalized Meal Plans with High User Acceptance.

Eating, central to human existence, is influenced by a myriad of factors, including nutrition, health, personal taste, cultural background, and flavor preferences. The challenge of devising personalized meal plans that effectively encompass these dimensions is formidable. A crucial shortfall in many existing meal-planning systems is poor user adherence, often stemming from a disconnect between the plan and the user's lifestyle, preferences, or unseen eating patterns. Our study introduces a pioneering algorithm, CFRL, which melds reinforcement learning (RL) with collaborative filtering (CF) in a unique synergy. This algorithm not only addresses nutritional and health considerations but also dynamically adapts to and uncovers latent user eating habits, thereby significantly enhancing user acceptance and adherence. CFRL utilizes Markov decision processes (MDPs) for interactive meal recommendations and incorporates a CF-based MDP framework to align with broader user preferences, translated into a shared latent vector space. Central to CFRL is its innovative reward-shaping mechanism, rooted in multi-criteria decision-making that includes user ratings, preferences, and nutritional data. This results in versatile, user-specific meal plans. Our comparative analysis with four baseline methods showcases CFRL's superior performance in key metrics like user satisfaction and nutritional adequacy. This research underscores the effectiveness of combining RL and CF in personalized meal planning, marking a substantial advancement over traditional approaches.

Optimal denial‐of‐service attack scheduling for remote state estimation with time‐varying interference power

AbstractThis article examines resource scheduling optimization for cyber‐physical systems that include a smart sensor, a remote estimator, and an attacker. The attacker's objective is to use denial‐of‐service (DoS) attack to jam the wireless channel, causing packet loss and reducing system performance. We consider a more realistic scenario where other interfering sources exist and conduct research based on the signal‐to‐interference‐plus‐noise ratio (SINR) model. Through this model, we investigate two cases: constant high or low interference power and time‐varying interference power. Different levels of attack power lead to varying packet loss rates, and the same attack power may result in different packet loss rates in different interference environments. Therefore, the attacker needs to consider the impact of other interferences to find the optimal attack strategy that increases the estimation error of the information‐physical system while reducing energy consumption. We describe this optimization problem in a Markov decision process (MDP) framework and demonstrate the existence of an optimal deterministic Markovian strategy. We also establish the monotonicity of the optimal strategy in two different cases and investigate the extreme case. Simulation results support our theoretical results.

Interactive Left-Turning of Autonomous Vehicles at Uncontrolled Intersections

This paper presents a novel interactive motion planning approach for left-turning of autonomous vehicles at uncontrolled intersections. A typical left-turning scenario that consists of agent vehicles and an autonomous ego vehicle is established. The autonomous ego vehicle aims to cross the uncontrolled intersection quickly and safely when meeting with the agent vehicles. A modified obstacle reciprocal collision avoidance (MORCA) prediction model is proposed to predict the trajectory of the agent vehicles with the considerations of longitudinal and lateral interaction-aware behaviors. The parameters of the MORCA prediction model are optimized using particle swarm optimization (PSO) with the inD datasets. Based on MORCA, a partially observable Markov decision process (POMDP) framework is established. The action and state spaces of the POMDP are extended reasonably to satisfy the requirement of real-time application. The simulation results show that the prediction precision is improved by 15% using the MORCA compared with the obstacle reciprocal collision avoidance (ORCA) prediction model. Meanwhile, the results manifest that the proposed MORCA-based POMDP planning method improves 43.8% in commuting efficiency compared with a traditional rule-based method, nearly as good as the vehicle-to-vehicle (V2V) communication is equipped. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The motivation of this article is to establish a safe and efficient planning framework for the left-turning of autonomous vehicles at uncontrolled intersections. Rule-based approaches were widely used in the previous studies but they are not optimal. In this paper, the lateral movements of vehicles at the intersection are considered for the first time. A MORCA prediction model is established and optimized to formulate a POMDP planning framework. The CARLA simulation demonstrates the effectiveness of the proposed MORCA-based POMDP planner. In future work, we will migrate the algorithm to a real-world test platform for experimentation.

Deep Reinforcement Learning-Based Scheduling for NR-U/WiGig Coexistence in Unlicensed mmWave Bands

This paper investigates the coexistence of the New Radio-based access to Unlicensed spectrum (NR-U) network and the Wireless Gigabit (WiGig) network in unlicensed millimeter-wave (mmWave) bands. To enable the NR-U network to achieve equitable and harmonious spectrum sharing with WiGig systems, we develop two new classes of user equipment (UE) scheduling schemes by exploiting the deep reinforcement learning (DRL) technique. Specifically, we first propose the distributed deep reinforcement learning scheduling (DeepDS) scheme, wherein multiple deep neural networks (DNNs) are used to make decisions for different panels in an independent fashion. Thereafter, to reduce the computational cost of adopting multiple DNNs, we design the centralized deep reinforcement learning scheduling (DeepCS) scheme that introduces the shared DNN framework to perform decisions for all panels in parallel at one time. The objective of both DeepDS and DeepCS is to maximize the total data rate of the NR-U network with as little interference to WiGig systems as possible, while satisfying the quality of service (QoS) requirement for each UE. We first formulate this problem into the constrained Markov decision process framework. To address the multi-constraint issue, we put forth a new DRL algorithm that incorporates the Lagrangian primal-dual optimization into the deep Q-network framework, referred to as adaptive multi-constraint deep Q-network (AMC-DQN). With AMC-DQN, both DeepDS and DeepCS can achieve their goals even without acquiring prior operations about the WiGig network. Simulation results show that compared with the state-of-the-art omniLBT and dirLBT, both DeepDS and DeepCS yield significant performance benefits in terms of the total network data rate. We also demonstrate the ability of DeepDS and DeepCS to satisfy the QoS requirements of different UEs and their robustness against various simulation setups. Furthermore, compared with DeepDS, DeepCS can save a large amount of computational cost although at the expense of a slightly lower data rate.

Deep LSTM and LSTM-Attention Q-learning based reinforcement learning in oil and gas sector prediction

Accurate prediction of stock market trends and movements holds great significance in the financial industry as it enables investors, traders, and decision-makers to make informed choices and optimize their investment strategies. In the context of the oil and gas sector, where stock prices are influenced by complex market dynamics and various external factors, reliable predictions are essential for effective decision-making and risk management. This study proposes Deep Long Short-Term Memory Q-Learning (DLQL) and Deep Long Short-Term Memory Attention Q-Learning (DLAQL) models and state-of-the-art Long Short-Term Memory (LSTM) for predicting stock prices in the oil and gas sector. The study utilizes historical stock price data of Cenovus Energy Inc. (CVE), MPLX LP (MPLX), Cheniere Energy Inc. (LNG), and Suncor Energy Inc. (SU) to create and validate these models. The research employs the Markov Decision Process (MDP) framework, a widely-used reinforcement learning technique, to train the deep LSTM Q-Learning and deep LSTM Attention Q-Learning models. This framework allows the models to learn optimal policies based on historical data, enabling them to make accurate predictions and adapt to changing market conditions. The findings of this study reveal that the proposed DLQL and DLAQL perform excellently well in terms of prediction accuracy in the oil and gas sector. The inclusion of attention mechanisms in the DLAQL model further enhances its performance by allowing it to focus on important features and capture relevant information. The results of this research underscore the potential of deep LSTM Q-Learning and deep LSTM Attention Q-Learning models in stock market prediction within the oil and gas sector. The application of these models can lead to improved decision-making, enhanced risk management, and increased profitability for market participants. Further exploration and refinement of these models, along with the incorporation of additional variables and market indicators, can contribute to the development of more sophisticated prediction models in the future. Overall, this study contributes to the advancement of stock market prediction techniques, specifically in the oil and gas sector, by introducing and evaluating the efficacy of deep LSTM Q-Learning and deep LSTM Attention Q-Learning models. The findings highlight the importance of accurate stock market predictions and demonstrate the potential benefits of leveraging these models within the MDP framework to support decision-making and risk management in the dynamic and competitive oil and gas industry.

Multi-Platform dynamic game and operation of hybrid Bike-Sharing systems based on reinforcement learning

The advent of electric bikes, or ebikes, has significantly enhanced competitiveness of bike-sharing systems, providing benefits to both riders (comfort during uphill and long-distance rides), platforms (more profit), and the environment. Operating such a hybrid bike-sharing system, i.e., with both bikes and ebikes, in a competitive multi-platform market, can be challenging due to the complex and unpredictable interplay among heterogeneous market participants, which becomes more pronounced with the ebike varying battery, and dynamic demand. Most related research is predicated on the assumption of a monopoly market, which is not always the case: in worldwide capital-oriented markets, many firms will quickly imitate and join in rapidly developing fields for profits. Thus, this paper addresses platforms’ hybrid bike-sharing system operation problem with time-varying ebike pricing and rebalancing strategy in consideration of competition. We consider two docked hybrid bike-sharing platforms with charging stations at the site. Platforms utilize trucks for their own rebalancing operation including bike, ebike and mixed bike/ebike relocation tasks. We combine the Markov decision process (MDP) model with game theory, and establish the dual-platform MDP framework in which one mainstream platform and one competing platform optimize their profits by dynamic pricing and bike/ebike rebalancing based on highly dynamic and stochastic demand. Users’ choice is described by a modified nested logit model and the endogenous demand is generated. We develop the tailored double dueling deep Q-network for solving dynamic gaming. A series of experiments are conducted based on the real-world dataset in Shenzhen and several strategy combinations are compared. The results show the win–win situation where both platforms improve profits with a higher market ratio and demonstrate how to introduce and operate ebikes in the system by analyzing detailed strategies in different games.

An integrated control strategy for simultaneous robot assignment, tool change and preventive maintenance scheduling using Heterogeneous Graph Neural Network

There has been a leap in the field of smart manufacturing with the advancement of automation systems, robotic technology, big data analytics, and state-of-the-art Artificial Intelligence (AI) and Machine Learning (ML) algorithms. Three very important aspects of smart manufacturing systems are system productivity, product quality, and maintenance of machines and equipment. These three issues are strongly interrelated and collectively determine the performance of a smart manufacturing system. Although there has been significant studies in production control, quality control and maintenance scheduling to address each of these aspects individually, there has been a lack of sufficient studies taking all of them into consideration in one control scheme. In this paper, a mobile multi-skilled robot operated Flexible Manufacturing System (FMS) is considered and a model that integrates robots, individual workstation processes and product quality is developed using a Heterogeneous Graph Structure. Heterogeneous Graph Neural Network (HGNN) is used to aggregate local information from different nodes of the graph model to create node embeddings that represent global information. A control problem is then formulated in the Decentralized Partially Observable Markov Decision Process (Dec-POMDP) framework to simultaneously consider robot assignment, product quality and maintenance scheduling. The problem is solved using Multi-Agent Reinforcement Learning (MARL). A case study is presented to demonstrate the effectiveness of the HGNN-MARL control strategy by comparing it to three baselines and the naive MARL policy without HGNN.

Effect of Wind on Electric Vehicle Energy Consumption: Sensitivity Analyses and Implications for Range Estimation and Optimal Routing

The energy consumption of electric vehicles (EVs) depends on multiple factors. As it affects vehicle range, energy consumption must be accurately predicted. After a summary of the relevant literature, this paper focuses on two sensitivity studies: one on the impact of wind on energy consumption, and the other on the identifiability of wind in the absence of vehicles’ speed and acceleration profiles. The studies show that wind has a significant impact on the energy consumption for a trip, and without high-resolution knowledge of the acceleration and instantaneous velocity, minor variations in the wind condition do not drastically alter the energy consumption distribution. After that, data sources for the information on the wind velocity and direction are discussed. A data-driven approach based on fuzzy set theory is proposed to incorporate wind into the energy prediction; the best model from this approach shows a notable improvement (3.62%) over the currently implemented production-level predictive model for energy consumption on a data set of 35,139 real-world trips; the improvement is even more pronounced (∼ 7%) for trips with more substantial headwind or tailwind level. Recognizing the interplay between range prediction and route selection, we consider a Markov Decision Process (MDP) framework for battery-charge- and travel-time-aware optimal route planning that accounts for the impact of the wind and includes stops at the charging stations. Finally, we propose a framework that includes wind in the operation of EVs, which consists of learning the impact of wind, incorporating wind forecasting into range and energy prediction, and using that prediction to perform optimal routing.

Smart Energy Borrowing and Relaying in Wireless-Powered Networks: A Deep Reinforcement Learning Approach

Wireless energy harvesting (EH) communication has long been considered a sustainable networking solution. However, it has been limited in efficiency, which has been a major obstacle. Recently, strategies such as energy relaying and borrowing have been explored to overcome these difficulties and provide long-range wireless sensor connectivity. In this article, we examine the reliability of a wireless-powered communication network by maximizing the net bit rate. To accomplish our goal, we focus on enhancing the performance of hybrid access points and information sources by optimizing their transmit power. Additionally, we aim to maximize the use of harvested energy, by using energy-harvesting relays for both information transmission and energy relaying. However, this optimization problem is complex, as it involves non-convex variables and requires combinatorial relay selection indicator optimization for decode and forward (DF) relaying. To simplify this problem, we utilize the Markov decision process and deep reinforcement learning framework based on the deep deterministic policy gradient algorithm. This approach enables us to tackle this non-tractable problem, which conventional convex optimization techniques would have difficulty solving in complex problem environments. The proposed algorithm significantly improved the end-to-end net bit rate of the smart energy borrowing and relaying EH system by 13.22%, 27.57%, and 14.12% compared to the benchmark algorithm based on borrowing energy with an adaptive reward for Quadrature Phase Shift Keying, 8-PSK, and 16-Quadrature amplitude modulation schemes, respectively.

RIS-Assisted Robust Beamforming for UAV Anti-Jamming and Eavesdropping Communications: A Deep Reinforcement Learning Approach

The reconfigurable intelligent surface (RIS) has been widely recognized as a rising paradigm for physical layer security due to its potential to substantially adjust the electromagnetic propagation environment. In this regard, this paper adopted the RIS deployed on an unmanned aerial vehicle (UAV) to enhance information transmission while defending against both jamming and eavesdropping attacks. Furthermore, an innovative deep reinforcement learning (DRL) approach is proposed with the purpose of optimizing the power allocation of the base station (BS) and the discrete phase shifts of the RIS. Specifically, considering the imperfect illegitimate node’s channel state information (CSI), we first reformulated the non-convex and non-conventional original problem into a Markov decision process (MDP) framework. Subsequently, a noisy dueling double-deep Q-network with prioritized experience replay (Noisy-D3QN-PER) algorithm was developed with the objective of maximizing the achievable sum rate while ensuring the fulfillment of the security requirements. Finally, the numerical simulations showed that our proposed algorithm outperformed the baselines on the system rate and at transmission protection level.

A novel maintenance strategy for manufacturing system considering working schedule and imperfect maintenance

Condition-based maintenance (CBM) strategies that consider product quality lack a thorough analysis of the impact of manufacturing enterprise’s working schedule, which does not align with real-life production scenarios. To address this gap, this study proposes a novel CBM strategy model for manufacturing systems that takes into account working schedule to determine the optimal balance between maintenance and production while considering product quality. The maintenance set includes imperfect maintenance and perfect maintenance to capture real-life manufacturing scenarios. The model establishes an optimal maintenance strategy using a Markov decision process (MDP) framework. A numerical study is conducted based on internal information and data collected by sensors from a commercial vehicle manufacturing enterprise to demonstrate the effectiveness of the proposed strategy. We proposed a dual-value iteration algorithm to solve the MDP model. Finally, a comprehensive sensitivity analysis is performed to investigate the impact of cost-related parameters on the strategy.

Markov Decision Process Framework for Control-Based Reinforcement Learning

For many years, reinforcement learning (RL) has proven to be very successful in solving a wide variety of learning and decision making under uncertainty (DMuU) problems, including those related to game playing and robotic control. Many different RL approaches, with varying levels of success, have been developed to address these problems.

Combining imitation and deep reinforcement learning to human-level performance on a virtual foraging task

We develop a framework to learn bio-inspired foraging policies using human data. We conduct an experiment where humans are virtually immersed in an open field foraging environment and are trained to collect the highest amount of rewards. A Markov Decision Process (MDP) framework is introduced to model the human decision dynamics. Then, Imitation Learning (IL) based on maximum likelihood estimation is used to train Neural Networks (NN) that map human decisions to observed states. The results show that passive imitation substantially underperforms humans. We further refine the human-inspired policies via Reinforcement Learning (RL) using the on-policy Proximal Policy Optimization (PPO) algorithm which shows better stability than other algorithms and can steadily improve the policies pre-trained with IL. We show that the combination of IL and RL match human performance and that the artificial agents trained with our approach can quickly adapt to reward distribution shift. We finally show that good performance and robustness to reward distribution shift strongly depend on combining allocentric information with an egocentric representation of the environment.

Memory‐based deep reinforcement learning for cognitive radar target tracking waveform resource management

AbstractA cognitive radar (CR) system can offer enhanced target tracking performance due to its intelligence on the perception‐action cycle, wherein a CR adaptively allocates the limited transmitting resources based on its perception of surrounding environments. To effectively manage the transmit waveform resource for the target tracking task, CR resource management problem is formulated under the partially observable Markov decision process framework. The sequential decision‐making and the inherent partial observability for target tracking problem are considered. In the proposed method, a long short‐term memory (LSTM)‐based twin delayed deep deterministic policy gradient (TD3) algorithm is developed to effectively solve the problem. A reward function is designed considering Haykin's cognitive executive attention mechanism for radar systems such that the CR resource management policy has stability in the decision of transmit waveform, which follows the principle of minimum disturbance. Simulation results demonstrate the superiority of the proposed LSTM memory‐based TD3 with improved target tracking performance and increased mean rewards for CR.