Markov Decision Process Research Articles

Optimization of High-Frequency Trading Strategies Using Deep Reinforcement Learning

This study presents a new method for optimising high-risk trading (HFT) strategies using deep learning (DRL). We propose a multi-time DRL framework integrating advanced neural network architectures with sophisticated business data processing techniques. The framework employs a combination of convolutional neural networks for manual order analysis, short-term memory networks for time series processing, and a multi-head listening mechanism for body fusion. We formulate the HFT problem based on Markov Decision Processes and use the Proximal Policy Optimization algorithm for training. The model is evaluated using tick-by-tick data from the NASDAQ exchange, including ten liquid stocks in 6 months. The experimental results show the superiority of our method, achieving a Sharpe ratio of 3.42, outperforming the learning model and machine learning based on benchmarks up to 33%. The proposed strategy has demonstrated strong performance across a wide range of regulatory markets and has shown potential for strategic objectives. Sensitivity analysis confirms the model's stability across a range of hyperparameters. Our findings suggest that the DRL-based approach can improve HFT performance and provide better market adaptation and risk management. This research leads to the continuous evolution of algorithmic trading strategies and shows the potential of AI-driven approaches in financial markets.

Optimization of emergency frequency control strategy for power systems considering both source and load uncertainties

With the increasing integration of renewable energy sources and the presence of numerous controllable loads such as electric vehicles and energy storage in the modern power system, higher nonlinearities and uncertainty both sources and loads are introduced. These factors pose challenges in achieving fast and accurate emergency frequency control. Therefore, this paper addresses the issue of dual source-load uncertainties in power system and presents an optimization strategy based on the Soft Actor Critic (SAC) algorithm that involves the participation of controllable loads in emergency frequency control. Firstly, the spatio-temporal uncertainties of wind farm power output on power supply side and power demand on the load side are described using Weibull and normal probability distributions, respectively. Secondly, an improved Markov Decision Process (MDP) model for emergency frequency control is established, which considers the characteristics of the dual source-load uncertainties. Finally, an optimization of the SAC algorithm is conducted based on Deep Reinforcement Learning (DRL), aiming to achieve rapid system frequency recovery and minimize the cost of removing controllable loads. The presented approach in the paper enhances the emergency frequency control strategy for uncertain power systems and effectively addresses the issue of source-load uncertainty compounded by fault power shortages.

Comparative Analysis of Value Iteration and Policy Iteration in Robotic Decision-Making

This study focuses on Markov Decision Processes (MDPs) as a framework for decision-making in robotics. MDPs are crucial for enabling autonomous systems to operate in dynamic environments. MDPs enable the development of optimal strategies that balance immediate and future rewards, making them essential for intelligent robotic behavior. The purpose of this study is to evaluate the performance differences of value iteration and strategy iteration algorithms in different robot tasks. The study highlights the strengths and weaknesses of each algorithm. This comparative analysis uniquely addresses a gap in existing research by evaluating both Value Iteration and Policy Iteration side by side, offering critical insights into their respective performances across diverse robotic tasks. Results indicate that Policy Iteration converges faster and adapts better in complex environments, making it ideal for real-time applications, while Value Iteration is more efficient in smaller state spaces.These findings provide critical insights for selecting algorithms tailored to specific robotic applications, highlighting their role in advancing intelligent robotic systems.

Reinforcement Learning for EV Fleet Smart Charging with On-Site Renewable Energy Sources

In 2020, the transportation sector was the second largest source of carbon emissions in the UK and in Newcastle upon Tyne, responsible for about 33% of total emissions. To support the UK’s target of reaching net zero emissions by 2050, electric vehicles (EVs) are pivotal in advancing carbon-neutral road transportation. Optimal EV charging requires a better understanding of the unpredictable output from on-site renewable energy sources (ORES). This paper proposes an integrated EV fleet charging schedule with a proximal policy optimization method based on a framework for deep reinforcement learning. For the design of the reinforcement learning environment, mathematical models of wind and solar power generation are created. In addition, the multivariate Gaussian distributions derived from historical weather and EV fleet charging data are utilized to simulate weather and charging demand uncertainty in order to create large datasets for training the model. The optimization problem is expressed as a Markov decision process (MDP) with operational constraints. For training artificial neural networks (ANNs) through successive transition simulations, a proximal policy optimization (PPO) approach is devised. The optimization approach is deployed and evaluated on a real-world scenario comprised of council EV fleet charging data from Leicester, UK. The results show that due to the design of the rewards function and system limitations, the charging action is biased towards the time of day when renewable energy output is maximum (midday). The charging decision by reinforcement learning improves the utilization of renewable energy by 2–4% compared to the random charging policy and the priority charging policy. This study contributes to the reduction in battery charging and discharging, electricity sold to the grid to create benefits and the reduction in carbon emissions.

Complexity Bounds for Deterministic Partially Observed Markov Decision Processes

Complexity Bounds for Deterministic Partially Observed Markov Decision Processes

Sum and Tensor of Quantitative Effects

Inspired by the seminal work of Hyland, Plotkin, and Power on the combination of algebraic computational effects via sum and tensor, we develop an analogous theory for the combination of quantitative algebraic effects. Quantitative algebraic effects are monadic computational effects on categories of metric spaces, which, moreover, have an algebraic presentation in the form of quantitative equational theories, a logical framework introduced by Mardare, Panangaden, and Plotkin that generalises equational logic to account for a concept of approximate equality. As our main result, we show that the sum and tensor of two quantitative equational theories correspond to the categorical sum (i.e., coproduct) and tensor, respectively, of their effects qua monads. We further give a theory of quantitative effect transformers based on these two operations, essentially providing quantitative analogues to the following monad transformers due to Moggi: exception, resumption, reader, and writer transformers. Finally, as an application, we provide the first quantitative algebraic axiomatizations to the following coalgebraic structures: Markov processes, labelled Markov processes, Mealy machines, and Markov decision processes, each endowed with their respective bisimilarity metrics. Apart from the intrinsic interest in these axiomatizations, it is pleasing they have been obtained as the composition, via sum and tensor, of simpler quantitative equational theories.

Real-Time Policy Optimization for UAV Swarms Based on Evolution Strategies

Multi-agent decision-making faces many challenges such as non-stationarity and sparse rewards, while the complexity and randomness of the real environment further complicate policy development. This paper addresses the high-dimensional policy optimization problems of unmanned aerial vehicle (UAV) swarms. By modeling the problem scenario as a Markov decision process, a real-time policy optimization algorithm based on evolution strategy (ES) pre-training is proposed. This approach combines decision-time planning with background planning to evaluate and integrate different sets of policy parameters in a temporal context. In the experimental phase, the policy network is trained using both ES and REINFORCE algorithms on a constructed simulation platform. Comparative experiments demonstrate the effectiveness of using ES for policy pre-training. Finally, the proposed real-time policy optimization algorithm further improves the performance of the swarm by approximately 10% in simulations, offering a feasible solution for adversarial games between swarms and extending the research scope of evolutionary algorithms.

User-Perceived Capacity: Theory, Computation, and Achievable Policies

User-perceived throughput is a novel performance metric attracting a considerable amount of recent attention because it characterizes the quality of the experience in mobile multimedia services. For instance, it gives a data rate of video streaming with which a user will not experience any lag or outage in watching video clips. However, its performance limit remains open. In this paper, we are interested in the achievable upper bound of user-perceived throughput, also referred to as the user-perceived capacity, and how to achieve it in typical wireless channels. We find that the user-perceived capacity is quite limited or even zero with channel state information at the receiver (CSIR) only. When both CSIR and channel state information at the transmitter (CSIT) are available, the user-perceived throughput can be substantially improved by power or even rate adaptation. A constrained Markov decision process (CMDP)-based approach is conceived to compute the user-perceived capacity with joint power–rate adaptation. It is rigorously shown that the optimal policy obeys a threshold-based rule with time, backlog, and channel gain thresholds. With power adaptation only, the user-perceived capacity is equal to the hard-delay-constrained capacity in our previous work and achieved by joint diversity and channel inversion.

Multi-UAV Escape Target Search: A Multi-Agent Reinforcement Learning Method

The multi-UAV target search problem is crucial in the field of autonomous Unmanned Aerial Vehicle (UAV) decision-making. The algorithm design of Multi-Agent Reinforcement Learning (MARL) methods has become integral to research on multi-UAV target search owing to its adaptability to the rapid online decision-making required by UAVs in complex, uncertain environments. In non-cooperative target search scenarios, targets may have the ability to escape. Target probability maps are used in many studies to characterize the likelihood of a target’s existence, guiding the UAV to efficiently explore the task area and locate the target more quickly. However, the escape behavior of the target causes the target probability map to deviate from the actual target’s position, thereby reducing its effectiveness in measuring the target’s probability of existence and diminishing the efficiency of the UAV search. This paper investigates the multi-UAV target search problem in scenarios involving static obstacles and dynamic escape targets, modeling the problem within the framework of decentralized partially observable Markov decision process. Based on this model, a spatio-temporal efficient exploration network and a global convolutional local ascent mechanism are proposed. Subsequently, we introduce a multi-UAV Escape Target Search algorithm based on MAPPO (ETS–MAPPO) for addressing the escape target search difficulty problem. Simulation results demonstrate that the ETS–MAPPO algorithm outperforms five classic MARL algorithms in terms of the number of target searches, area coverage rate, and other metrics.

Relaxed Equilibria for Time-Inconsistent Markov Decision Processes

This paper considers an infinite-horizon Markov decision process (MDP) that allows for general nonexponential discount functions in both discrete and continuous time. Because of the inherent time inconsistency, we look for a randomized equilibrium policy (i.e., relaxed equilibrium) in an intrapersonal game between an agent’s current and future selves. When we modify the MDP by entropy regularization, a relaxed equilibrium is shown to exist by a nontrivial entropy estimate. As the degree of regularization diminishes, the entropy-regularized MDPs approximate the original MDP, which gives the general existence of a relaxed equilibrium in the limit by weak convergence arguments. As opposed to prior studies that consider only deterministic policies, our existence of an equilibrium does not require any convexity (or concavity) of the controlled transition probabilities and reward function. Interestingly, this benefit of considering randomized policies is unique to the time-inconsistent case.

Medication recommendation for Parkinson’s disease based on dynamics of symptom progression

Parkinson’s Disease (PD) is a chronic condition, with extensive research on initial medication selection for treatment, but limited guidance on long-term medication management. This study aims to identify optimal medication adjustment strategies based on patient clusters, focusing on either maximizing time spent in favorable health states or minimizing time spent in unfavorable ones while avoiding adverse effects. To guide treatment, we developed decision models using prescription dosages converted into standardized units for various medications. Using data from the Parkinson’s Progression Markers Initiative, we employed a multivariate time-series clustering approach to capture symptom progression dynamics. This analysis identified four distinct clusters: two representing desirable and undesirable states, and two highlighting motor-focused and non-motor-focused failures across multiple domains. We developed two separate Markov Decision Process (MDP) models to address these dual objectives, which were then integrated into a comprehensive framework that suggests optimal actions and cautions against risky ones for each patient state. This model provides valuable insights for clinical decision-making by offering flexible guidance on adjusting medication intensity rather than prescribing specific medication types, enhancing its applicability in clinical practice.

Optimal control of discrete event systems under uncertain environment based on supervisory control theory and reinforcement learning

Discrete event systems (DESs) are powerful abstract representations for large human-made physical systems in a wide variety of industries. Safety control issues on DESs have been extensively studied based on the logical specifications of the systems in various literature. However, when facing the DESs under uncertain environment which brings into the implicit specifications, the classical supervisory control approach may not be capable of achieving the performance. So in this research, we propose a new approach for optimal control of DESs under uncertain environment based on supervisory control theory (SCT) and reinforcement learning (RL). Firstly, we use SCT to gather deliberative planning algorithms with the aim to safe control. Then we convert the supervised system to Markov Decision Process simulation environments that is suitable for optimal algorithm training. Furthermore, a SCT-based RL algorithm is designed to maximize performance of the system based on the probabilistic attributes of the state transitions. Finally, a case study on the autonomous navigation task of a delivery robot is provided to corroborate the proposed method by multiple simulation experiments. The result shows the proposed approach owning 8.27%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} performance improvement compared with the non-intelligent methods. This research will contribute to further studying the optimal control of human-made physical systems in a wide variety of industries.

Discounted fully probabilistic design of decision rules

Axiomatic fully probabilistic design (FPD) of optimal decision rules strictly extends the decision making (DM) theory represented by Markov decision processes (MDP). This means that any MDP task can be approximated by an explicitly found FPD task whereas many FPD tasks have no MDP equivalent. MDP and FPD model the closed loop — the coupling of an agent and its environment — via a joint probability density (pd) relating the involved random variables, referred to as behaviour. Unlike MDP, FPD quantifies agent's aims and constraints by an ideal pd. The ideal pd is high on the desired behaviours, small on undesired behaviours and zero on forbidden ones. FPD selects the optimal decision rules as the minimiser of Kullback-Leibler's divergence of the closed-loop-modelling pd to its ideal twin. The proximity measure choice follows from the FPD axiomatics.MDP minimises the expected total loss, which is usually the sum of discounted partial losses. The discounting reflects the decreasing importance of future losses. It also diminishes the influence of errors caused by:▪ the imperfection of the employed environment model;▪ roughly-expressed aims;▪ the approximate learning and decision-rules design.The established FPD cannot currently account for these important features. The paper elaborates the missing discounted version of FPD. This non-trivial filling of the gap in FPD also employs an extension of dynamic programming, which is of an independent interest.

Multi-Agent Deep Reinforcement Learning-Based Distributed Voltage Control of Flexible Distribution Networks with Soft Open Points

The increasing number of distributed generators (DGs) leads to the frequent occurrence of voltage violations in distribution networks. The soft open point (SOP) can adjust the transmission power between feeders, leading to the evolution of traditional distribution networks into flexible distribution networks (FDN). The problem of voltage violations can be effectively tackled with the flexible control of SOPs. However, the centralized control method for SOP may make it difficult to achieve real-time control due to the limitations of communication. In this paper, a distributed voltage control method is proposed for FDN with SOPs based on the multi-agent deep reinforcement learning (MADRL) method. Firstly, a distributed voltage control framework is proposed, in which the updating algorithm of the intelligent agent of MADRL is expounded considering experience sharing. Then, a Markov decision process for multi-area SOP coordinated voltage control is proposed, where the control areas are divided based on electrical distance. Finally, an IEEE 33-node test system and a practical system in Taiwan are used to verify the effectiveness of the proposed method. It shows that the proposed multi-area SOP coordinated control method can achieve real-time control while ensuring a better control effect.

The Shared Experience Actor–Critic (SEAC) Approach for Allocating Radio Resources and Mitigating Resource Collisions in 5G-NR-V2X Mode 2 Under Aperiodic Traffic Conditions

5G New Radio (NR)-V2X, standardized by 3GPP Release 16, includes a distributed resource allocation Mode, known as Mode 2, that allows vehicles to autonomously select transmission resources using either sensing-based semi-persistent scheduling (SB-SPS) or dynamic scheduling (DS). In unmanaged 5G-NR-V2X scenarios, SB-SPS loses effectiveness with aperiodic and variable data. DS, while better for aperiodic traffic, faces challenges due to random selection, particularly in high traffic density scenarios, leading to increased collisions. To address these limitations, this study models the Cellular V2X network as a decentralized multi-agent networked Markov decision process (MDP), where each vehicle agent uses the Shared Experience Actor–Critic (SEAC) technique to optimize performance. The superiority of SEAC over SB-SPS and DS is demonstrated through simulations, showing that the SEAC with an N-step approach achieves an average improvement of approximately 18–20% in enhancing reliability, reducing collisions, and improving resource utilization under high vehicular density scenarios with aperiodic traffic patterns.

Safedrive dreamer: Navigating safety–critical scenarios in autonomous driving with world models

Achieving stable and reliable autonomous driving in complex traffic environments while ensuring safety under unpredictable conditions is a critical challenge in autonomous driving technology. To address this issue, this study proposes the Safedrive Dreamer navigation framework, which aims to reduce the reliance on trial-and-error learning in real-world scenarios, thereby mitigating the risks associated with dynamic driving conditions and enhancing vehicle foresight. This framework integrates the predictive capabilities of world models with the constrained Markov decision process (CMDP) and safety reinforcement learning to accurately anticipate future environmental changes. This ensures the reliability of autonomous driving routes, thereby improving both safety and efficiency. Furthermore, to reduce trial-and-error costs in real-world applications, this study employs PAC-Bayesian methods to derive generalization error bounds between simulations and reality, enabling a more effective transfer of knowledge and experience from simulations to real-world scenarios. Validation experiments in simulated and real environments showed that Safedrive Dreamer significantly outperformed existing autonomous driving solutions by 3.8% in key safety metrics, excelling in collision avoidance and risk reduction. This study provides new insights into the integration of world models into decision-making processes to enhance decision-making capabilities in safety–critical applications, thereby contributing significantly to the improvement of autonomous driving safety and reliability.

Traffic signal phase control at urban isolated intersections: an adaptive strategy utilizing the improved D3QN algorithm

Abstract The intelligent control of traffic signals at urban single intersections has emerged as an effective approach to mitigating urban traffic congestion. However, the existing fixed phase control strategy of traffic signal lights lacks capability to dynamically adjust signal phase switching based on real-time traffic conditions leading to traffic congestion. In this paper, an adaptive real-time control method employed by the traffic signal phase at a single intersection is considered based on the improved Double Dueling Deep Q Network (D3QN) algorithm. Firstly, the traffic signal phase control problem is modeled as a Markov decision process, with its state, action, and reward defined. Subsequently, to enhance the convergence speed and learning performance of the D3QN algorithm, attenuation action selection strategy and priority experience playback technology based on tree summation structure are introduced. Then, traffic flow data from various traffic scenarios are utilized to train the traffic signal control model based on the improved D3QN to obtain the optimal signal phase switch strategy. Finally, the effectiveness and optimal performance of the improved D3QN-based traffic signal control strategy are validated across diverse traffic scenarios. The simulation results show that, compared with the control strategy based on AC, DQN, DDQN, D3QN, and C-D3QN algorithms, the cumulative reward of the proposed I-D3QN strategy is increased by at least 6.57%, and the average queue length and average waiting time are reduced by at least 9.64% and 7.61%, which can effectively reduce the congestion at isolated intersections and significantly improve traffic efficiency.

Adaptive and transparent decision-making in autonomous robots through graph-structured world models

ABSTRACT The growing ubiquity of autonomous robots across different fields necessitates that agents adapt to diverse tasks and ensure transparency and intelligibility in their decision-making processes. This study presents a novel framework that combines a graph-structured world model with large language models (LLMs) to address these requirements. First, a latent space is created to capture the reachability between distinct states. Next, a graph is constructed within this latent space by clustering an offline dataset that effectively captures the complex dynamics of the environment. Subsequently, LLMs are employed to redefine the reward function and relabel the dataset, thereby establishing a well-defined Markov decision process based on the previously learned graph. This relabeling process ensures that the agent's decision space aligns with user intentions. Consequently, a predictive world model is obtained, offering insights into potential future states and facilitating graph-based planning. Moreover, by inputting the planned path of the agent in the graph-structured world model into LLMs, natural language explanations can be generated to provide transparency in the decision-making process. Experiments on the D4RL benchmark validated the effectiveness of our approach in long-horizon planning, its adaptability to different user tasks, and its inherent explainability.

Priority-based intelligent resolution method of multi-aircraft flight conflicts

Abstract The rising demand for air traffic will inevitably result in a surge in both the number and complexity of flight conflicts, necessitating intelligent strategies for conflict resolution. This study addresses the critical challenges of scalability and real-time performance in multi-aircraft flight conflict resolution by proposing a comprehensive method that integrates a priority ranking mechanism with a conflict resolution model based on the Markov decision process (MDP). Within this framework, the proximity between aircraft in a multi-aircraft conflict set is dynamically assessed to establish a conflict resolution ranking mechanism. The problem of multi-aircraft conflict resolution is formalised through the MDP, encompassing the design of state space, discrete action space and reward function, with the transition function implemented via simulation prediction using model-free methods. To address the positional uncertainty of aircraft in real-time scenarios, the conflict detection mechanism introduces the aircraft’s positional error. A deep reinforcement learning (DRL) environment is constructed incorporating actual airspace structures and traffic densities, leveraging the Actor Critic using Kronecker-factored Trust Region (ACKTR) algorithm to determine resolution actions. The experimental results indicate that with 20–30 aircraft in the airspace, the success rate can reach 94% for the training set and 85% for the test set. Furthermore, this study analyses the impact of varying aircraft numbers on the success rate within a specific airspace scenario. The outcomes of this research provide valuable insights for the automation of flight conflict resolution.

Managing Resources for Shared Micromobility: Approximate Optimality in Large-Scale Systems

We consider the problem of managing resources in shared micromobility systems (bike sharing and scooter sharing). An important task in managing such systems is periodic repositioning/recharging/sourcing of units to avoid stockouts or excess inventory at nodes with unbalanced flows. We consider a discrete-time model; each period begins with an initial inventory at each node in the network, and then, customers (demand) materialize at the nodes. Each customer picks up a unit at the origin node and drops it off at a randomly sampled destination node with an origin-specific probability distribution. We model the above network inventory management problem as an infinite horizon discrete-time discounted Markov decision process (MDP) and prove the asymptotic optimality of a novel mean-field approximation to the original MDP as the number of stations becomes large. To compute an approximately optimal policy for the mean-field dynamics, we provide an algorithm with a running time that is logarithmic in the desired optimality gap. Lastly, we compare the performance of our mean field-based policy with state-of-the-art heuristics via numerical experiments, including experiments using Austin scooter-sharing data. This paper was accepted by Jeannette Song, operations management. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.02023 .