Finite Markov Decision Processes Research Articles

Omega-Regular Decision Processes

Regular decision processes (RDPs) are a subclass of non-Markovian decision processes where the transition and reward functions are guarded by some regular property of the past (a lookback). While RDPs enable intuitive and succinct representation of non-Markovian decision processes, their expressive power coincides with finite-state Markov decision processes (MDPs). We introduce omega-regular decision processes (ODPs) where the non-Markovian aspect of the transition and reward functions are extended to an omega-regular lookahead over the system evolution. Semantically, these lookaheads can be considered as promises made by the decision maker or the learning agent about her future behavior. In particular, we assume that, if the promised lookaheads are not met, then the payoff to the decision maker is falsum (least desirable payoff), overriding any rewards collected by the decision maker. We enable optimization and learning for ODPs under the discounted-reward objective by reducing them to lexicographic optimization and learning over finite MDPs. We present experimental results demonstrating the effectiveness of the proposed reduction.

Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge

The problem of sample complexity of online reinforcement learning is often studied in the literature without taking into account any partial knowledge about the system dynamics that could potentially accelerate the learning process. In this paper, we study the sample complexity of online Q-learning methods when some prior knowledge about the dynamics is available or can be learned efficiently. We focus on systems that evolve according to an additive disturbance model of the form S_{h+1} = ƒ(S_h, A_h) + W_h, where ƒ represents the underlying system dynamics, and W_h are unknown disturbances independent of states and actions. In the setting of finite episodic Markov decision processes with S states, A actions, and episode length H, we present an optimistic Q-learning algorithm that achieves Õ(Poly(H)√T) regret under perfect knowledge of ƒ, where T is the total number of interactions with the system. This is in contrast to the typical Õ(Poly(H)√SAT) regret for existing Q-learning methods. Further, if only a noisy estimate ƒ_hat of ƒ is available, our method can learn an approximately optimal policy in a number of samples that is independent of the cardinalities of state and action spaces. The sub-optimality gap depends on the approximation error ƒ_hat − ƒ, as well as the Lipschitz constant of the corresponding optimal value function. Our approach does not require modeling of the transition probabilities and enjoys the same memory complexity as model-free methods.

Risk probability optimization of finite horizon piecewise deterministic Markov decision processes

This paper investigates the piecewise deterministic Markov decision processes (PDMDPs) under the risk probability criterion. The optimality problem is to minimize the probability that the finite horizon total costs are no more than the cost goal. Under some suitable conditions, the value iteration algorithm is adopted to verify the existence of a solution to the optimality problem. Meanwhile, some new facts are established to prove that the value function is the unique solution to the optimality problem, and the existence of an optimal policy. Finally, two examples are presented to explain the application of risk probability PDMDPs, where the first one illustrates the verification of the main conditions, and the second one shows the calculation of the value function and an optimal risk probability policy.

A Gradient-Aware Search Algorithm for Constrained Markov Decision Processes.

The canonical solution methodology for finite constrained Markov decision processes (CMDPs), where the objective is to maximize the expected infinite-horizon discounted rewards subject to the expected infinite-horizon discounted costs' constraints, is based on convex linear programming (LP). In this brief, we first prove that the optimization objective in the dual linear program of a finite CMDP is a piecewise linear convex (PWLC) function with respect to the Lagrange penalty multipliers. Next, we propose a novel, provably optimal, two-level gradient-aware search (GAS) algorithm which exploits the PWLC structure to find the optimal state-value function and Lagrange penalty multipliers of a finite CMDP. The proposed algorithm is applied in two stochastic control problems with constraints for performance comparison with binary search (BS), Lagrangian primal-dual optimization (PDO), and LP. Compared with the benchmark algorithms, it is shown that the proposed GAS algorithm converges to the optimal solution quickly without any hyperparameter tuning. In addition, the convergence speed of the proposed algorithm is not sensitive to the initialization of the Lagrange multipliers.

Modeling and Energy-Optimal Control for Freight Trains based on Data-Driven Approaches

This paper focuses on the energy optimization problem of traction substations. The paper addresses the difficulty of considering time-varying parameters and environmental characteristics of freight train in mechanism modeling. In response, a new optimization method called “Modeling and Energy-Optimal Control for Freight Trains based on Data-Driven Approaches” is proposed. Firstly, a data-driven model considering the “Train-Track-Power grid” (TTP) is constructed based on recorded data. The energy optimization problem is regarded as a finite Markov decision process. A dynamic programming (DP) algorithm is utilized to optimize the train’s control force. The data-driven model is then used to solve for the train’s speed curve, traction substation power, and contact network voltage. Secondly, a multi-input multi-output self-organizing fuzzy neural network (MIMO-SOFNN) model is designed during the modeling process, and an Levenberg–Marquardt algorithm with self-adaption damping factor (SA-LM) is proposed for model parameter learning. Finally, experimental analysis is conducted to validate the high accuracy of the MIMO-SOFNN model compared to five other models. The effectiveness of the SA-LM algorithm is also verified through a comparison with five other algorithms. In the energy optimization experiments, when compared with the actual operation data of a freight railway company in China, the proposed energy optimization method in this paper reduces the energy consumption of the traction substation by 34.8%.

A hybrid heuristic-reinforcement learning-based real-time control model for residential behind-the-meter PV-battery systems

The behind-the-meter (BTM) energy management problem has recently attracted a lot of attention due to the increase in the number of residential photovoltaic (PV)-battery energy storage systems (BESSs). In this work, the use of deep reinforcement learning (DRL) combined with a novel heuristic model for real-time control of home batteries is investigated. The control problem is formulated as a finite Markov Decision Process with discrete time steps, where a proximal policy optimization (PPO) algorithm is employed to train the DRL agent with discrete action space. The agent is trained using real-world measured data to learn the policy for sequential charge/discharge tasks, aiming to minimize daily electricity costs. The battery power is calculated using an innovative heuristic model considering the agent's decision and the battery's available capacity, ensuring demand-supply balance through PV self-consumption and load demand shifting. The performance of the model is evaluated by comparing it to four RL agents and two benchmark models based on rule-based and scenario-based stochastic optimization strategies. The results confirm that the presented model outperforms its counterparts, offering €80.38 savings on electricity bills over 46 days of the test data set. This figure exceeds the savings of the rule-based and stochastic models by €15.64 and €19.38, respectively.

Optimal EV Fast Charging Station Deployment Based on a Reinforcement Learning Framework

This study aims to determine the optimal deployment plan for EV fast charging stations in a transportation network with a limited budget. The objective of the deployment problem is to maximize the quality of service (QoS) with respect to both waiting time and range anxiety from the perspective of EV customers. With the rapid growth of the electric vehicle (EV) market penetration, state-of-the-art algorithms based on mathematical programming are limited in handling high-dimensional optimization problems adequately. Unlike previous studies, we make the first attempt to formulate the fast charging station deployment problem (FCSDP) as a finite discrete Markov decision process (MDP) in a novel reinforcement learning (RL) framework to alleviate the curse of dimensionality problem. Since creating a supervised training dataset is impractical due to the high computational complexity of the FCSDP, we propose a recurrent neural network (RNN) with an attention mechanism to learn the model parameters and determine the optimal policy in a completely unsupervised manner. Finally, numerical experiments are conducted on multiple problem sizes to evaluate the performance of the RNN-based RL framework. Simulation results show that the proposed approach outperforms the comparing algorithms in terms of solution quality and computation time.

A Data-Driven Iterative Learning Approach for Optimizing the Train Control Strategy

The energy-efficient train control (EETC) problem is investigated in this paper. And a soft actor-critic-based method is proposed to optimize the train driving strategy. Firstly, EETC problem is converted to the inverse problem, i.e., minimizing the trip time of the journey with constant energy consumption. Based on the conversion, the EETC problem is reformulated as a finite Markov decision process which can be solved by deep reinforcement learning algorithms. Secondly, an optimization method based on the soft actor-critic (SAC) method is designed to calculate the optimal driving strategy of the train with introducing the reservoir sampling method. Finally, some case studies are conducted to verify the effectiveness and performance of the proposed method. Simulation results demonstrate that a good energy-saving performance can be achieved. In single interval, the SAC-based method can reduce about 1.65% of the energy consumption compared with numerical method. And the energy consumption reduction can be extended to be 6.49% when the proposed approach is applied in multiple intervals.

Order scheduling optimization in manufacturing enterprises based on MDP and dynamic programming

In the era of Industry 4.0, order scheduling is a crucial link in the production of manufacturing enterprises. In view of order scheduling in manufacturing enterprises, a finite horizon Markov decision process model is proposed in this work based on two sets of equipment and three types of orders with different production lead times to maximize the revenue in manufacturing production systems. Then, the dynamic programming model is incorporated into the optimal order scheduling strategy. Python is employed to simulate the order scheduling in manufacturing enterprises. Based on survey data, the superiority of the proposed model compared to traditional first come, first served order scheduling is verified by experimental cases. Finally, sensitivity analysis is conducted on the longest service hours of the devices and the order completion rate to explore the applicability of the proposed order scheduling strategy.

Turnpikes in Finite Markov Decision Processes and Random Walk

In this paper we revise the theory of turnpikes in discounted Markov decision processes, prove the turnpike theorem for the undiscounted model, and apply the results to the specific random walk.KeywordsturnpikeMarkov decision processdiscounted rewardaverage rewardrandom walkstochastic knapsack problem

A reinforcement learning-based sleep scheduling algorithm for compressive data gathering in wireless sensor networks

Compressive data gathering (CDG) is an adequate method to reduce the amount of data transmission, thereby decreasing energy expenditure for wireless sensor networks (WSNs). Sleep scheduling integrated with CDG can further promote energy efficiency. Most of existing sleep scheduling methods for CDG were formulated as centralized optimization problems which introduced many extra control message exchanges. Meanwhile, a few distributed methods usually adopted stochastic decision which could not adapt to variance in residual energy of nodes. A part of nodes were prone to prematurely run out of energy. In this paper, a reinforcement learning-based sleep scheduling algorithm for CDG (RLSSA-CDG) is proposed. Active nodes selection is modeled as a finite Markov decision process. The mode-free Q learning algorithm is used to search optimal decision strategies. Residual energy of nodes and sampling uniformity are considered into the reward function of the Q learning algorithm for load balance of energy consumption and accurate data reconstruction. It is a distributed algorithm that avoids large amounts of control message exchanges. Each node takes part in one step of the decision process. Thus, computation overhead for sensor nodes is affordable. Simulation experiments are carried out on the MATLAB platform to validate the effectiveness of the proposed RLSSA-CDG against the distributed random sleep scheduling algorithm for CDG (DSSA-CDG) and the original sparse-CDG algorithm without sleep scheduling. The simulation results indicate that the proposed RLSSA-CDG outperforms the two contrast algorithms in terms of energy consumption, network lifetime, and data recovery accuracy. The proposed RLSSA-CDG reduces energy consumption by 4.64% and 42.42%, respectively, compared to the DSSA-CDG and the original sparse-CDG, prolongs life span by 57.3%, and promotes data recovery accuracy by 84.7% compared to the DSSA-CDG.

A Graph Reinforcement Learning-Based Decision-Making Platform for Real-Time Charging Navigation of Urban Electric Vehicles

To provide efficient charging behavior decision-making for urban electric vehicles (EVs), this article proposes a new platform for real-time EV charging navigation (EVCN) based on graph reinforcement learning. Considering the interaction of EVs with charging stations (CSs) and traffic networks, the navigation goal of the “vehicle-station-network” coupled system is to minimize the charging cost and traveling time of EV owners. Specifically, to realize data acquisition and decision-making output, we first characterize the EV charging and traveling behavior as the dynamic interaction process of graph-structured networks. A graph convolutional network is used to extract the environment information required for EVCN, and the generated environment feature is fed into the underlying network of deep reinforcement learning (DRL), which can help the agent better understand massive graph-structured data. Then, the real-time navigation problem is duly formulated as a finite Markov decision process. A sequential scheduling pattern is built according to the sorting of EV charging urgency and solved by a Rainbow-based DRL algorithm. It achieves the sequential recommendation of CSs and planning of traveling routes for multiple EVs. Case studies are conducted within a practical zone in Nanjing, China. Simulation results verify the developed platform and the solving method.

Decentralized energy management system for smart microgrids using reinforcement learning

AbstractThis paper presents a novel fully decentralized and intelligent energy management system (EMS) for a smart microgrid based on reinforcement learning (RL) strategy. The purpose of the proposed EMS is to maximize the benefit of all microgrid entities comprising customers and distributed energy resources (DERs). Due to unpredictable features of renewable energy sources and variability of consumers’ demands, designing the microgrid EMS is a complicated task. To overcome this issue, the multi‐agent hour‐ahead energy management problem is modelled as a finite Markov decision process. The microgrid entities are considered as intelligent agents. The optimal policy of agents is obtained through a newly developed framework of the model‐free Q‐learning algorithm to maximize the benefit of all renewable and non‐renewable energy resources and battery energy storage system. The degradation model of the battery is considered to reduce the number of battery replacements. To ensure customers’ comfort, customers’ expenses are decreased without demand curtailment via introducing two types of load shifting techniques. The microgrid operation is analysed under four scenarios comprising no‐learning, generator‐learning, customer‐learning, and whole‐learning. the performance of the proposed algorithm is compared to the Monte Carlo method and simulation results on the real power‐grid dataset show the superiority of the algorithm.

Bi-level deep reinforcement learning for PEV decision-making guidance by coordinating transportation-electrification coupled systems

The random charging and dynamic traveling behaviors of massive plug-in electric vehicles (PEVs) pose challenges to the efficient and safe operation of transportation-electrification coupled systems (TECSs). To realize real-time scheduling of urban PEV fleet charging demand, this paper proposes a PEV decision-making guidance (PEVDG) strategy based on the bi-level deep reinforcement learning, achieving the reduction of user charging costs while ensuring the stable operation of distribution networks (DNs). For the discrete time-series characteristics and the heterogeneity of decision actions, the FEVDG problem is duly decoupled into a bi-level finite Markov decision process, in which the upper-lower layers are used respectively for charging station (CS) recommendation and path navigation. Specifically, the upper-layer agent realizes the mapping relationship between the environment state and the optimal CS by perceiving the PEV charging requirements, CS equipment resources and DN operation conditions. And the action decision output of the upper-layer is embedded into the state space of the lower-layer agent. Meanwhile, the lower-level agent determines the optimal road segment for path navigation by capturing the real-time PEV state and the transportation network information. Further, two elaborate reward mechanisms are developed to motivate and penalize the decision-making learning of the dual agents. Then two extension mechanisms (i.e., dynamic adjustment of learning rates and adaptive selection of neural network units) are embedded into the Rainbow algorithm based on the DQN architecture, constructing a modified Rainbow algorithm as the solution to the concerned bi-level decision-making problem. The average rewards for the upper-lower levels are ¥ -90.64 and ¥ 13.24 respectively. The average equilibrium degree of the charging service and average charging cost are 0.96 and ¥ 42.45, respectively. Case studies are conducted within a practical urban zone with the TECS. Extensive experimental results show that the proposed methodology improves the generalization and learning ability of dual agents, and facilitates the collaborative operation of traffic and electrical networks.

Constructing MDP Abstractions Using Data With Formal Guarantees

This letter is concerned with a data-driven technique for constructing finite Markov decision processes (MDPs) as finite abstractions of discrete-time stochastic control systems with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unknown</i> dynamics while providing formal closeness guarantees. The proposed scheme is based on notions of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stochastic bisimulation functions</i> (SBF) to capture the probabilistic distance between state trajectories of an unknown stochastic system and those of finite MDP. In our proposed setting, we first reformulate corresponding conditions of SBF as a robust convex program (RCP). We then propose a scenario convex program (SCP) associated to the original RCP by collecting a finite number of data from trajectories of the system. We ultimately construct an SBF between the data-driven finite MDP and the unknown stochastic system with a given confidence level by establishing a probabilistic relation between optimal values of the SCP and the RCP. We also propose two different approaches for the construction of finite MDPs from data. We illustrate the efficacy of our results over a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nonlinear</i> jet engine compressor with unknown dynamics. We construct a data-driven finite MDP as a suitable substitute of the original system to synthesize controllers maintaining the system in a safe set with some probability of satisfaction and a desirable confidence level.

Turnpikes in finite Markov decision processes and random walk

В данной статье мы пересматриваем магистральное свойство для дисконтированных марковских процессов принятия решений, доказываем магистральную теорему для модели без дисконтирования и применяем полученные результаты для специфического случайного блуждания.

Additive valuations of streams of payoffs that satisfy the time value of money principle: A characterization and robust optimization

This paper characterizes those preferences over bounded infinite utility streams that satisfy the time value of money principle and an additivity property, and the subset of these preferences that, in addition, are either impatient or patient. Based on this characterization, the paper introduces a concept of optimization that is robust to a small imprecision in the specification of the preference, and proves that the set of feasible streams of payoffs of a finite Markov decision process admits such a robust optimization.

Inexact GMRES Policy Iteration for Large-Scale Markov Decision Processes

Policy iteration enjoys a local quadratic rate of contraction, but its iterations are computationally expensive for Markov decision processes (MDPs) with a large number of states. In light of the connection between policy iteration and the semismooth Newton method and taking inspiration from the inexact variants of the latter, we propose inexact policy iteration, a new class of methods for large-scale finite MDPs with local contraction guarantees. We then design an instance based on the deployment of the generalized minimal residual method (GMRES) for the approximate policy evaluation step, which we call inexact GMRES policy iteration. Finally, we demonstrate the superior practical performance of inexact GMRES policy iteration on an MDP with 10000 states, where it achieves a × 5.8 and × 2.2 speedup with respect to policy iteration and optimistic policy iteration, respectively.

An axiomatic approach to Markov decision processes

This paper presents an axiomatic approach to finite Markov decision processes where the discount rate is zero. One of the principal difficulties in the no discounting case is that, even if attention is restricted to stationary policies, a strong overtaking optimal policy need not exists. We provide preference foundations for two criteria that do admit optimal policies: 0-discount optimality and average overtaking optimality. As a corollary of our results, we obtain conditions on a decision maker’s preferences which ensure that an optimal policy exists. These results have implications for disciplines where dynamic programming problems arise, including automatic control, dynamic games, and economic development.

Intelligent scheduling of double-deck traversable cranes based on deep reinforcement learning

ABSTRACT Cranes are used extensively in manufacturing workshops to move jobs, but their high complexity and dynamics lead to difficult workshop production scheduling. To address this issue, this article proposes a deep reinforcement learning-based method combined with discrete event simulation to minimize the makespan of the double-deck traversable crane flexible job-shop scheduling problem (DTCFJSP). Specifically, the problem is first formulated as a finite Markov decision process by introducing state representation, an action space and a reward function. Then, a new double-deep Q-learning network is incorporated to create a selection strategy for optimal actions in different states. The results of experiments conducted in this study show that the average efficiency of the double-deck traversable crane is approximately 12% higher than that of regular cranes, and the application of deep reinforcement learning in crane scheduling is feasible and effective.