Finite-state Markov Decision Processes Research Articles

Omega-Regular Decision Processes

Regular decision processes (RDPs) are a subclass of non-Markovian decision processes where the transition and reward functions are guarded by some regular property of the past (a lookback). While RDPs enable intuitive and succinct representation of non-Markovian decision processes, their expressive power coincides with finite-state Markov decision processes (MDPs). We introduce omega-regular decision processes (ODPs) where the non-Markovian aspect of the transition and reward functions are extended to an omega-regular lookahead over the system evolution. Semantically, these lookaheads can be considered as promises made by the decision maker or the learning agent about her future behavior. In particular, we assume that, if the promised lookaheads are not met, then the payoff to the decision maker is falsum (least desirable payoff), overriding any rewards collected by the decision maker. We enable optimization and learning for ODPs under the discounted-reward objective by reducing them to lexicographic optimization and learning over finite MDPs. We present experimental results demonstrating the effectiveness of the proposed reduction.

An Exact Solution to Wordle

Optimal Strategies for Wordle and its Variants Motivated by the recent, explosive popularity of Wordle and a suite of suboptimal attempts to solve the game, “An Exact Solution to Wordle,” by Dimitris Bertsimas and Alex Paskov, proposes a new and scalable framework to exactly solve Wordle and any of its variants. Namely through modeling the game as a finite-state Markov decision process and deriving several theoretical and computational speed-ups, Bertsimas and Paskov outline an algorithm to solve the game. Ultimately, they find that “SALET” is the best starting word and that the best strategy solves the game in 3.42 moves in expectation. Results and analysis are also extended for recent variants and settings, such as Wordle Hard Mode or optimizing against a worst-case situation.

A Risk-Averse Preview-Based Q-Learning Algorithm: Application to Highway Driving of Autonomous Vehicles

A risk-averse preview-based <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -learning planner is presented for navigation of autonomous vehicles (AVs). To this end, the multilane road ahead of a vehicle is represented by a finite-state nonstationary Markov decision process (MDP). A risk assessment unit module is then presented, which leverages the preview information provided by sensors along with a stochastic reachability module to assign reward values to the MDP states and update them as scenarios develop. A sampling-based risk-averse preview-based <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -learning algorithm is finally developed, which generates samples using the preview information and reward function to learn risk-averse optimal planning strategies without actual interaction with the environment. The risk factor is imposed on the objective function to avoid fluctuation of the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> values, which can jeopardize the vehicle’s safety and/or performance. The overall hybrid automaton model of the system is leveraged to develop a feasibility check unit module that detects unfeasible plans and enables the planner system to react proactively to the changes of the environment. Finally, to verify the efficiency of the presented algorithm, its implementation on two highway driving scenarios of an AV in a varying traffic density is considered.

Scheduling Schemes for Age Optimization in IoT Systems With Limited Retransmission Times

Age of Information (AoI) is a recently introduced metric to capture data freshness. In this article, we consider a multiuser single-destination Internet of Things (IoT) system with periodic state updating, and investigate the scheduling methods of minimizing the long-term average AoI with limited retransmission times. In view of the retransmission mode, the AoI optimization problems for retransmission with and without feedback are analyzed. For the retransmission without feedback, we formulate the expected decision loss (EDL) function of the AoI optimization problem with the finite retransmission times and propose a loss-greedy policy by minimizing the EDL at each step. For the retransmission with feedback, a potential optimal solution is to construct the AoI optimization problem as an infinite-horizon Markov decision process (MDP) and solve the corresponding Bellman optimal equations. However, this optimal solution is prone to suffer from the curse of dimensionality and is hard to implement. To address this issue, we decouple the infinite-horizon MDP into the finite-state MDP in each single frame, and then propose a low-complexity slot-based max-weight (SBMW) policy to minimize the long-term average AoI. Numerical results show that, compared with an ALOHA-like baseline policy, the proposed Loss-Greedy policy can achieve up to 48% reduction of the average AoI for the retransmission without feedback, while the SBMW policy can reduce the average AoI by 57% in the retransmission with feedback. The average performance gain of the proposed policies over the state-of-the-art policies is at least 10%.

Monitoring policy in the context of preventive treatment of cardiovascular disease.

Preventing chronic diseases is an essential aspect of medical care. To prevent chronic diseases, physicians focus on monitoring their risk factors and prescribing the necessary medication. The optimal monitoring policy depends on the patient's risk factors and demographics. Monitoring too frequently may be unnecessary and costly; on the other hand, monitoring the patient infrequently means the patient may forgo needed treatment and experience adverse events related to the disease. We propose a finite horizon and finite-state Markov decision process to define monitoring policies. To build our Markov decision process, we estimate stochastic models based on longitudinal observational data from electronic health records for a large cohort of patients seen in the national U.S. Veterans Affairs health system. We use our model to study policies for whether or when to assess the need for cholesterol-lowering medications. We further use our model to investigate the role of gender and race on optimal monitoring policies.

CVaR Optimization for MDPs

We study the problem of Conditional Value-at-Risk (CVaR) optimization for a finite-state Markov Decision Process (MDP) with total discounted costs and the reduction of this problem to a stochastic game with perfect information. The CVaR optimization problem for finite and infinite-horizon MDPs can be reformulated as a zero-sum stochastic game with a compact state space. This game has the following property: while the second player has perfect information including the knowledge of the decision chosen by the first player at the current time instance, the first player does not directly observe the augmented component of the state and does not know current and past decisions chosen by the second player. Using methods of convex analysis, we show optimal policies exist for this game and an optimal policy of the first player optimizes CVaR of the total discounted costs. In addition to proving existence of optimal policies, we provide algorithms for their computation.

Automating Vehicles by Risk-Averse Preview-based Q-Learning Algorithm

A risk-averse preview-based Q-learning planner is presented for navigation of autonomous vehicles. To this end, the multi-lane road ahead of a vehicle is represented by a finite-state non-stationary Markov decision process (MDP). A sampling-based risk-averse preview-based Q-learning algorithm is finally developed that generates samples using the preview information and reward function to learn risk-averse optimal planning strategies without actual interaction with the environment. The risk factor is imposed on the objective function to avoid fluctuation of the Q values, which can jeopardize the vehicle's safety and/or performance. Theoretical results are provided to bound the number of samples required to guarantee ϵ-optimal planning with a high probability. Finally, to verify the efficiency of the presented algorithm, its implementation on highway driving of an autonomous vehicle in a varying traffic density is considered.

Reinforcement Learning-Based Routing Algorithm in Satellite-Terrestrial Integrated Networks

Satellite-terrestrial integrated network (STIN) is an indispensable component of the Next Generation Internet (NGI) due to its wide coverage, high flexibility, and seamless communication services. It uses the part of satellite network to provide communication services to the users who cannot communicate directly in terrestrial network. However, existing satellite routing algorithms ignore the users’ request resources and the states of the satellite network. Therefore, these algorithms cannot effectively manage network resources in routing, leading to the congestion of satellite network in advance. To solve this problem, we model the routing problem in satellite network as a finite-state Markov decision process and formulate it as a combinatorial optimization problem. Then, we put forth a Q-learning-based routing algorithm (QLRA). By maximizing users’ utility, our proposed QLRA algorithm is able to select the optimal paths according to the dynamic characteristics of satellite network. Considering that the convergence speed of QLRA is slow due to the routing loop or ping-pong effect in the process of routing, we propose a split-based speed-up convergence strategy and also design a speed-up Q-learning-based routing algorithm, termed SQLRA. In addition, we update the Q value of each node from back to front in the learning process, which further accelerate the convergence speed of SQLRA. Experimental results show that our improved routing algorithm SQLRA greatly enhances the performance of satellite network in terms of throughput, delay, and bit error rate compared with other routing algorithms.

Research on autonomous collision avoidance of merchant ship based on inverse reinforcement learning

To learn the optimal collision avoidance policy of merchant ships controlled by human experts, a finite-state Markov decision process model for ship collision avoidance is proposed based on the analysis of collision avoidance mechanism, and an inverse reinforcement learning (IRL) method based on cross entropy and projection is proposed to obtain the optimal policy from expert’s demonstrations. Collision avoidance simulations in different ship encounters are conducted and the results show that the policy obtained by the proposed IRL has a good inversion effect on two kinds of human experts, which indicate that the proposed method can effectively learn the policy of human experts for ship collision avoidance.

Convergence of Value Functions for Finite Horizon Markov Decision Processes with Constraints

This paper is concerned with finite horizon countable state Markov decision processes (MDPs) having an absorbing set as a constraint. Convergence of value iteration is discussed to investigate the asymptotic behavior of value functions as the time horizon tends to infinity. It turns out that the value function exhibits three different limiting behaviors according to the critical value $$\lambda _*$$ , the so-called generalized principal eigenvalue, of the associated ergodic problem. Specifically, we prove that (i) if $$\lambda _*<0$$ , then the value function converges to a solution to the corresponding stationary equation; (ii) if $$\lambda _*>0$$ , then, after a suitable normalization, it approaches a solution to the corresponding ergodic problem; (iii) if $$\lambda _*=0$$ , then it diverges to infinity with, at most, a logarithmic order. We employ this convergence result to examine qualitative properties of the optimal Markovian policy for a finite horizon MDP when the time horizon is sufficiently large.

Deep Reinforcement Learning-based resource allocation strategy for Energy Harvesting-Powered Cognitive Machine-to-Machine Networks

Machine-to-Machine (M2M) communication is a promising technology that may realize the Internet of Things (IoTs) in future networks. However, due to the features of massive devices and concurrent access requirement, it will cause performance degradation and enormous energy consumption. Energy Harvesting-Powered Cognitive M2M Networks (EH-CMNs) as an attractive solution is capable of alleviating the escalating spectrum deficient to guarantee the Quality of Service (QoS) meanwhile decreasing the energy consumption to achieve Green Communication (GC) became an important research topic. In this paper, we investigate the resource allocation problem for EH-CMNs underlaying cellular uplinks. We aim to maximize the energy efficiency of EH-CMNs with consideration of the QoS of Human-to-Human (H2H) networks and the available energy in EH-devices. In view of the characteristic of EH-CMNs, we formulate the problem to be a decentralized Discrete-time and Finite-state Markov Decision Process (DFMDP), in which each device acts as agent and effectively learns from the environment to make allocation decision without the complete and global network information. Owing to the complexity of the problem, we propose a Deep Reinforcement Learning (DRL)-based algorithm to solve the problem. Numerical results validate that the proposed scheme outperforms other schemes in terms of average energy efficiency with an acceptable convergence speed.

Modeling pedestrian-cyclist interactions in shared space using inverse reinforcement learning

The objective of this study is to model the microscopic behaviour of mixed traffic (cyclist-pedestrian) interactions in non-motorized shared spaces. Video data were collected at two locations of Robson Square non-motorized shared space in downtown Vancouver, British Columbia. Trajectories of cyclists and pedestrians involved in interactions were extracted using computer vision algorithms. The extracted trajectories were used to obtain several variables that describe elements of road users’ behaviour including longitudinal and lateral distances, speed and speed differences, interaction angle, and cyclist acceleration and yaw rate. The road users behaviour was modeled as utility-based intelligent rational agents using the finite-state Markov Decision Process (MDP) framework with unknown reward functions. The study implemented Inverse Reinforcement Learning (IRL) using two algorithms: the Maximum Entropy (ME) algorithm, and the Feature Matching (FM) algorithm to recover/estimate the reward function weights of cyclists in two types of interactions with pedestrians: following and overtaking interactions. Reward function weights infer cyclist preferences during their interactions with pedestrians in non-motorized shared spaces, and can form the key component in developing agent based microsimulation model for road users. Furthermore, the estimated reward functions were used to estimate cyclists’ optimal policy for such interactions. A simulation platform was developed using the estimated reward functions and the cyclist optimal policies to simulate cyclist trajectories for the validation dataset. Results show that the Maximum Entropy (ME) IRL algorithm outperformed the Feature Matching (FM) IRL algorithm, and generally provided reasonable results for modeling such interactions in non-motorized shared spaces, considering the high degrees of freedom in movement and the more-complex road users interactions in such facilities. This research is considered an important step toward developing a full Agent-Based Model (ABM) for road users in shared space facilities to evaluate the safety and efficiency of such facilities.

Deep Deterministic Policy Gradient (DDPG)-Based Resource Allocation Scheme for NOMA Vehicular Communications

This paper investigates the resource allocation problem in vehicular communications based on multi-agent Deep Deterministic Policy Gradient (DDPG), in which each Vehicle-to-Vehicle (V2V) communication acts as agent and adopts Non-Orthogonal Multiple Access (NOMA) technology to share the frequency spectrum that pre-allocated to Vehicle-to-Infrastructure (V2I) communications. Different with conventional D2D communications, the fast varying channel condition due to the high mobility in vehicular environment causes the difficulty of collecting instantaneous Channel State Information (CSI) at base station. Meanwhile, one tremendous challenge faced by vehicular communications is how to maximize the sum-rate of V2I communications simultaneously guaranteeing the latency and reliability requirements for the transmission of safety-critical information in V2V communications. In response, we formulate the resource allocation problem as a decentralized Discrete-time and Finite-state Markov Decision Process (DFMDP), in which allocation decisions are made by multiple agents that do not have complete and global network information. Due to the complexity of the problem, we propose a DDPG algorithm which is capable of handling continuous high dimensional action spaces to find the optimal allocation strategy. Numerical results verify that each agent can effectively learn from the environment by means of the proposed DDPG algorithm to maximize the sum-rate of V2I communications while satisfying the stringent latency and reliability constraints of V2V communications.

Reinforcement Learning (RL)-Based Energy Efficient Resource Allocation for Energy Harvesting-Powered Wireless Body Area Network.

Wireless body area networks (WBANs) have attracted great attention from both industry and academia as a promising technology for continuous monitoring of physiological signals of the human body. As the sensors in WBANs are typically battery-driven and inconvenient to recharge, an energy efficient resource allocation scheme is essential to prolong the lifetime of the networks, while guaranteeing the rigid requirements of quality of service (QoS) of the WBANs in nature. As a possible alternative solution to address the energy efficiency problem, energy harvesting (EH) technology with the capability of harvesting energy from ambient sources can potentially reduce the dependence on the battery supply. Consequently, in this paper, we investigate the resource allocation problem for EH-powered WBANs (EH-WBANs). Our goal is to maximize the energy efficiency of the EH-WBANs with the joint consideration of transmission mode, relay selection, allocated time slot, transmission power, and the energy constraint of each sensor. In view of the characteristic of the EH-WBANs, we formulate the energy efficiency problem as a discrete-time and finite-state Markov decision process (DFMDP), in which allocation strategy decisions are made by a hub that does not have complete and global network information. Owing to the complexity of the problem, we propose a modified Q-learning (QL) algorithm to obtain the optimal allocation strategy. The numerical results validate the effectiveness of the proposed scheme as well as the low computation complexity of the proposed modified Q-learning (QL) algorithm.

Optimization of Cybercafe Internet Service and Costs under Stochastic Demand

We consider a set of cybercafes faced with an optimal choice of bandwidth for internet service under stochastic stationary demand. The choice is made over uniformly time horizons with a goal of optimizing costs. Considering customer demand and operating costs of internet service at cybercafes, we formulate a finite state markov decision process model where states of a markov chain represent possible states of demand for internet service. An operational cost matrix is generated; representing the long run measure of performance for the markov decision process problem. The problem is to determine an optimal bandwidth adjustment policy at cybercafes so that the long run operational costs are minimized for the given state of demand. Using dynamic programming, the optimal bandwidth adjustment policies are determined at least cost over a finite period planning horizon. Results from the case study demonstrate the existence of an optimal state-dependent option for bandwidth adjustment policy and costs of internet service at cybercafes.

Optimal Path-Planning of Nonholonomic Terrain Robots for Dynamic Obstacle Avoidance Using Single-Time Velocity Estimator and Reinforcement Learning Approach

A single-time velocity estimator-based reinforcement learning (RL) algorithm, integrated with a chaotic metaheuristic optimization technique is proposed in this article for the optimal path-planning of the nonholonomic robots considering a moving/stationary obstacle avoidance strategy. The additional contribution of the present study is by employing the Terramechanics principles to incorporate the effects of wheel sinkage into the deformable terrain on the dynamics of the robot aiming to find the optimal compensating force/torque magnitude to sustain a robust and smooth motion. The designed systematic control-oriented system incorporates a cost function of weighted components associated with the target-tracking and the obstacle avoidance. The designed velocity estimator contributes to the finite-state Markov decision process (MDP) in order to train the transition probabilities of the problem objectives. Based on the obtained results, the optimal solution for the Q-learning in terms of the adjusting factor for the minimized tracking error and obstacle collision risk propagation profiles is found at 0.22. The results further confirm the promising capacity of the proposed optimization-based RL algorithm for the collision avoidance control of the nonholonomic robots on deformable terrains.

Energy Efficiency-Delay Tradeoff in Energy-Harvesting-Based D2D Communication: An Experimental Learning Approach

Efficient energy management methods are expected to become cornerstones for realizing the energy-harvesting-based device-to-device communication underlaying cellular network (EH-DCCN). But, the quality of service (such as delay performance) also needs to be considered when we pursue the high energy efficiency. Our aim is to investigate a tradeoff perspective between energy efficiency and delay under the EH-DCCN. Thus, we model a discrete-time and finite-state Markov decision process to study a long-term energy efficiency and delay tradeoff (EDT) problem. Simultaneously, an experience-based irrelative state-action abstraction (EISA) algorithm based on Q-learning (QL) is proposed to address the EDT problem and accelerate the convergence rate. Simulation results reveal that the EISA algorithm can achieve the same optimal performance as the typical QL method with rapid convergence speed.

Finite state estimation and control of a multi-input CSTR benchmark

The problem of curse-of-dimensionality in finite state and action Markov decision processes is considered using iterative clustering of closed-loop data and repeated discretization of the state space process model. The performance of the control design approach is demonstrated using a multi-input van der Vusse continuous stirred tank reactor control benchmark. It is demonstrated that the finite state description provides a simple implementation of Bayesian state estimation using cell filters, and dynamic programming gives means to conduct optimization of closed-loop performance of a nonlinear stochastic multidimensional chemical plant.

Approximate Safety Verification and Control of Partially Observable Stochastic Hybrid Systems

Assuring safety in stochastic hybrid systems is particularly difficult when only noisy or partial observations of the state are available. We first review a formulation of the probabilistic safety problem under partial hybrid observations as a dynamic program over an equivalent information state. Two methods for approximately solving the dynamic program are presented. The first approximates the hybrid system as a finite state Markov decision process, so that the information state is a probability mass function. The second method approximates an indicator function over the safe region using radial basis functions, to represent the information state as a Gaussian mixture. In both cases we discretize the hybrid observation process, then use point-based value iteration to under-approximate the safety probability and synthesize a safety-preserving control policy. We obtain error bounds and convergence results in both cases, assuming switched affine dynamics and additive Gaussian noise on the continuous states and observations. We compare the performance of the finite state and Gaussian mixture approaches on a simple numerical example.

Optimal sensor scheduling for multiple linear dynamical systems

We consider the design of an optimal collision-free sensor schedule for a number of sensors which monitor different linear dynamical systems correspondingly. At each time, only one of all the sensors can send its local estimate to the remote estimator. A preliminary work for the two-sensor scheduling case has been studied in the literature. The generalization into multiple-sensor scheduling case is shown to be nontrivial. We first find a necessary condition of the optimal solution which can significantly reduce the feasible optimal solution space without loss of performance. By modelling a finite-state Markov decision process (MDP) problem, we can numerically search an asymptotic periodic schedule which is proven to be optimal. Some simple but effective suboptimal schedules for any systems are proposed. We also find a lower bound of the optimal cost, which enables us to quantify the performance gap between any suboptimal schedule and an optimal one.