Underlying Markov Decision Process Research Articles

A kernel-based approximate dynamic programming approach: Theory and application

This article proposes a novel kernel-based Dynamic Programming (DP) approximation method to tackle the typical curse of dimensionality of stochastic DP problems over the finite time horizon. Such a method utilizes kernel functions in combination with Support Vector Machine (SVM) regression to determine an approximate cost function for the entire state space of the underlying Markov Decision Process (MDP), by leveraging cost function computed for selected representative states. Kernel functions are used to define the so-called kernel matrix, while the parameter vector of the given kernel-based cost function approximation is computed by moving backwards in time from the terminal condition and by applying SVM regression. This way, the difficulty of selecting a proper set of features is also tackled. The proposed method is then extended to the infinite time horizon case. To show the effectiveness of the proposed approach, the resulting Recursive Residual Approximate Dynamic Programming (RR-ADP) algorithm is applied to the sensor scheduling design in multi-process remote state estimation problems.

Energy-efficient production control of a make-to-stock system with buffer- and time-based policies

Increasing energy efficiency in manufacturing has significant environmental and cost benefits. Turning on or off a machine dynamically while considering the production rate requirements can offer substantial energy savings. In this work, we examine the optimal policies to control production and turn on and off a machine that operates in working, idle, off, and warmup modes for the case where demand inter-arrival, production, and warmup times have phase-type distributions. The optimal control problem that minimises the expected costs associated with the energy usage in different energy modes and the inventory and backlog costs is solved using a linear program associated with the underlying Markov Decision Process. We also present a matrix-geometric method to evaluate the steady-state performance of the system under a given threshold control policy. We show that when the inter-arrival time distribution is not exponential, the optimal control policy depends on both the current phase of the inter-arrival time and inventory position. The phase-dependent policy implemented by estimating the current phase based on the time elapsed since the last arrival yields a buffer- and time-based policy to control the energy mode and production. We show that policies that only use the inventory position information can be effective if the control parameters are chosen appropriately. However, the control policies that use both the inventory and time information further improve the performance.

Deep Q-Network Approach for Train Timetable Rescheduling Based on Alternative Graph

The disturbance of local areas with complex railway networks and high traffic density not only impedes the efficient use of rail networks in those areas, but also propagates delays to the entire railway network. This has motivated research on train rescheduling problems in high-density local areas to minimize train delays by modifying their planned arrival and departure times. In this paper, we present a train rescheduling method based on Q-learning in reinforcement learning. More specifically, we used deep neural networks to approximate the action-value function, and the underlying Markov decision process (MDP) is based on the alternative graph formulation for the train rescheduling problem. In the proposed MDP formulation, the status of the alternative graph corresponding to the current schedule is defined as the state, and the alternative arc corresponds to the action the agent can take. The MDP is approximately solved via deep Q-learning in which deep neural networks are used to approximate the action-value function in Q-learning. Although the size of the alternative graph depends on the number of trains, our MDP formulation is independent of the number of trains, which makes the proposed method more scalable. The evaluation of the method was performed on a simple railway network and a real-world example in Seoul, South Korea, with randomly generated initial train schedules and train delays. The experimental result showed that the proposed method is comparable to the mixed-integer-linear-programming (MILP)-based exact approach with respect to the quality of the solution.

Finite-Sample Analysis of Two-Time-Scale Natural Actor–Critic Algorithm

Actor-critic style two-time-scale algorithms are one of the most popular methods in reinforcement learning, and have seen great empirical success. However, their performance is not completely understood theoretically. In this paper, we characterize the <i>global</i> convergence of an online natural actor-critic algorithm in the tabular setting using a single trajectory of samples. Our analysis applies to very general settings, as we only assume ergodicity of the underlying Markov decision process. In order to ensure enough exploration, we employ an <inline-formula><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula>-greedy sampling of the trajectory. For a fixed and small enough exploration parameter <inline-formula><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula>, we show that the two-time-scale natural actor-critic algorithm has a rate of convergence of <inline-formula><tex-math notation="LaTeX">$\tilde{\mathcal {O}}(1/T^{1/4})$</tex-math></inline-formula>, where <inline-formula><tex-math notation="LaTeX">$T$</tex-math></inline-formula> is the number of samples, and this leads to a sample complexity of <inline-formula><tex-math notation="LaTeX">$\tilde{\mathcal {O}}(1/\delta ^{8})$</tex-math></inline-formula> samples to find a policy that is within an error of <inline-formula><tex-math notation="LaTeX">$\delta$</tex-math></inline-formula> from the <i>global optimum</i>. Moreover, by carefully decreasing the exploration parameter <inline-formula><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> as the iterations proceed, we present an improved sample complexity of <inline-formula><tex-math notation="LaTeX">$\tilde{\mathcal {O}}(1/\delta ^{6})$</tex-math></inline-formula> for convergence to the global optimum.

Three-layer model for the control of epidemic infection over multiple social networks

This paper presents a hierarchical approach for controlling the spread of an epidemic disease. The approach consists of a three-layer architecture where a set of two-layer multiple social networks is governed by a (third) top-layer consisting of an optimal control policy. Each of the two-layer social networks is modeled by a microscopic Markov chain. On top of all the two-layer networks is an optimal control policy that has been developed by using an underlying Markov Decision Process (MDP) model. Mathematical models pertaining to the top-level MDP as well as two-layer microscopic Markov chains have been presented. Practical implementation methodology using the proposed models has also been discussed along with a numerical example. The results in the numerical example illustrate the control of an epidemic using the optimal policy. Directions for further research and characterization of the optimal policy have also been discussed with the help of the same numerical example.Article HighlightsAn optimal approach for controlling the spread of an epidemic infection.The approach is able to model the uncertainties involved in the problem.The approach is able to cater for the underlying social network.

Reinforcement Learning Framework for Server Placement and Workload Allocation in Multiaccess Edge Computing

Cloud computing is a reliable solution to provide distributed computation power. However, real-time response is still challenging regarding the enormous amount of data generated by the IoT devices in 5G and 6G networks. Thus, multi-access edge computing (MEC), which consists of distributing the edge servers in the proximity of end-users to have low latency besides the higher processing power, is increasingly becoming a vital factor for the success of modern applications. This paper addresses the problem of minimizing both, the network delay, which is the main objective of MEC, and the number of edge servers to provide a MEC design with minimum cost. This MEC design consists of edge servers placement and base stations allocation, which makes it a joint combinatorial optimization problem (COP). Recently, reinforcement learning (RL) has shown promising results for COPs. However, modeling real-world problems using RL when the state and action spaces are large still needs investigation. We propose a novel RL framework with an efficient representation and modeling of the state space, action space and the penalty function in the design of the underlying Markov Decision Process (MDP) for solving our problem.

Managing chance-constrained hydropower with reinforcement learning and backoffs

The control of multi-reservoir hydropower systems is a crucial cogwheel in the current transition of power systems towards renewable energy. In particular in northern regions where inflows vary enormously through the year, the satisfaction of storage constraints cannot be ensured at all times. We thus add chance constraints to the underlying Markov Decision Process problem, and solve it with a Reinforcement Learning, policy gradient approach. “Backoffs”, common in other areas of optimal control, are introduced to better manage the numerous chance constraints as a single joint constraint. Stochastic Dynamic Programming (SDP) is used as a benchmark approach. We show numerically that our approach can deliver high quality policies, and just as importantly, that a three-reservoir system is solved in proportionally little more time than a one-reservoir system.

Rollout approach to sensor scheduling for remote state estimation under integrity attack

We consider the sensor scheduling problem for remote state estimation under integrity attacks. We seek to optimize a trade-off between the energy consumption of communications and the state estimation error covariance when the acknowledgment (ACK) information, sent by the remote estimator to the local sensor, is compromised. The sensor scheduling problem is formulated as an infinite horizon discounted optimal control problem with infinite states. We first analyze the underlying Markov decision process (MDP) and show that the optimal scheduling without ACK attack is of the threshold type. Thus, we can simplify the problem by replacing the original state space with a finite state space. For the simplified MDP, when the ACK is under attack, the problem is modeled as a partially observable Markov decision process (POMDP). We analyze the induced MDP that uses a belief vector as its state for the POMDP. We investigate the properties of the exact optimal solution via contractive models and show that the threshold type of solution for the POMDP cannot be readily obtained. A suboptimal solution is then obtained via a rollout approach, which is a prominent class of reinforcement learning (RL) methods based on approximation in value space. We present two variants of rollout and provide performance bounds of those variants. Finally, numerical examples are used to demonstrate the effectiveness of the proposed rollout methods by comparing them with a finite history window approach that is widely used in RL for POMDP.

An Optimal Computing Budget Allocation Tree Policy for Monte Carlo Tree Search

We analyze a tree search problem with an underlying Markov decision process, in which the goal is to identify the best action at the root that achieves the highest cumulative reward. We present a new tree policy that optimally allocates a limited computing budget to maximize a lower bound on the probability of correctly selecting the best action at each node. Compared to widely used upper confidence bound (UCB) tree policies, the new tree policy presents a more balanced approach to manage the exploration and exploitation tradeoff when the sampling budget is limited. Furthermore, UCB assumes that the support of reward distribution is known, whereas our algorithm relaxes this assumption. Numerical experiments demonstrate the efficiency of our algorithm in selecting the best action at the root.

Optimistic Posterior Sampling for Reinforcement Learning: Worst-Case Regret Bounds

We present an algorithm based on posterior sampling (aka Thompson sampling) that achieves near-optimal worst-case regret bounds when the underlying Markov decision process (MDP) is communicating with a finite, although unknown, diameter. Our main result is a high probability regret upper bound of [Formula: see text] for any communicating MDP with S states, A actions, and diameter D. Here, regret compares the total reward achieved by the algorithm to the total expected reward of an optimal infinite-horizon undiscounted average reward policy in time horizon T. This result closely matches the known lower bound of [Formula: see text]. Our techniques involve proving some novel results about the anti-concentration of Dirichlet distribution, which may be of independent interest. Funding: This work was supported in part by an NSF CAREER award [CMMI 1846792] awarded to author S. Agrawal.

Bounded rational agents playing a public goods game.

An agent-based model for human behavior in the well-known public goods game (PGG) is developed making use of bounded rationality, but without invoking mechanisms of learning. The underlying Markov decision process is driven by a path integral formulation of reward maximization. The parameters of the model can be related to human preferences accessible to measurement. Fitting simulated game trajectories to available experimental data, we demonstrate that our agents are capable of modeling human behavior in PGG quite well, including aspects of cooperation emerging from the game. We find that only two fitting parameters are relevant to account for the variations in playing behavior observed in 16 cities from all over the world. We thereby find that learning is not a necessary ingredient to account for empirical data.

Battery energy storage control using a reinforcement learning approach with cyclic time-dependent Markov process

Scheduling efficient energy management system operations to respond to the unstable customer demand, electricity prices, and weather increases the complexity of the control systems and requires a flexible and cost-effective control policy. This study develops an intelligent and real-time battery energy storage control based on a reinforcement learning model focused on residential houses connected to the grid and equipped with solar photovoltaic panels and a battery energy storage system. Because the reinforcement learning’s performance is very dependent on the design of the underlying Markov decision process, a cyclic time-dependent Markov Process is uniquely designed to capture existing daily cyclic patterns in demand, electricity price, and solar energy. The Markov Process is successfully used in the Q-learning algorithm, resulting in more efficient battery energy control and saving electricity costs. The proposed Q-learning algorithm is compared with benchmark models of a deterministic equivalent solution and a One-step Roll-out algorithm. Numerical experiments show the gap between the deterministic equivalent solution and Q-learning approaches for one-month electricity cost decreased from 7.99% to 3.63% for house 27 and 6.91% to 3.26% for house 387 when the discrete size of demand, solar energy, price, and battery energy level adjusted to 20. Accordingly, the better performance of the proposed Q-learning is demonstrated compared to the One-step Roll-out algorithm. Moreover, the effect of discrete size of state-space parameters on the adaptive Q-learning performance and computational time are investigated. Variations in the electricity price significantly affect the Q-learning algorithm’s performance more than other parameters.

A projected primal-dual gradient optimal control method for deep reinforcement learning

In this contribution, we start with a policy-based Reinforcement Learning ansatz using neural networks. The underlying Markov Decision Process consists of a transition probability representing the dynamical system and a policy realized by a neural network mapping the current state to parameters of a distribution. Therefrom, the next control can be sampled. In this setting, the neural network is replaced by an ODE, which is based on a recently discussed interpretation of neural networks. The resulting infinite optimization problem is transformed into an optimization problem similar to the well-known optimal control problems. Afterwards, the necessary optimality conditions are established and from this a new numerical algorithm is derived. The operating principle is shown with two examples. It is applied to a simple example, where a moving point is steered through an obstacle course to a desired end position in a 2D plane. The second example shows the applicability to more complex problems. There, the aim is to control the finger tip of a human arm model with five degrees of freedom and 29 Hill’s muscle models to a desired end position.

Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

This paper describes sufficient conditions for the existence of optimal policies for partially observable Markov decision processes (POMDPs) with Borel state, observation, and action sets, when the goal is to minimize the expected total costs over finite or infinite horizons. For infinite-horizon problems, one-step costs are either discounted or assumed to be nonnegative. Action sets may be noncompact and one-step cost functions may be unbounded. The introduced conditions are also sufficient for the validity of optimality equations, semicontinuity of value functions, and convergence of value iterations to optimal values. Since POMDPs can be reduced to completely observable Markov decision processes (COMDPs), whose states are posterior state distributions, this paper focuses on the validity of the above-mentioned optimality properties for COMDPs. The central question is whether the transition probabilities for the COMDP are weakly continuous. We introduce sufficient conditions for this and show that the transition probabilities for a COMDP are weakly continuous, if transition probabilities of the underlying Markov decision process are weakly continuous and observation probabilities for the POMDP are continuous in total variation. Moreover, the continuity in total variation of the observation probabilities cannot be weakened to setwise continuity. The results are illustrated with counterexamples and examples.

Competing Cognitive Resilient Networks

We introduce competing cognitive resilient network (CCRN) of mobile radios challenged to optimize data throughput and networking efficiency under dynamic spectrum access and adversarial threats (e.g., jamming). Unlike the conventional approaches, CCRN features both communicator and jamming nodes in a friendly coalition to take joint actions against hostile networking entities. In particular, this paper showcases hypothetical blue force and red force CCRNs and their competition for open spectrum resources. We present state-agnostic and stateful solution approaches based on the decision theoretic framework. The state-agnostic approach builds on multiarmed bandit to develop an optimal strategy that enables the exploratory-exploitative actions from sequential sampling of channel rewards. The stateful approach makes an explicit model of states and actions from an underlying Markov decision process and uses multiagent ${Q}$ -learning to compute optimal node actions. We provide a theoretical framework for CCRN and propose new algorithms for both approaches. Simulation results indicate that the proposed algorithms outperform some of the most important algorithms known to date.

Risk-Sensitive Reinforcement Learning

We derive a family of risk-sensitive reinforcement learning methods for agents, who face sequential decision-making tasks in uncertain environments. By applying a utility function to the temporal difference (TD) error, nonlinear transformations are effectively applied not only to the received rewards but also to the true transition probabilities of the underlying Markov decision process. When appropriate utility functions are chosen, the agents' behaviors express key features of human behavior as predicted by prospect theory (Kahneman & Tversky, 1979 ), for example, different risk preferences for gains and losses, as well as the shape of subjective probability curves. We derive a risk-sensitive Q-learning algorithm, which is necessary for modeling human behavior when transition probabilities are unknown, and prove its convergence. As a proof of principle for the applicability of the new framework, we apply it to quantify human behavior in a sequential investment task. We find that the risk-sensitive variant provides a significantly better fit to the behavioral data and that it leads to an interpretation of the subject's responses that is indeed consistent with prospect theory. The analysis of simultaneously measured fMRI signals shows a significant correlation of the risk-sensitive TD error with BOLD signal change in the ventral striatum. In addition we find a significant correlation of the risk-sensitive Q-values with neural activity in the striatum, cingulate cortex, and insula that is not present if standard Q-values are used.

Transmit Power Control Policies for Energy Harvesting Sensors With Retransmissions

This paper addresses the problem of finding outage- optimal power control policies for wireless energy harvesting sensor (EHS) nodes with automatic repeat request (ARQ)-based packet transmissions. The power control policy of the EHS specifies the transmission power for each packet transmission attempt, based on all the information available at the EHS. In particular, the acknowledgement (ACK) or negative acknowledgement (NACK) messages received provide the EHS with partial information about the channel state. We solve the problem of finding an optimal power control policy by casting it as a partially observable Markov decision process (POMDP). We study the structure of the optimal power policy in two ways. First, for the special case of binary power levels at the EHS, we show that the optimal policy for the underlying Markov decision process (MDP) when the channel state is observable is a threshold policy in the battery state. Second, we benchmark the performance of the EHS by rigorously analyzing the outage probability of a general fixed-power transmission scheme, where the EHS uses a predetermined power level at each slot within the frame. Monte Carlo simulation results illustrate the performance of the POMDP approach and verify the accuracy of the analysis. They also show that the POMDP solutions can significantly outperform conventional ad hoc approaches.

Bounds and Policies for Dynamic Routing in Loss Networks

We consider the problem of maximizing a weighted sum of expected rewards in steady-state in multiclass loss networks under dynamic routing and admission control, with Poisson arrivals and exponentially distributed holding times. By aggregating the underlying Markov decision process, we derive linear programming relaxations that contain the achievable performance region under all admissible policies and lead to a series of progressively tighter upper bounds. These relaxations allow stronger bounds at the expense of higher computational times. We further propose a series of routing and admission control policies from the relaxations that outperform, in computational experiments, other heuristic policies, such as the dynamic alternative routing with trunk reservation and the least busy alternative routing, variations of which are used in practice. The suboptimality guarantees defined as best bound/best policy range from 0 to 2.5% under symmetry conditions (symmetric network topology, arrival rates, link capacities, rewards), and from 4% to 10% under asymmetry conditions. We discuss the qualitative behavior of these policies and obtain insights about their performance.

The Underlying Markov Decision Process in the Single-Leg Airline Yield-Management Problem

We introduce the terms dynamic and static, respectively, to identify the prevailing approaches to the single-leg airline yield-management problem: those allowing customers of different fare classes to book concomitantly (dynamic), and those assuming that the demands for the different fare classes arrive separately in a predetermined order (static). We present a coherent frame-work linking these seemingly disparate models through the underlying dynamic program common to both. We develop a discrete-time Markov decision process formulation mirroring that of Janakiram et al. Transp. Sci. 33, 147–167 (1999) to solve the single-leg problem without cancellations, overbooking, or discounting. Borrowing a result from the queueing-control literature, we prove the concavity of the associated optimal value functions and, subsequently, the optimality of a booking limit policy. We then apply this same technique to the more influential papers from the single-leg literature, at once unifying the static and dynamic models and establishing the connection between the yield-management and queueing-control problems. Finally, we propose an omnibus formulation that yields the static and dynamic models as special cases.

Theory for automatic learning under partially observed Markov-dependent noise

A vigorous branch of automatic learning is directed at the task of locating a global minimum of an unknown multimodal function f( theta ) on the basis of noisy observations L( theta (i))=f( theta (i))+W( theta (i)) taken at sequentially-chosen control points ( theta (i)). In all preceding convergence deviations known to the authors, the noise is postulated to depend on the past only through control selection. Here they allow the observation noise sequence to be stochastically dependent, in particular, a function of an unknown underlying Markov decision process, the observations being the stagewise losses. In a sense, in order to be made precise, the algorithm offered is shown to attain asymptotically optimal performance, and rates are assured. A motivating example from queueing theory is offered, and connections with classical problems of Markov control theory and other disciplines are mentioned. >