Discrete-time Markov Decision Process Research Articles

Risk-sensitive discounted Markov decision processes with unbounded reward functions and Borel spaces

This paper attempts to study the risk-sensitive discounted discrete-time Markov decision processes in Borel spaces, in which the reward functions are allowed to be unbounded from above and from below. We find mild conditions imposed on the primitive data of the decision processes, which not only ensure the existence of a solution to the optimality equation (OE in short), but also are the generalization of the bounded reward case. Furthermore, using the OE and a novel technique, we prove the existence of an optimal policy out of the class of randomized history-dependent policies. Finally, we illustrate our results with an inventory system.

Discrete-time Markov decision process for performance analysis of virtual machine allocation schemes in C-RAN

Cloud Radio Access Network (C-RAN) has been proposed as a cloud architecture to provide a common connected resource pool management. It separates the functionalities of the traditional Base Station (BS) into two parts: the Base Band Unit (BBU) and the Remote Radio Head (RRH). BBUs functions are implemented on the Virtual Machines (VMs) in the cloud over commodity hardware, serving User Equipments (UEs) by generating VMs. In this regard, a more accurate adjustment of the number of VMs in a BBU is a challenge. This paper proposes a VM allocation scheme in C-RAN using the hysteresis and migration mechanisms jointly. The goal is to ensure the system’s load balancing and preserve energy. The hysteresis mechanism with different thresholds adjusts the number of active VMs required to serve all UEs by putting idle VMs in sleep mode and activating them as needed. Whereas, the load balancing is achieved by the proposed allocation scheme and the migration of calls from most loaded to least loaded BBU. We use Discrete-Time Markov Decision Processes (MDPs) to enhance both the performance and energy efficiency in the VM allocation schemes. These objectives are achieved by the different decisions of a controller, who should choose the best decision at each time slot. We analyze the performance of the proposed scheme by Probabilistic Model Checking (PMC). Performance requirements are specified using Probabilistic Computation Tree Logic (PCTL) suitable for MDP models. We also check multi-objective properties, by generating Pareto curves, to evaluate the tradeoff based on which the controller can decide. We use the PRISM model checker to build and analyze the MDP models.

Parameter synthesis for Markov models: covering the parameter space

AbstractMarkov chain analysis is a key technique in formal verification. A practical obstacle is that all probabilities in Markov models need to be known. However, system quantities such as failure rates or packet loss ratios, etc. are often not—or only partially—known. This motivates considering parametric models with transitions labeled with functions over parameters. Whereas traditional Markov chain analysis relies on a single, fixed set of probabilities, analysing parametric Markov models focuses on synthesising parameter values that establish a given safety or performance specification $$\varphi $$ φ . Examples are: what component failure rates ensure the probability of a system breakdown to be below 0.00000001?, or which failure rates maximise the performance, for instance the throughput, of the system? This paper presents various analysis algorithms for parametric discrete-time Markov chains and Markov decision processes. We focus on three problems: (a) do all parameter values within a given region satisfy $$\varphi $$ φ ?, (b) which regions satisfy $$\varphi $$ φ and which ones do not?, and (c) an approximate version of (b) focusing on covering a large fraction of all possible parameter values. We give a detailed account of the various algorithms, present a software tool realising these techniques, and report on an extensive experimental evaluation on benchmarks that span a wide range of applications.

A simulation-based approximate dynamic programming approach to dynamic and stochastic resource-constrained multi-project scheduling problem

We consider the dynamic and stochastic resource-constrained multi-project scheduling problem which allows for the random arrival of projects and stochastic task durations. Completing projects generates rewards, which are reduced by a tardiness cost in the case of late completion. Multiple types of resource are available, and projects consume different amounts of these resources when under processing. The problem is modelled as an infinite-horizon discrete-time Markov decision process and seeks to maximise the expected discounted long-run profit. We use an approximate dynamic programming algorithm (ADP) with a linear approximation model which can be used for online decision making. Our approximation model uses project elements that are easily accessible by a decision-maker, with the model coefficients obtained offline via a combination of Monte Carlo simulation and least squares estimation. Our numerical study shows that ADP often statistically significantly outperforms the optimal reactive baseline algorithm (ORBA). In experiments on smaller problems however, both typically perform suboptimally compared to the optimal scheduler obtained by stochastic dynamic programming. ADP has an advantage over ORBA and dynamic programming in that ADP can be applied to larger problems. We also show that ADP generally produces statistically significantly higher profits than common algorithms used in practice, such as a rule-based algorithm and a reactive genetic algorithm.

How good must failure predictions be to make local spare parts stock superfluous?

Thanks to Industry 4.0 technologies, predictive algorithms can provide advance demand information on spare parts demand. Understanding how the goodness of predictions affects on-hand inventory and costs is important for decision makers before integrating these models into existing systems. We consider a spare parts inventory problem for multiple technical systems that are supported by one local stockpoint. Each system has a single critical component that is subject to random failures. Signals are generated to predict component failures. The signal that corresponds to a failure is generated a certain amount of time before the failure, referred to as the demand lead time. However, not every signal results in a failure and some failures are undetected. A component is replaced from the stock when a failure occurs. In case of stock-outs, an emergency shipment takes place. We formulate a discrete-time Markov decision process model to optimize the replenishment decisions with the objective of minimizing the long-run average cost per period. We investigate the effect of precision (i.e., the fraction of true signals among all signals) and sensitivity (i.e., the fraction of detected failures among all failures) of the predictions and the demand lead time on the costs, order-up-to levels, average on-hand inventory and emergency shipments under the optimal policy. In the worst case, the precision, sensitivity or demand lead time is zero. We show analytically that the optimal policy and optimal costs only depend on the sensitivity and the demand lead time through their product. In numerical experiments, we observe a Pareto principle for the reduction of costs in precision (e.g., a 30% perfectness in precision brings a 70% reduction in optimal cost compared to the worst case) and an inverse Pareto principle in the product of sensitivity and demand lead time (e.g., 70% perfectness in the sensitivity or demand lead time only brings 30% reduction in optimal cost compared to the worst case). Finally, we observe that the local spare parts stock only becomes superfluous when the signals are really close to perfect.

Reinforcement Learning for Omega-Regular Specifications on Continuous-Time MDP

Continuous-time Markov decision processes (CTMDPs) are canonical models to express sequential decision-making under dense-time and stochastic environments. When the stochastic evolution of the environment is only available via sampling, model-free reinforcement learning (RL) is the algorithm-of-choice to compute optimal decision sequence. RL, on the other hand, requires the learning objective to be encoded as scalar reward signals. Since doing such translations manually is both tedious and error-prone, a number of techniques have been proposed to translate high-level objectives (expressed in logic or automata formalism) to scalar rewards for discrete-time Markov decision processes. Unfortunately, no automatic translation exists for CTMDPs. We consider CTMDP environments against the learning objectives expressed as omega-regular languages. Omega-regular languages generalize regular languages to infinite-horizon specifications and can express properties given in popular linear-time logic LTL. To accommodate the dense-time nature of CTMDPs, we consider two different semantics of omega-regular objectives: 1) satisfaction semantics where the goal of the learner is to maximize the probability of spending positive time in the good states, and 2) expectation semantics where the goal of the learner is to optimize the long-run expected average time spent in the ''good states'' of the automaton. We present an approach enabling correct translation to scalar reward signals that can be readily used by off-the-shelf RL algorithms for CTMDPs. We demonstrate the effectiveness of the proposed algorithms by evaluating it on some popular CTMDP benchmarks with omega-regular objectives.

Switching decisions in a hybrid MTS/MTO production system comprising multiple machines considering setup

A hybrid make-to-stock (MTS) and make-to-order (MTO) production system was developed and analyzed in this study using a Markov decision process (MDP) model. Previous research successfully modeled cases with and without consideration of setup to obtain optimal solutions for such systems; however, the analyzed models were limited to a single machine. Therefore, this study formulated a hybrid MTS/MTO production system comprising multiple machines using a discrete-time MDP without prioritizing either MTS or MTO. All machines were considered hybrid machines that could be switched between MTS and MTO production by incurring a setup period. The MDP model was formulated using the state space, action space of the switching decision, transition probability, and cost function. The objective was to minimize the average cost per period, which comprised the MTS inventory holding costs, MTO job backlog costs, setup costs, and lost sales costs. A series of numerical investigations was undertaken to examine the number of machines that should be switched simultaneously within a given period and evaluate the dedication of different numbers of machines to a particular process, which are unique conditions to the assumption of multiple machines. The results indicated that switching only one machine at a time in a given period was sufficient and that the average cost remained unchanged even when machines were partially dedicated to MTS and/or MTO production. Thus, the proposed switching decisions were shown to realize the efficient control of hybrid MTS/MTO production systems.

Risk-Sensitive Average Optimality for Discrete-Time Markov Decision Processes

In this paper we study the risk-sensitive average optimality for discrete-time Markov decision processes with denumerable states and unbounded costs. We derive the multiplicative Poisson equation under the suitable ergodicity conditions via an approximation method. Moreover, we prove the existence of a unique solution to the risk-sensitive average cost optimality equation and give an equivalent characterization of the set of all optimal stationary policies. Finally, we present the policy iteration algorithm and show its convergence.

Optimality of a Network Monitoring Agent and Validation in a Real Probe

The evolution of commodity hardware makes it possible to use this type of equipment to implement traffic monitoring systems. A preliminary empirical evaluation of a network traffic probe based on Linux indicates that the system performance has significant losses as the network rate increases. To assess this issue, we consider a model with two tandem queues and a moving server. In this system, we formulate a three-dimensional Markov Decision Process in continuous time. The goal of the proposed model is to determine the position of the server in each time slot so as to optimize the system performance which is measured in terms of throughput. We first formulate an equivalent discrete-time Markov Decision Process and we propose a numerical method to characterize the solution of our problem in a general setting. The solution we obtain in this problem has been tested for a wide range of scenarios and, in all the instances, we observe that the optimality is close to a threshold type policy. We also consider a real probe and we validate the good performance of threshold policies in real applications.

The optimal probability of the risk for finite horizon partially observable Markov decision processes

<abstract><p>This paper investigates the optimality of the risk probability for finite horizon partially observable discrete-time Markov decision processes (POMDPs). The probability of the risk is optimized based on the criterion of total rewards not exceeding the preset goal value, which is different from the optimal problem of expected rewards. Based on the Bayes operator and the filter equations, the optimization problem of risk probability can be equivalently reformulated as filtered Markov decision processes. As an advantage of developing the value iteration technique, the optimality equation satisfied by the value function is established and the existence of the risk probability optimal policy is proven. Finally, an example is given to illustrate the effectiveness of using the value iteration algorithm to compute the value function and optimal policy.</p></abstract>

Increasing Biomanufacturing Yield with Bleed–Feed: Optimal Policies and Insights

Problem definition: Bleed–feed is a novel technology that allows biomanufacturers to skip intermediary bioreactor setups. However, the specific time at which bleed–feed is performed is critical for success. The process is stringently regulated, and its implementation involves unique tradeoffs in operational decision making. Our analysis formalizes the operational challenges related to bleed–feed decisions, and our results inform biomanufacturers and policymakers on the potential impact of this technology on current practice. Academic/practical relevance: Operations management (OM) methodologies have not yet been widely adopted in the biomanufacturing industry. This research presents one of the first attempts to demonstrate how OM can complement biomanufacturing to improve operational decisions. We present a practically relevant problem and a rigorous solution approach that is relevant to both OM and biomanufacturing. Methodology: We develop a finite-horizon, discrete-time Markov decision processes model and analyze the structural characteristics of optimal bleed–feed policies. Moreover, we characterize the behavior of the value function as a function of regulatory restrictions. As a salient feature, the MDP model captures both the biological dynamics of fermentation and the operational tradeoffs in biomanufacturing. Results: We show that optimal bleed–feed policies have a three-way control-limit structure under mild conditions that were validated with industry data. Moreover, our analysis reveals that the marginal benefits of bleed–feed diminish as additional bleed–feeds are performed. As a practically relevant benchmark, we consider a risk-averse heuristic and identify sufficient conditions for its optimality. Managerial implications: Our analysis (supported with an industry case study) shows that bleed–feed implementation can provide benefits. Real-world implementation at MSD resulted in an 82.5% improvement in the batch yield per setup (using one bleed–feed). We find that low-risk fermentation systems benefit the most from bleed–feed implementation. We also find that the performance gap between optimal policies and the risk-averse heuristic is higher when the failure risks or the critical biomass levels are lower. Funding: This work was supported by the Nederlandse Organisatie voor Wetenschappelijk Onderzoek, NWO-VENI scheme. Supplemental Material: The online appendices are available at https://doi.org/10.1287/msom.2022.1163 .

A stochastic dynamic programming approach for the machine replacement problem

This paper addresses both the modeling and the resolution of the replacement problem for a population of machines. The main objective is the computation of a minimum cost replacement policy, which, based on the status of each machine, determines whether one or more machines have to be replaced over a given finite time horizon.The replacement problem of a set of machines can be regarded as a sequential decision-making problem under uncertainty. Thanks to this, we propose a novel formulation for such problems consisting of a composition of discrete-time multi-state Markov Decision Processes (MDPs), one for each specific machine. The underlying optimization problem is formulated as a stochastic Dynamic Programming (DP), and then solved by using the principles of the backward DP algorithm. Moreover, to deal with the curse of dimensionality due to the high-cardinality state–space of real-world/industrial applications, a new generalized multi-trajectory Least-Squares Temporal Difference (LSTD) based method is introduced. The resulting algorithm computes an approximate optimal cost function by: (i) running Monte Carlo simulations over different trajectories of a given length; (ii) embedding the policy improvement step within the recursive LSTD iterations; (iii) enforcing an off-policy mechanism to improve the LSTD exploration capabilities. A study on the convergence properties of the proposed approach is also provided. Several numerical examples are given to illustrate its effectiveness in terms of parametric sensitivity, computational burden, and performance of the computed policies compared with some heuristics defined in the literature.

Critical chain based Proactive-Reactive scheduling for Resource-Constrained project scheduling under uncertainty

Project scheduling problems under both resource constraints and uncertainty have been widely studied due to their real world relevance. In this paper, we design and implement a new integrated proactive–reactive solution approach based on the critical chain method (CCM) to proactively generate a robust and reliable baseline schedule for the class of resource-constrained project scheduling problem (RCPSP) under uncertainty. A discrete-time Markov decision process model is applied for the reactive scheduling phase, which embeds the look-up table method in reinforcement learning to dynamically schedule and adjust schedule reactively using the baseline schedule during project execution. The cost values in the look-up table are calculated based on the occupation of a project buffer and feeding buffers in the baseline schedule generated by the CCM. We conduct computation experiments on the benchmark instances to test our algorithm. The results show that our approach is able to obtain quality solutions efficiently, and competitive with the benchmark algorithms for small- and medium-sized instances.

AoI-Aware Joint Spectrum and Power Allocation for Internet of Vehicles: A Trust Region Policy Optimization-Based Approach

In Internet of Vehicles (IoV), information freshness is a significant indicator to indemnify road traffic safety, which is measured by Age of Information (AoI). In this article, we consider the coexistence scenario of vehicular user pairs and cellular users, where the base station (BS) acts as an agent to allocate channels and transmit power for vehicular user pairs. With the goal of minimizing the sum of the average AoI of all links and the average power consumption of all vehicular user pairs, we formulate this optimization problem as a discrete-time Markov decision process (MDP) problem and adopt the trust region policy optimization (TRPO) algorithm, which has the advantage of fast convergence and high stability. Then, an AoI-aware joint spectrum and power dynamic allocation scheme based on the TRPO algorithm is proposed. Simulation results show that the TRPO-based scheme significantly outperforms both the deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> network (DQN)-based scheme and the random scheme in terms of average cumulative reward, convergence speed, and stability.

Some advances on constrained Markov decision processes in Borel spaces with random state-dependent discount factors

ABSTRACT This paper addresses a class of discrete-time Markov decision processes in Borel spaces with a finite number of cost constraints. The constrained control model considers costs of discounted type with state-dependent discount factors which are subject to external disturbances. Our objective is to prove the existence of optimal control policies and characterize them according to certain optimality criteria. Specifically, by rewriting appropriately our original constrained problem as a new one on a space of occupation measures, we apply the direct method to show solvability. Next, the problem is defined as a convex program, and we prove that the existence of a saddle point of the associated Lagrangian operator is equivalent to the existence of an optimal control policy for the constrained problem. Finally, we turn our attention to multi-objective optimization problems, where the existence of Pareto optimal policies can be obtained from the existence of saddle-points of the aforementioned Lagrangian or equivalently from the existence of optimal control policies of constrained problems.

An AGC Dynamic Optimization Method Based on Proximal Policy Optimization

The increasing penetration of renewable energy introduces more uncertainties and creates more fluctuations in power systems than ever before, which brings great challenges for automatic generation control (AGC). It is necessary for grid operators to develop an advanced AGC strategy to handle fluctuations and uncertainties. AGC dynamic optimization is a sequential decision problem that can be formulated as a discrete-time Markov decision process. Therefore, this article proposes a novel framework based on proximal policy optimization (PPO) reinforcement learning algorithm to optimize power regulation among each AGC generator in advance. Then, the detailed modeling process of reward functions and state and action space designing is presented. The application of the proposed PPO-based AGC dynamic optimization framework is simulated on a modified IEEE 39-bus system and compared with the classical proportional−integral (PI) control strategy and other reinforcement learning algorithms. The results of the case study show that the framework proposed in this article can make the frequency characteristic better satisfy the control performance standard (CPS) under the scenario of large fluctuations in power systems.

Optimal Admission Control for Multiclass Queues with Time-Varying Arrival Rates via State Abstraction

We consider a novel queuing problem where the decision-maker must choose to accept or reject randomly arriving tasks into a no buffer queue which are processed by N identical servers. Each task has a price, which is a positive real number, and a class. Each class of task has a different price distribution, service rate, and arrives according to an inhomogenous Poisson process. The objective is to decide which tasks to accept so that the total price of tasks processed is maximised over a finite horizon. We formulate the problem using a discrete time Markov Decision Process (MDP) with a hybrid state space. We show that the optimal value function has a specific structure, which enables us to solve the hybrid MDP exactly. Moreover, we rigorously prove that as the gap between successive decision epochs grows smaller, the discrete time solution approaches the optimal solution to the original continuous time problem. To improve the scalability of our approach to a greater number of servers and task classes, we present an approximation based on state abstraction. We validate our approach on synthetic data, as well as a real financial fraud data set, which is the motivating application for this work.

Optimal Capacity Rationing Policy for a Container Leasing System with Multiple Kinds of Customers and Substitutable Containers

In this paper, we consider a container leasing firm that has elementary and premium containers, which are downward substitutable and for use by elementary contract customers (ECCs), premium contract customers (PCCs), as well as walk-in customers (WICs). ECCs can be satisfied by elementary containers or premium ones at discounted prices while PCCs only accept premium containers. WICs can be satisfied by any type of container at different prices. The objective is to maximise the expected total rental revenue by managing its limited capacity. We formulate this problem as a discrete-time Markov Decision Process and show the submodularity and concavity of the value function. Based on this, we show that the optimal policy can be characterised by a series of rationing thresholds, a series of substitution thresholds and a priority threshold, all of which depend on the system states. We further give conditions under which the optimal policy can be simplified. Numerical experiments are conducted to show the impact of the substitution of two items on the revenue, to compare the performance of the optimal policy with those of the commonly used policies and to investigate the influence of arrival rates on the optimal policy. Last, we extend the basic model to consider different rental durations, ECCs’ acceptance behaviour and endogenous prices for WICs. This paper was accepted by Jayashankar Swaminathan, operations management. Funding: This work was supported by National Natural Science Foundation of China/Research Grants Council of Hong Kong Joint Research Scheme [Grant 71661167009, N_PolyU531/16], National Natural Science Foundation of China [Grants 72171016, 71831001], and British Academy\Leverhulme Small Research [Grant SRG19\190059]. This work was also supported by Zhejiang Shuren University Research [Grant KXJ0121605] and Beijing Logistics Informatics Research Base. Supplemental Material: The e-companion and data files are available at https://doi.org/10.1287/mnsc.2022.4425 .

When to Preprocess? Keeping Information Fresh for Computing-Enable Internet of Things

Age of information (AoI), a notion that measures the information freshness, is an essential performance measure for time-critical applications in Internet of Things (IoT). With the surge of computing resources at the IoT devices, it is possible to preprocess the information packets that contain the status update before sending them to the destination so as to alleviate the transmission burden. However, the additional time and energy expenditure induced by computing also make the optimal updating a non-trivial problem. In this paper, we consider a time-critical IoT system, where the IoT device is capable of preprocessing the status update before the transmission. Particularly, we aim to jointly design the preprocessing and transmission so that the weighted sum of the average AoI of the destination and the energy consumption of the IoT device is minimized. Due to the heterogeneity in transmission and computation capacities, the durations of distinct actions of the IoT device are non-uniform. Therefore, we formulate the status updating problem as an infinite horizon average cost semi-Markov decision process (SMDP) and then transform it into a discrete-time Markov decision process. We demonstrate that the optimal policy is of threshold type with respect to the AoI. Equipped with this, a structure-aware relative policy iteration algorithm is proposed to obtain the optimal policy of the SMDP. Our analysis shows that preprocessing is more beneficial in regimes of high AoIs, given it can reduce the time required for updates. We further prove the switching structure of the optimal policy in a special scenario, where the status updates are transmitted over a reliable channel, and derive the optimal threshold. Finally, simulation results demonstrate the efficacy of preprocessing and show that the proposed policy outperforms two baseline policies.

A Reinforcement Learning Approach for Optimizing the Age-of-Computing-Enabled IoT

Age of Information (AoI) is a newly rising metric for measuring the freshness of information. In this article, we consider a multidevice computing-enabled Internet of Things (IoT) system with a common destination, in which the status update sampled by the device can be offloaded directly to the destination for computing or computed by the device and then delivered to the destination, and jointly design offloading and scheduling policies to minimize the average weighted sum of AoI and energy consumption. The challenge lies in computing mode selection and its strong coupling with scheduling decisions. To address this issue, we formulate the optimization problem as a bilevel discrete-time Markov decision process (MDP) and approximate the optimal solution by relative value iteration. Furthermore, the threshold structure of the MDP policy is shown. However, with the expansion of the system scale, the MDP policy will suffer from the curse of dimensionality. In light of this, we develop a learning-based algorithm based on emerging deep reinforcement learning (DRL) to reduce the dimensionality of state space and utilize a late experience storage method to train two heterogeneous artificial neural networks (ANNs) synchronously during the training process. Simulation results show the structure of the MDP policy and verify the performance of the DRL policy is near-optimal.