Discrete-time Markov Decision Model Research Articles

A partially observable discrete time Markov decision process with a fractional discounted reward

We give a new partially observable discrete time Markov decision model with general state and action spaces, under a fractional discounted criterion. In order to discuss an optimal policy for this model, by using the parametric method we transform this model to a partially observable Markov decision model under the usual discounted criterion, which is moreover transformed to the equivalent completely observable Markov decision model by using Bayes’s formula and the notion of an information policy. The existence, uniqueness and the continuity property of the solution of the optimality equation, and the optimal value function related our model are discussed.

Optimal condition-based harvesting policies for biomanufacturing operations with failure risks

ABSTRACTThe manufacture of biological products from live systems such as bacteria, mammalian, or insect cells is called biomanufacturing. The use of live cells introduces several operational challenges including batch-to-batch variability, parallel growth of both desired antibodies and unwanted toxic byproducts in the same batch, and random shocks leading to multiple competing failure processes. In this article, we develop a stochastic model that integrates the cell-level dynamics of biological processes with operational dynamics to identify optimal harvesting policies that balance the risks of batch failures and yield/quality tradeoffs in fermentation operations. We develop an infinite horizon, discrete-time Markov decision model to derive the structural properties of the optimal harvesting policies. We use IgG1 antibody production as an example to demonstrate the optimal harvesting policy and compare its performance against harvesting policies used in practice. We leverage insights from the optimal policy to propose smart stationary policies that are easier to implement in practice.

Emergency orders in the periodic-review inventory system with fixed ordering costs and compound Poisson demand

Emergency orders have shorter lead times but incur extra costs compared to normal orders. We present a discrete-time Markov decision model where normal orders are issued according to a reorder point policy with a fixed order quantity, whereas emergency orders are controlled by a state-dependent reorder point policy with a target stock level. A rapid policy iteration algorithm is used to find and evaluate the policy that minimizes the long-run average cost per review period. In addition to fixed and variable costs for normal and emergency orders our model includes linear holding and backorder costs. The review period is of any given length. Neither the normal order nor the emergency order lead time are required to be integer multiples of the review period.The numerical study shows that the mixed policy found from our Markov decision model generally outperforms the best pure replenishment policy using either only normal or only emergency orders. Our model provides results that are similar to or slightly better than the results obtained with earlier models in the literature. Moreover, because our model accommodates compound Poisson demand, we are able to demonstrate that considerable cost reductions can then be obtained with the mixed policy when compared to the best pure replenishment policy. Finally, using sensitivity analysis we observe that a result on when a mixed policy is most beneficial, which has been found to hold for the simpler model without fixed ordering costs, seems to hold also for the more complex model that we investigate.

Optimal maintenance of two stochastically deteriorating machines with an intermediate buffer

We consider a manufacturing system in which an input generating installation transfers a raw material to a subsequent production unit. Both machines deteriorate stochastically with usage and may fail. For each machine the deteriorating process is described by some known transition probabilities between different degrees of deterioration. A buffer has been built between the two machines in order to cope with unexpected failures of the installation. A discrete-time Markov decision model is formulated for the optimal preventive maintenance of both machines. The maintenance times are geometrically distributed and the cost structure includes operating costs, storage costs, maintenance costs and costs due to the lost production. It is proved that for fixed buffer content and for fixed deterioration degree of one machine, the average-cost optimal policy initiates a preventive maintenance of the other machine if and only if its degree of deterioration exceeds some critical level. We study, by means of numerical results, the effect of the variation of some parameters on the optimal policy and on the minimum average cost. For the case in which the maintenance times follow continuous distributions, an approximate discrete-time Markov decision model is proposed.

Cost-Effective Inventory Control Under Risk

Inventory systems play an important role in manufacturing and service providing organisations to support fluency of production or service. Evolution in time and environment of risk usually characterise operation of such systems. Proper control rules are required for decision making on replenishment orders. If stochastic dynamics of system motion exhibits the Markovian property, then Markov decision models can be employed to reveal a cost-effective inventory control policy. A single-item periodic-review inventory system in the maintenance centre of amanufacturing or transportation organisation with aPoisson arrival stream of demand requirements for spare components is introduced in the paper. The corresponding discrete-time Markov decision model with finite state and action spaces is described for sequential decisions on order sizes over an infinite planning horizon. The long-run expected average cost per unit time is the criterion of interest to be minimised by an optimal inventory control policy.

Finite state Markov decision models with average reward criteria

This paper deals with a discrete time Markov decision model with a finite state space, arbitrary action space, and bounded reward function under the average reward criteria. We consider four average reward criteria and prove the existence of persistently nearly optimal strategies in various classes of strategies for models with complete state information. We show that such strategies exist in any class of strategies satisfying the following condition: along any trajectory at different epochs the controller knows different information about the past. Though neither optimal nor stationary nearly optimal strategies may exist, we show that for some nonempty set of states the described nearly optimal strategies may be chosen either stationary or optimal.

On Stationary Strategies in Borel Dynamic Programming

We consider a discrete time Markov decision model with Borel state and action spaces. It is proved that the supremum of the expected total rewards under all randomized stationary strategies is equal to the supremum of these rewards under all (nonrandomized) stationary strategies.

Sensitivity-analysis in discounted Markovian decision problems

This paper deals with a finite-state, finiteaction discrete-time Markov decision model. A linear programming procedure is developed for the computation of optimal policies over the entire range of the discount factor. Furthermore, a procedure is presented for the computation of a Blackwell optimal policy.

Markov decision chains with unbounded costs and applications to the control of queues

A discrete-time Markov decision model with a denumerable set of states and unbounded costs is considered. It is shown that the optimality equation of dynamic programming along with some additional, easily checked, conditions may be used to establish the optimality or ∊ -optimality of policies with respect to the average expected cost criterion. The results are used to derive optimal policies in two queueing examples.

Markov decision chains with unbounded costs and applications to the control of queues

A discrete-time Markov decision model with a denumerable set of states and unbounded costs is considered. It is shown that the optimality equation of dynamic programming along with some additional, easily checked, conditions may be used to establish the optimality or ∊ -optimality of policies with respect to the average expected cost criterion. The results are used to derive optimal policies in two queueing examples.

The asymptotic behaviour of the minimal total expected cost for the denumerable state Markov decision model

This paper considers the discrete time Markov decision model with a denumerable state space and finite action space. Under certain conditions it is proved that the minimal total expected cost for a planning horizon of n epochs minus n times the minimal long-run average expected cost per unit time has a finite limit as n → ∞ for each initial state.

The asymptotic behaviour of the minimal total expected cost for the denumerable state Markov decision model

This paper considers the discrete time Markov decision model with a denumerable state space and finite action space. Under certain conditions it is proved that the minimal total expected cost for a planning horizon of n epochs minus n times the minimal long-run average expected cost per unit time has a finite limit as n → ∞ for each initial state.

A new optimality criterion for discrete dynamic programming

This paper concerns a discrete-time Markov decision model with an infinite planning horizon. A new optimality criterion and the related optimal policy, termed R-optimal one, are proposed. The criterion is much effective comparing with the existing criteria because of its availability both for discounting case and nondiscounting case in the same form. It is shown that there exists a stationary R-optimal policy and it can be found in finitely many steps by the policy iteration method.

Computing a Bias-Optimal Policy in a Discrete-Time Markov Decision Problem

This paper treats a discrete-time Markov decision model with an infinite planning horizon and no discounting. A “bias-optimal” policy for this decision problem satisfies a criterion that is more selective than maximizing the gain rate. The problem of computing a bias-optimal policy, also treated by Veinott in 1966, is here parsed into a sequence of three simple Markov decision problems, each of which can be solved by linear programming or policy iteration.