Continuous-time Markov Decision Processes Research Articles

Optimization problems in chemical reactions using continuous-time Markov chains

In this paper we generalize two models for chemical reactions, based on continuous-time Markov chains, into continuous-time Markov decision process. We propose a mathematical optimization approach for solving an average optimality criterion in state-discrete continuous-time Markov decision process. Our proposal extends the c-variable method used in discrete-time decision process by introducing a new linear constraint for continuous time. The advantage of our approach is that it reduces the continuous-time Markov decision process to a discrete-time Markov decision process where the linear constraints make the problem computationally tractable. The usefulness of the method is illustrated in chemical reactions where the concentration dynamics is modelled as a continuous-time Markov chain. The first application is a single reversible reaction for the formation of the amidogen radical where we found the optimal temperature that minimizes the average expected rate of H formation at steady state. The second is a chemical reaction network for the proton transfer, hydration and tautomeric reaction of anthocyanin pigments, in this case we found an optimal strategy over a set of values of \(\hbox {H}^{+}\) that minimizes the average expected total number of molecules at steady state.

Efficient approximation of optimal control for continuous-time Markov games

We study the time-bounded reachability problem for continuous-time Markov decision processes (CTMDPs) and games (CTMGs). Existing techniques for this problem use discretisation techniques to partition time into discrete intervals of size ε, and optimal control is approximated for each interval separately. Current techniques provide an accuracy of O(ε2) on each interval, which leads to an infeasibly large number of intervals. We propose a sequence of approximations that achieve accuracies of O(ε3), O(ε4), and O(ε5), that allow us to drastically reduce the number of intervals that are considered. For CTMDPs, the performance of the resulting algorithms is comparable to the heuristic approach given by Buchholz and Schulz, while also being theoretically justified. All of our results generalise to CTMGs, where our results yield the first practically implementable algorithms for this problem. We also provide memoryless strategies for both players that achieve similar error bounds.

Finite-horizon optimality for continuous-time Markov decision processes with unbounded transition rates

In this paper we focus on the finite-horizon optimality for denumerable continuous-time Markov decision processes, in which the transition and reward/cost rates are allowed to be unbounded, and the optimality is over the class of all randomized history-dependent policies. Under mild reasonable conditions, we first establish the existence of a solution to the finite-horizon optimality equation by designing a technique of approximations from the bounded transition rates to unbounded ones. Then we prove the existence of ε (≥ 0)-optimal Markov policies and verify that the value function is the unique solution to the optimality equation by establishing the analog of the Itô-Dynkin formula. Finally, we provide an example in which the transition rates and the value function are all unbounded and, thus, obtain solutions to some of the unsolved problems by Yushkevich (1978).

Transient Reward Approximation for Continuous-Time Markov Chains

We are interested in the analysis of very large continuous-time Markov chains (CTMCs) with many distinct rates. Such models arise naturally in the context of reliability analysis, e.g., of computer network performability analysis, of power grids, of computer virus vulnerability, and in the study of crowd dynamics. We use abstraction techniques together with novel algorithms for the computation of bounds on the expected final and accumulated rewards in continuous-time Markov decision processes (CTMDPs). These ingredients are combined in a partly symbolic and partly explicit (symblicit) analysis approach. In particular, we circumvent the use of multi-terminal decision diagrams, because the latter do not work well if facing a large number of different rates. We demonstrate the practical applicability and efficiency of the approach on two case studies.

Finite-horizon optimality for continuous-time Markov decision processes with unbounded transition rates

In this paper we focus on the finite-horizon optimality for denumerable continuous-time Markov decision processes, in which the transition and reward/cost rates are allowed to be unbounded, and the optimality is over the class of all randomized history-dependent policies. Under mild reasonable conditions, we first establish the existence of a solution to the finite-horizon optimality equation by designing a technique of approximations from the bounded transition rates to unbounded ones. Then we prove the existence of ε (≥ 0)-optimal Markov policies and verify that the value function is the unique solution to the optimality equation by establishing the analog of the Itô-Dynkin formula. Finally, we provide an example in which the transition rates and the value function are all unbounded and, thus, obtain solutions to some of the unsolved problems by Yushkevich (1978).

Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach

In this paper, we investigate an optimization problem for continuous-time Markov decision processes with both impulsive and continuous controls. We consider the so-called constrained problem where the objective of the controller is to minimize a total expected discounted optimality criterion associated with a cost rate function while keeping other performance criteria of the same form, but associated with different cost rate functions, below some given bounds. Our model allows multiple impulses at the same time moment. The main objective of this work is to study the associated linear program defined on a space of measures including the occupation measures of the controlled process and to provide sufficient conditions to ensure the existence of an optimal control.

Optimal query assignment for wireless sensor networks

Increasing computing capabilities of modern sensors have enabled current wireless sensor networks to process queries within the network. This complements the traditional features of the sensor networks such as sensing the environment and communicating the data. Query processing, however, poses Quality of Service challenges such as query waiting time and validity (age) of the data. We focus on the processing cost of queries as a trade-off between the time queries wait to be processed and the age of the data provided to the queries. To model this trade-off, we propose a Continuous Time Markov Decision Process which assigns queries either to the sensor network, where queries wait to be processed, or to a central database, which provides stored and possibly outdated data. To compute an optimal query assignment policy, a Discrete Time Markov Decision Process, shown to be stochastically equivalent to the initial continuous time process, is formulated. A comparative numerical analysis of the performance of the optimal assignment policy and of several heuristics, derived from practice, is performed. This provides a theoretical support for the design and implementation of WSN applications, while ensuring a close-to-optimum performance of the system.

Optimal Threshold Policy for In-Home Smart Grid with Renewable Generation Integration

In-home Smart Grid (SG), the integration of Renewable Power Systems (RPSs) with Conventional Power Systems (CPSs), calls for cost-effective management for the electricity usages of end users' household appliances. In this paper, by taking the charging process of RPSs and multiple types of household appliances in to consideration, we have developed analytical models to characterize the electricity cost in the in-home smart grid. Based on these models, we formulate the electricity cost minimization problem as a finite-horizon continuous-time Markov decision process (CTMDP), from which we obtain a threshold policy to minimize the cost. Numerical results show that the threshold policy can manage the electricity usage very effectively.

Impulsive Control for Continuous-Time Markov Decision Processes

In this paper our objective is to study continuous-time Markov decision processes on a general Borel state space with both impulsive and continuous controls for the infinite time horizon discounted cost. The continuous-time controlled process is shown to be nonexplosive under appropriate hypotheses. The so-called Bellman equation associated to this control problem is studied. Sufficient conditions ensuring the existence and the uniqueness of a bounded measurable solution to this optimality equation are provided. Moreover, it is shown that the value function of the optimization problem under consideration satisfies this optimality equation. Sufficient conditions are also presented to ensure on the one hand the existence of an optimal control strategy, and on the other hand the existence of a ε-optimal control strategy. The decomposition of the state space into two disjoint subsets is exhibited where, roughly speaking, one should apply a gradual action or an impulsive action correspondingly to obtain an optimal or ε-optimal strategy. An interesting consequence of our previous results is as follows: the set of strategies that allow interventions at time t = 0 and only immediately after natural jumps is a sufficient set for the control problem under consideration.

Impulsive Control for Continuous-Time Markov Decision Processes

In this paper our objective is to study continuous-time Markov decision processes on a general Borel state space with both impulsive and continuous controls for the infinite time horizon discounted cost. The continuous-time controlled process is shown to be nonexplosive under appropriate hypotheses. The so-called Bellman equation associated to this control problem is studied. Sufficient conditions ensuring the existence and the uniqueness of a bounded measurable solution to this optimality equation are provided. Moreover, it is shown that the value function of the optimization problem under consideration satisfies this optimality equation. Sufficient conditions are also presented to ensure on the one hand the existence of an optimal control strategy, and on the other hand the existence of a ε-optimal control strategy. The decomposition of the state space into two disjoint subsets is exhibited where, roughly speaking, one should apply a gradual action or an impulsive action correspondingly to obtain an optimal or ε-optimal strategy. An interesting consequence of our previous results is as follows: the set of strategies that allow interventions at time t = 0 and only immediately after natural jumps is a sufficient set for the control problem under consideration.

PH-graphs for analyzing shortest path problems with correlated traveling times

This paper presents a new approach to model weighted graphs with correlated weights at the edges. Such models are important to describe many real world problems like routing in computer networks or finding shortest paths in traffic models under realistic assumptions. Edge weights are modeled by phase type distributions (PHDs), a versatile class of distributions based on continuous time Markov chains (CTMCs). Correlations between edge weights are introduced by adding dependencies between the PHDs of adjacent edges using transfer matrices. The new model class, denoted as PH graphs (PHGs), allows one to formulate many shortest path problems as the computation of an optimal policy in a continuous time Markov decision process (CTMDP). The basic model class is defined, methods to parameterize the required PHDs and transfer matrices based on measured data are introduced and the formulation of basic shortest path problems as solutions of CTMDPs with the corresponding solution algorithms are also provided. Numerical examples for some typical stochastic shortest path problems demonstrate the usability of the new approach.

Strong average optimality criterion for continuous-time Markov decision processes

This paper deals with continuous-time Markov decision processes with the unbounded transition rates under the strong average cost criterion. The state and action spaces are Borel spaces, and the costs are allowed to be unbounded from above and from below. Under mild conditions, we first prove that the finite-horizon optimal value function is a solution to the optimality equation for the case of uncountable state spaces and unbounded transition rates, and that there exists an optimal deterministic Markov policy. Then, using the two average optimality inequalities, we show that the set of all strong average optimal policies coincides with the set of all average optimal policies, and thus obtain the existence of strong average optimal policies. Furthermore, employing the technique of the skeleton chains of controlled continuous-time Markov chains and Chapman-Kolmogorov equation, we give a new set of sufficient conditions imposed on the primitive data of the model for the verification of the uniform exponential ergodicity of continuous-time Markov chains governed by stationary policies. Finally, we illustrate our main results with an example.

Randomized and Relaxed Strategies in Continuous-Time Markov Decision Processes

One of the goals of this article is to describe a wide class of control strategies, which includes the traditional relaxed strategies, as well as the so called randomized strategies which appeared earlier only in the framework of semi-Markov decision processes. If the objective is the total expected cost up to the accumulation of jumps, then without loss of generality one can consider only Markov relaxed strategies. Under a simple condition, the Markov randomized strategies are also sufficient. An example shows that the mentioned condition is important. Finally, without any conditions, the class of so called Poisson-related strategies is also sufficient in the optimization problems. All the results are applicable to the discounted model, they may be useful also for the case of long-run average cost.

On the First Passage $g$-Mean-Variance Optimality for Discounted Continuous-Time Markov Decision Processes

This paper considers the discounted continuous-time Markov decision processes (MDPs) in Borel spaces and with unbounded transition rates. The discount factors are allowed to depend on states and actions. Main attention is concentrated on the set $F_g$ of stationary policies attaining a given mean performance $g$ up to the first passage of the continuous-time MDP to an arbitrarily fixed target set. Under suitable conditions, we prove the existence of a $g$-mean-variance optimal policy that minimizes the first passage variance over the set $F_g$ using a transformation technique, and also give the value iteration and policy iteration algorithms for computing the $g$-variance value function and a $g$-mean-variance optimal policy, respectively. Two examples are analytically solved to demonstrate the application of our results.

Average Optimality for Continuous-Time Markov Decision Processes Under Weak Continuity Conditions

This paper considers the average optimality for a continuous-time Markov decision process in Borel state and action spaces, and with an arbitrarily unbounded nonnegative cost rate. The existence of a deterministic stationary optimal policy is proved under the conditions that allow the following; the controlled process can be explosive, the transition rates are weakly continuous, and the multifunction defining the admissible action spaces can be neither compact-valued nor upper semicontinuous.

Average Optimality for Continuous-Time Markov Decision Processes Under Weak Continuity Conditions

This paper considers the average optimality for a continuous-time Markov decision process in Borel state and action spaces, and with an arbitrarily unbounded nonnegative cost rate. The existence of a deterministic stationary optimal policy is proved under the conditions that allow the following; the controlled process can be explosive, the transition rates are weakly continuous, and the multifunction defining the admissible action spaces can be neither compact-valued nor upper semicontinuous.

Strong n-discount and finite-horizon optimality for continuous-time Markov decision processes

This paper studies the strong n(n = −1, 0)-discount and finite horizon criteria for continuous-time Markov decision processes in Polish spaces. The corresponding transition rates are allowed to be unbounded, and the reward rates may have neither upper nor lower bounds. Under mild conditions, the authors prove the existence of strong n(n = −1, 0)-discount optimal stationary policies by developing two equivalence relations: One is between the standard expected average reward and strong −1-discount optimality, and the other is between the bias and strong 0-discount optimality. The authors also prove the existence of an optimal policy for a finite horizon control problem by developing an interesting characterization of a canonical triplet.

Formal Synthesis of Control Policies for Continuous Time Markov Processes From Time-Bounded Temporal Logic Specifications

We consider the control synthesis problem for continuous-time Markov decision processes (CTMDPs), whose expected behaviors are measured by the satisfaction of Continuous Stochastic Logic (CSL) formulas. We present an extension of the CSL for CTMDPs involving sequential task specifications with continuous time constraints on the execution of those tasks. We then exploit model checking and time-bounded reachability algorithms to solve the CSL control synthesis problem. To illustrate the feasibility and effectiveness of our approach, we apply our methods to generate the optimal rate constants of a coupled set of biochemical reactions.

A mean-variance optimization problem for continuous-time Markov decision processes

This paper deals with the mean-variance optimization problem for continuous-time Markov decision processes in Polish spaces. The transition and reward rates are allowed to be unbounded, and the paper focuses on an optimality criterion that improves the usual excepted (or mean) discounted reward criterion. Especially, we aim to nd the conditions for the existence of a mean-variance optimal policy under the Polish spaces. First, under suitable conditions, we prove that the variance minimization problem can be transformed into an equivalent discounted-cost optimization problem by using the so-called rst passage decomposition method. Then, we obtain the so-called mean-variance optimality equation and the existence of a mean-variance optimal policy that minimizes the variance over the set of policies with optimal reward. Finally, we present some examples to illustrate our results.

OPTIMAL DYNAMIC PRICING FOR USED PRODUCTS IN REMANUFACTURING OVER AN INFINITE HORIZON

The return of used products (cores) is the beginning of remanufacturing. Although an appropriate pricing policy can effectively manage the returns, a static pricing policy cannot match the returns and demands because of the high uncertainties in both sides, which in turn results in high inventory cost or lost-sale cost. In this paper, we apply a dynamic pricing policy commonly used in retail setting to the core acquisition management in remanufacturing and study the pricing problem for used products with the objective of minimizing average cost over an infinite horizon. We formulate the pricing problem as a continuous-time Markov decision process and characterize the structural properties of the optimal policy. We also conduct a numerical study to investigate the benefit of dynamic pricing.