Reliability Model of Cloud Computing Job Scheduling Based on Discrete Time Markov Chain

Linear programming algorithms for semi-Markovian decision processes

In recent years, substantial research has been devoted to monitoring and predicting performance degradations in real-world complex systems within large entities such as nuclear power plants, electrical grids, and distributed computing systems. Special challenges are posed by the fact that such systems operate in uncertain environments, are highly dynamic, and exhibit emergent behaviors that can lead to catastrophic failure. Discrete Time Markov chains (DTMCs) provide important tools for analysis of such systems, because they represent dynamic behavior succinctly, provide a means to measure uncertainty, and can be used to make quantitative measurements of the potential for change to system performance. Moreover, DTMCs can be extended to be time-inhomogeneous, i.e. to represent behavior that varies over long durations. To date, DTMCs have been proposed for tasks such as fault detection and long-term condition equipment monitoring in real-world complex systems. However, the scope of these models has generally been restricted to describing states and state transitions that directly concern fault conditions or states of degradation. Less work has been done on using DTMCs to represent a more complete range of states a system may enter into during normal operation. Of special interest are sequences of states that involve failure scenarios, in which a system evolves from a normal operating state into undesirable state that leads to widespread performance degradation. Unfortunately, use of large DTMCs often involves large search spaces, a problem which in part motivates our work. This paper describes progress made on developing an approach for using larger, more detailed DTMC models of operational complex systems to uncover potential failure scenarios. The approach uses a combination of methods to perturb a DTMC, simulate alternative system evolutions, and identify scenarios in which a system proceeds from normal operation to failure. Key to the approach is the use of graph theory techniques to reduce the size of the search space involved in exploring alternative behaviors. We show how graph theory techniques can be used to identify critical state transitions which can be perturbed to simulate performance degradation. Using critical transitions, it is also possible to estimate the rate of performance degradation and to understand how this rate is likely to change in response to increased failure incidence. Examples are provided of the use of this approach on a DTMC of significant size to identify failure scenarios in a distributed resource allocation system.

This paper investigates the Markov property of the fading process measured on land mobile satellite (LMS) channel and determines the appropriate order of discrete state and discrete time Markov chain, which is capable to reproduce the original fading process. The investigated radio link is an L- band terrestrial-satellite channel, measured on board of a moving vehicle in different environments (urban and highway), therefore the channel is effected by multipath/shadowing propagation impairments. As a first approach of the problem, the analog received power process will be transformed into a digital fading process in order to handle it as a two-state (fade/non-fade according to the given threshold) process. To correctly reproduce the high order statistics of the fading process an appropriate discrete time and discrete state homogenous Markov chain has to be selected. The order of the model influences its capability to reproduce the channel memory. With properly selected threshold, by transforming the analog fading process into a fading/non-fading digital process a Chi-square test can be applied to determine the order of the process. Our investigations are showing that the order of the two-state fade/non-fade process slightly depends on the threshold of separation the fade and interfade events. The results can be applied also to test the order in case of more than two states, when the quantization of the attenuation amplitude performed for multiple levels.

Effective maintenance management is essential to reduce the adverse effect of equipment failure to operation. This can be accomplished by accurately predicting the equipment failure such that appropriate actions can be planned and taken in order to minimize the impact of equipment failure to operation. This paper presents a model to assess system reliability for a degraded multi-state system based on discrete time Markov process and continuous time Markov process. The selection of which model to use is based on the type of available data. The system degradation was quantified by discrete level of system's performance rate with system states ranging from perfect functioning state to complete failure. At any point in time, the system can experience random failures from any degraded state upon which general repair will be performed. This research also explored a method of estimating of transition probabilities as well as definition of states for the Markov process by utilizing system performance data and data clustering method. The results proved the applicability of both discrete time Markov chain and continuous time Markov process in assessing the reliability of multi-state systems using the system's performance data. The results are then utilized to perform equipment replacement analysis due to deterioration based on the expected demand.

We study the problem of model checking Interval-valued Discrete-time Markov Chains (IDTMC). IDTMCs are discrete-time finite Markov Chains for which the exact transition probabilities are not known. Instead in IDTMCs, each transition is associated with an interval in which the actual transition probability must lie. We consider two semantic interpretations for the uncertainty in the transition probabilities of an IDTMC. In the first interpretation, we think of an IDTMC as representing a (possibly uncountable) family of (classical) discrete-time Markov Chains, where each member of the family is a Markov Chain whose transition probabilities lie within the interval range given in the IDTMC. We call this semantic interpretation Uncertain Markov Chains (UMC). In the second semantics for an IDTMC, which we call Interval Markov Decision Process (IMDP), we view the uncertainty as being resolved through non-determinism. In other words, each time a state is visited, we adversarially pick a transition distribution that respects the interval constraints, and take a probabilistic step according to the chosen distribution. We introduce a logic ω-PCTL that can express liveness, strong fairness, and ω-regular properties (such properties cannot be expressed in PCTL). We show that the ω-PCTL model checking problem for Uncertain Markov Chain semantics is decidable in PSPACE (same as the best known upper bound for PCTL) and for Interval Markov Decision Process semantics is decidable in coNP (improving the previous known PSPACE bound for PCTL). We also show that the qualitative fragment of the logic can be solved in coNP for the UMC interpretation, and can be solved in polynomial time for a sub-class of UMCs. We also prove lower bounds for these model checking problems. We show that the model checking problem of IDTMCs with LTL formulas can be solved for both UMC and IMDP semantics by reduction to the model checking problem of IDTMC with ω-PCTL formulas.

We investigate the problem of model checking Interval-valued Discrete-time Markov Chains (IDTMC). IDTMCs are discrete-time finite Markov Chains for which the exact transition probabilities are not known. Instead in IDTMCs, each transition is associated with an interval in which the actual transition probability must lie. We consider two semantic interpretations for the uncertainty in the transition probabilities of an IDTMC. In the first interpretation, we think of an IDTMC as representing a (possibly uncountable) family of (classical) discrete-time Markov Chains, where each member of the family is a Markov Chain whose transition probabilities lie within the interval range given in the IDTMC. This semantic interpretation we call Uncertain Markov Chains (UMC). In the second semantics for an IDTMC, which we call Interval Markov Decision Process (IMDP), we view the uncertainty as being resolved through non-determinism. In other words, each time a state is visited, we adversarially pick a transition distribution that respects the interval constraints, and take a probabilistic step according to the chosen distribution. We show that the PCTL model checking problem for both Uncertain Markov Chain semantics and Interval Markov Decision Process semantics is decidable in PSPACE. We also prove lower bounds for these model checking problems.

Motivated by a wide range of applications arising from stochastic networks (such as communication networks and/or manufacturing systems), this work focuses on a class of large-scale Markov chains in discrete time. In accordance with the rates of change of different states, we formulate the problem as a singularly perturbed Markov chain by introducing a small parameter /spl epsiv/>0. Under simple conditions, we show that aggregated process converges weakly to a Markov chain. In addition, we examine scaled and unscaled occupation measures and obtain their asymptotic properties.

PO-01-156 MARKOV MODELLING OF PHASE SINGULARITY INTERACTION EFFECTS IN HUMAN ATRIAL AND VENTRICULAR FIBRILLATION

A nonhomogeneous discrete-time three-state Markov chain model is developed in this study to quantify the bedload and suspended load discharge under unsteady flow for mixed size sediment particles. When flow is subject to sudden changes, the particle holding time, defined as the amount of time for a sediment particle staying on the bed or in the moving state, needs to be carefully evaluated. The time step used in this study for single-step motion in the discrete-time Markov chain is represented by a characteristic timescale for particle motion. The transition probabilities are functions of flow conditions and particle properties. Specifically, the likelihood of particle movement between the bedload layer and the bed surface is evaluated by the entrainment probability. Exchange of sediment particles between the bedload layer and suspended load layer is quantified by the suspension probability. A nonhomogeneous Markov chain ensures the transition probabilities are time dependent as they are a functio...

The prediction of numbers of newborns is an important issue in hospital management. Relying on the inherent non-aftereffect property, discrete-time Markov chain (DTMC) is a candidate for solving the problem. But the classical DTMC is unable to handle the uncertainty of states, especially when the state space is not discrete, which would lead to instable predicted results. In order to overcome the limitation of the existing DTMC model, a belief Markov chain (BMC) model is proposed by synthesizing the classical DTMC and Dempster-Shafer theory effectively. Depending on the advantages of Dempster-Shafer theory in expressing uncertainty, the proposed BMC model is capable of dealing with various uncertainties, which improves and perfects the classical DTMC model. An illustrative example demonstrates the effectiveness of the proposed model. Moreover, a comparison between the proposed BMC model and the classical and fuzzy states modified DTMC models is given to show the superiority of the proposed model against the other two. Finally, the stability of the proposed model has been proven.

An optimal and a quasi-optimal algorithm for filtering mixed Markov processes in discrete time, in which the discrete component is a Markov chain while the continuous component consists of sections of Markov sequences, are synthesized. Using the criterion of minimum a posteriori risk, a Bayes decision rule is obtained for one form of the loss function. The well-known and the quasi-optimal filtering algorithm synthesized here are compared using statistical modeling on a computer.

We are interested in quasi-stationarity and quasi-ergodicity when the absorbing boundary is moving. First we show that, in the moving boundary case, the quasi-stationary distribution and the quasi-limiting distribution are not well-defined when the boundary is oscillating periodically. Then we show the existence of a quasi-ergodic distribution for any discrete-time irreducible Markov chain defined on a finite space state in the fixed boundary case. Finally we use this last result to show the quasi-ergodicity in the moving boundary case.

Distributions appropriate to the description of long-term behaviour within an irreducible class of discrete-time denumerably infinite Markov chains are considered. The first four sections are concerned with general reslts, extending recent work on this subject. In Section 5 these are applied to the branching process, and give refinements of several well-known results. The last section deals with the semi-infinite random walk with an absorbing barrier at the origin.

Distributions appropriate to the description of long-term behaviour within an irreducible class of discrete-time denumerably infinite Markov chains are considered. The first four sections are concerned with general reslts, extending recent work on this subject. In Section 5 these are applied to the branching process, and give refinements of several well-known results. The last section deals with the semi-infinite random walk with an absorbing barrier at the origin.

Markov jump processes are continuous-time stochastic processes widely used in a variety of applied disciplines. Inference typically proceeds via Markov chain Monte Carlo (MCMC), the state-of-the-art being a uniformization-based auxiliary variable Gibbs sampler. This was designed for situations where the process parameters are known, and Bayesian inference over unknown parameters is typically carried out by incorporating it into a larger Gibbs sampler. This strategy of sampling parameters given path, and path given parameters can result in poor Markov chain mixing.In this thesis, we focus on the problem of path and parameter inference for Markov jump processes.In the first part of the thesis, a simple and efficient MCMC algorithm is proposed to address the problem of path and parameter inference for Markov jump processes. Our scheme brings Metropolis-Hastings approaches for discrete-time hidden Markov models to the continuous-time setting, resulting in a complete and clean recipe for parameter and path inference in Markov jump processes. In our experiments, we demonstrate superior performance over Gibbs sampling, a more naive Metropolis-Hastings algorithm we propose, as well as another popular approach, particle Markov chain Monte Carlo. We also show our sampler inherits geometric mixing from an ‘ideal’ sampler that is computationally much more expensive.In the second part of the thesis, a novel collapsed variational inference algorithm is proposed. Our variational inference algorithm leverages ideas from discrete-time Markov chains, and exploits a connection between Markov jump processes and discrete-time Markov chains through uniformization. Our algorithm proceeds by marginalizing out the parameters of the Markov jump process, and then approximating the distribution over the trajectory with a factored distribution over segments of a piecewise-constant function. Unlike MCMC schemes that marginalize out transition times of a piecewise-constant process, our scheme optimizes the discretization of time, resulting in significant computational savings. We apply our ideas to synthetic data as well as a dataset of check-in recordings, where we demonstrate superior performance over state-of-the-art MCMC methods.