Inhomogeneous Continuous-time Markov Chains Research Articles

Numerical Computation of Distributions in Finite-State Inhomogeneous Continuous Time Markov Chains, Based on Ergodicity Bounds and Piecewise Constant Approximation

In this paper it is shown, that if a possibly inhomogeneous Markov chain with continuous time and finite state space is weakly ergodic and all the entries of its intensity matrix are locally integrable, then, using available results from the perturbation theory, its time-dependent probability characteristics can be approximately obtained from another Markov chain, having piecewise constant intensities and the same state space. The approximation error (the taxicab distance between the state probability distributions) is provided. It is shown how the Cauchy operator and the state probability distribution for an arbitrary initial condition can be calculated. The findings are illustrated with the numerical examples.

Infrastructure deterioration modeling with an inhomogeneous continuous time Markov chain: A latent state approach with analytic transition probabilities

AbstractMarkov chains have been widely used to characterize performance deterioration of infrastructure assets, to model maintenance effectiveness, and to find the optimal intervention strategies. For long‐lived assets such as bridges, the time‐homogeneity assumptions of Markov chains should be carefully checked. For this purpose, this research proposes a regime‐switching continuous‐time Markov chain of which the state transition probabilities depend on another, latent, Markov chain that characterizes the overall aging regime of an asset. With the aid of a state‐augmentation technique, closed‐form solutions for the transition probabilities are analytically derived, making the statistical analysis simple. A case study is presented using the open Ontario Bridge Condition data for provincial highway bridges. The case study demonstrates that the proposed method allows to (1) estimate a statistically superior model to the homogeneous Markov chain and (2) obtain results with comparable accuracy in approximately 48% of the computation time of the state‐of‐the‐art inhomogeneous Markov chain.

The Application of Continuous-Time Markov Chain Models in the Analysis of Choice Flume Experiments

Abstract An inhomogeneous continuous-time Markov chain model is proposed to quantify animal preference and avoidance behaviour in a choice experiment. We develop and apply our model to a choice flume experiment designed to assess the preference or avoidance responses of sea bass (Dicentrarchus labrax) exposed to chlorinated seawater. Due to observed fluctuations in chlorine levels, a stochastic process was applied to describe and account for uncertainty in chlorine concentrations. A hierarchical model was implemented to account for differences between eight experimental runs and use Bayesian methods to quantify preference/avoidance after accounting for observed shoaling behaviour. The application of our method not only overcomes the need to track individuals during an experiment but also circumvents temporal autocorrelation and any violations of independence. Our model therefore surpasses current methods in choice chamber studies, incorporating variability in the environment and group-level dynamics to yield results that scale and generalise to the real-world.

Facilitating Numerical Solutions of Inhomogeneous Continuous Time Markov Chains Using Ergodicity Bounds Obtained with Logarithmic Norm Method

The problem considered is the computation of the (limiting) time-dependent performance characteristics of one-dimensional continuous-time Markov chains with discrete state space and time varying intensities. Numerical solution techniques can benefit from methods providing ergodicity bounds because the latter can indicate how to choose the position and the length of the “distant time interval” (in the periodic case) on which the solution has to be computed. They can also be helpful whenever the state space truncation is required. In this paper one such analytic method—the logarithmic norm method—is being reviewed. Its applicability is shown within the queueing theory context with three examples: the classical time-varying M/M/2 queue; the time-varying single-server Markovian system with bulk arrivals, queue skipping policy and catastrophes; and the time-varying Markovian bulk-arrival and bulk-service system with state-dependent control. In each case it is shown whether and how the bounds on the rate of convergence can be obtained. Numerical examples are provided.

Optimal investment decision under switching regimes of subsidy support

We address the problem of making a managerial decision when the investment project is subsidized, which results in the resolution of an infinite-horizon optimal stopping problem of a switching diffusion driven by either a homogeneous or an inhomogeneous continuous-time Markov chain. We provide a characterization of the value function (and optimal strategy) of the optimal stopping problem. On the one hand, broadly, we can prove that the value function is the unique viscosity solution to a system of HJB equations. On the other hand, when the Markov chain is homogeneous and the switching diffusion is one-dimensional, we obtain stronger results: the value function is the difference between two convex functions.

Asymptotic expansions for solutions of parabolic systems associated with multi-scale switching diffusions

This work develops asymptotic expansions of systems of partial differential equations associated with multi-scale switching diffusions. The switching process is modeled by using an inhomogeneous continuous-time Markov chain. In the model, there are two small parameters e and δ. The first one highlights the fast switching, whereas the other delineates the slow diffusion. Assuming that e and δ are related in that e = δ γ , our results reveal that different values of γ lead to different behaviors of the underlying systems, resulting in different asymptotic expansions. Although our motivation comes from stochastic problems, the approach is mainly analytic and is constructive. The asymptotic series are rigorously justified with error bounds provided. An example is provided to demonstrate the results.

Truncation Bounds for Approximations of Inhomogeneous Continuous-Time Markov Chains

Weakly ergodic continuous-time countable Markov chains are studied. We obtain uniform in time bounds for approximations via truncations by analogous smaller chains under some natural assumptions.MSC codesinhomogeneous continuous-time Markov processesapproximationstruncationsweak ergodicity

Computational Security for Context-Awareness in Vehicular Ad-Hoc Networks

Vehicular ad hoc networks can be viewed as a typical context-aware system where the experienced context frequently varies as vehicles move along road, e.g., signal-to-noise ratio (SNR), velocity, and traffic flow. In particular, the adopted security protection mechanisms often depend on the node state, location, and/or surrounding risk, which need the capability of context-aware security quantification. This paper views the security level as a user’s inherent property that is only correlated with the user’s behaviours and the situated context and independent of the suffered attack ways. We propose a formalized methodology to especially quantify the security level in real time from the perspective of state transition probability through estimating the stable probability of staying in the security state in inhomogeneous continuous time Markov chain. This paradigm enables users to customize the security protection mechanisms for adapting to the frequently varying context. We conduct the extensive numerical calculations and empirical analysis to comprehensively investigate the response of the proposed security quantification framework to the various combinations of the concerned parameters, e.g., SNR, velocity, and traffic flow. The results show that the proposed framework is capable of capturing the real-time security level adaptively to the vehicular context and provides a dependable decision basis to security protection, which can restrict the security to a target value.

Euler–Lagrange approach to model heterogeneities in stirred tank bioreactors – Comparison to experimental flow characterization and particle tracking

The aim of this work is the validation of an Euler–Lagrange modeling approach coupling a CFD-based compartment model (Eulerian approach) and a stochastic model based on a Continuous-Time Markov Chain (Lagrangian approach). The turbulent flow structure and the mixing process in a bioreactor stirred by an axial Mixel TT impeller is characterized by PIV and tracer experiments. Comparison between experimental and numerical data shows that the CFD-based compartment model is able to reproduce accurately the spatial heterogeneities inside the bioreactor. The trajectory of a small tracer particle which perfectly follows the fluid flow is measured by optical trajectography. It is then simulated by a stochastic model which is either based on an homogeneous or on an inhomogeneous Continuous Time Markov Chain (CTMC). Comparison of residence and circulation time distributions in three zones defined inside the bioreactor shows that the inhomogeneous CTMC model predicts with an excellent accuracy the particle trajectories inside the bioreactor. The modeling approach proposed here could be an useful tool to design scale-down bioreactors in order to reproduce at lab-scale the stress levels encountered in large-scale production bioreactors and to characterize and compare different bioreactor configurations.

On the Numerical Analysis of Inhomogeneous Continuous-Time Markov Chains

Inhomogeneous continuous-time Markov chains play an important role in different application areas. In contrast to homogeneous continuous-time Markov chains, where a large number of numerical analysis techniques are available and have been compared, few results about the performance of numerical techniques in the inhomogeneous case are known. This paper presents a new variant of the uniformization technique, the most efficient approach for homogeneous Markov chains. The new uniformization technique allows for the stable computation of strict bounds for the transient distribution of inhomogeneous continuous-time Markov chains, which is not possible with other numerical techniques that provide only an approximation of the distribution and asymptotic bounds. Furthermore, another variant of uniformization is presented that computes an approximation of the transient distribution and is shown to outperform standard differential equation solvers if transition rates change slowly.

Uniform quasi-ergodicity and stability of inhomogeneous continuous-time Markov chains

Uniform quasi-ergodicity and stability of inhomogeneous continuous-time Markov chains