Homogeneous Continuous-time Markov Chain Research Articles

Optimal condition based maintenance using attribute Bayesian control chart

Condition-based maintenance (CBM) has been emerged as a relatively new trend in maintenance management. Instead of conducting preventive maintenance actions in specified time intervals, the CBM program collects information through condition monitoring, then recommends maintenance actions based on the observed data. On the other hand, Bayesian control charts use the posterior probability of being the system in an unhealthy state as the chart statistic. An attribute Bayesian control chart is employed in this study to monitor a deteriorating system and plan CBM actions based on a continuous-time homogeneous Markov chain. The system consists of three states: healthy, unhealthy, and failure states. A partially observable Markov decision process (POMDP) is developed, which optimally determines the sample size, sampling interval, and warning limit to minimize the long-term expected cost per time unit. Numerical examples and sensitivity analyses are conducted to clarify the performance of the proposed attribute control chart. To the best of the authors’ knowledge, this is the first study of the applications of attribute Bayesian control charts in condition-based maintenance.

An approximation of the inpatient distribution in hospitals with patient relocation using Markov chains

Many hospitals struggle with insufficient capacity for their inpatients. As a result, hospitals may benefit from an approach that evaluates the occupancy of inpatient wards. In this study, we approximate the occupancy distributions of inpatient wards, accounting for the cases where patients relocate due to a shortage of beds. The approximation employs a homogeneous continuous-time Markov chain to evaluate each ward as a queue containing multiple classes of patients. We avoid computational intractability by evaluating each ward separately and accommodating patients arriving from the remaining wards by interrupting the arrival processes, where the interruption times follow hyper-exponential distributions. Numerical experimentation shows that our approach is robust concerning the type of length-of-stay distribution and generally results in a minor loss of accuracy. Further validation indicates that our model reflects the occupancy distributions of inpatient wards in a Danish hospital.

Non-homogeneous continuous time Markov chain model for information dissemination on Indonesian Twitter users

Nonhomogeneous Continuous-Time Markov Chain (NH-CTMC) is a stochastic process that can be used to model problems where the future state depends only on the current state and is independent of the past. The transition intensity in NH-CTMC is not constant but is a function of time. In this paper, NH-CTMC is employed to model information dissemination on Twitter, where transitions occur only from followee groups to follower groups. Information is considered spread on Twitter when followers retweet posts from their followees. The tweet-retweet process on Twitter satisfies the Markov property, as a retweet from a follower depends only on the tweet posted just before by the corresponding followee. The probability of a tweet spreading is determined by the transition intensity, assumed to be a Sigmoid function whose parameters are estimated using Maximum Likelihood Estimation (MLE). This method is applied to Twitter data from Indonesia related to discussions on Covid-19 vaccination. The results indicate that information about Covid-19 vaccination on Twitter spreads rapidly from followees to followers in the first 20 hours, and then slows down after 40 hours. The NH-CTMC model outperforms the Homogeneous Continuous-Time Markov Chain (H-CTMC) approach, where the transition intensity (tweet spreading intensity) is assumed to be constant.

TAPAS

TAPAS is a new tool for efficient evaluation of dependability and performability attributes of systems composed of many interconnected components. The tool solves homogeneous continuous time Markov chains described by stochastic automata network models structured in submodels with absorbing states. The measures of interest are defined by a reward structure based on submodels composed through transition-based synchronization. The tool has been conceived in a modular and flexible fashion, to easily accommodate new features. Currently, it implements an array of state-based solvers that addresses the state explosion problem through powerful mathematical techniques, including Kronecker algebra, Tensor Trains and Exponential Sums. A simple, yet representative, case study is adopted, to present the tool and to show the feasibility of the supported methods, in particular frommemory consumption point of view.

A Phase-Type Expansion Approach for the Performability of Composite Web Services

The purpose of web service composition is to satisfy complex requirements by constructing a composite service through loosely coupling web services over the Internet. Services available for composition may experience changes in performance and reliability at any time. Therefore, it is desirable to estimate performance and reliability of the composite service based on the data of atomic services such that it is checked up before service deployment whether the service-level agreement is satisfied. In this article, a phase-type expansion approach to establish an expanded homogeneous continuous time Markov chain of composite web services is proposed. By explicitly including failure states and restart states into the model, both performance and reliability can be computed for the composite service via phase-type fitting method based on the observations on execution times of atomic services. Experimental results based on real-world web services are provided to demonstrate the efficacy of the proposed approach.

Information diffusion model with homogeneous continuous time Markov chain on Indonesian Twitter users

In this paper, a homogeneous continuous time Markov chain (CTMC) is used to model information diffusion or dissemination, also to determine influencers on Twitter dynamically. The tweeting process can be modeled with a homogeneous CTMC since the properties of Markov chains are fulfilled. In this case, the tweets that are received by followers only depend on the tweets from the previous followers. Knowledge Discovery in Database (KDD) in Data Mining is used to be research methodology including pre-processing, data mining process using homogeneous CTMC, and post-processing to get the influencers using visualization that predicts the number of affected users. We assume the number of affected users follows a logarithmic function. Our study examines the Indonesian Twitter data users with tweets about covid19 vaccination resulted in dynamic influencer rankings over time. From these results, it can also be seen that the users with the highest number of followers are not necessarily the top influencer.

Bounds on the Rate of Convergence for MtX/MtX/1 Queueing Models

We apply the method of differential inequalities for the computation of upper bounds for the rate of convergence to the limiting regime for one specific class of (in)homogeneous continuous-time Markov chains. Such an approach seems very general; the corresponding description and bounds were considered earlier for finite Markov chains with analytical in time intensity functions. Now we generalize this method to locally integrable intensity functions. Special attention is paid to the situation of a countable Markov chain. To obtain these estimates, we investigate the corresponding forward system of Kolmogorov differential equations as a differential equation in the space of sequences l1.

Synchronization for stochastic coupled networks with Lévy noise via event-triggered control

This paper addresses the realization of almost sure synchronization problem for a new array of stochastic networks associated with delay and Lévy noise via event-triggered control. The coupling structure of the network is governed by a continuous-time homogeneous Markov chain. The nodes in the networks communicate with each other and update their information only at discrete-time instants so that the network workload can be minimized. Under the framework of stochastic process including Markov chain and Lévy process, and the convergence theorem of non-negative semi-martingales, we show that the Markovian coupled networks can achieve the almost sure synchronization by event-triggered control methodology. The results are further extended to the directed topology, where the coupling structure can be asymmetric. Furthermore, we also proved that the Zeno behavior can be excluded under our proposed approach, indicating that our framework is practically feasible. Numerical simulations are provided to demonstrate the effectiveness of the obtained theoretical results.

On reliability analysis of a load-sharing k-out-of-n: G system with interacting Markov subsystems

Dependency plays a key role in the design stage of multi-component manufacturing systems. To capture stochastic dependencies between subsystems, this paper considers a load-sharing -out-of-: G system which contains non-identical multi-state subsystems. The state evolution of each subsystem follows a continuous-time homogeneous Markov chain. Most existing research on -out-of-: G systems focuses on independent subsystems. However, a subsystem failure usually leads to a higher failure rate for each surviving subsystem within the system. This paper aims to study the interaction among subsystems and analyse the system reliability performance. The transition rate matrix for a given subsystem is assumed to depend on the total number of the remaining surviving subsystems, which can be represented as the sum of a baseline transition rate matrix and a stochastic dependency matrix. This paper proves that the transition rate matrix of the system is the generalised Kronecker sum of transition rate matrices of subsystems, and develops an explicit method to calculate the generalised Kronecker sum. The theory of aggregated stochastic processes is employed to obtain closed-form formulas for reliability indexes. A case study of multi-engine aircraft systems is provided where numerical examples are given to illustrate the developed model and obtained results.

Modeling Precious Metal Returns through Fractional Jump-Diffusion Processes Combined with Markov Regime-Switching Stochastic Volatility

This paper is aimed at developing a stochastic volatility model that is useful to explain the dynamics of the returns of gold, silver, and platinum during the period 1994–2019. To this end, it is assumed that the precious metal returns are driven by fractional Brownian motions, combined with Poisson processes and modulated by continuous-time homogeneous Markov chains. The calibration is carried out by estimating the Jump Generalized Autoregressive Conditional Heteroscedasticity (Jump-GARCH) and Markov regime-switching models of each precious metal, as well as computing their Hurst exponents. The novelty in this research is the use of non-linear, non-normal, multi-factor, time-varying risk stochastic models, useful for an investors’ decision-making process when they intend to include precious metals in their portfolios as safe-haven assets. The main empirical results are as follows: (1) all metals stay in low volatility most of the time and have long memories, which means that past returns have an effect on current and future returns; (2) silver and platinum have the largest jump sizes; (3) silver’s negative jumps have the highest intensity; and (4) silver reacts more than gold and platinum, and it is also the most volatile, having the highest probability of intensive jumps. Gold is the least volatile, as its percentage of jumps is the lowest and the intensity of its jumps is lower than that of the other two metals. Finally, a set of recommendations is provided for the decision-making process of an average investor looking to buy and sell precious metals.

Modeling of Repairable Multi-State Systems with time-varying failure and repair rates using Equivalent Systems Dynamics Models

This paper treats with the reliability assessment of a Repairable Multi-State System (RMSS) by means of a Nonhomogeneous Continuous-Time Markov Chain (NH-CTMC). A RMSS run on different operating conditions that may be considered acceptable or unacceptable according to a defined demand level. In these cases, the commonly used technique is Homogeneous Continuous-Time Markov Chain (H-CTMC), since its solution is mathematically tractable. However, the H-CMTC involve that the time between state transitions is exponentially distributed, and the failure and repair rates are constants. It's certainly not true if the system components age with the operation or if the repair activities depend on the instant of time when the failure occurred. In these cases, the failure and repair rates are time-varying and the NH-CTMC is needed to be considered. Nevertheless, for these models the analytical solution may not exist and the use of others techniques is required. This paper proposes the use of an Equivalent Systems Dynamics Model (ESDM) to model a NH-CTMC. A ESDM represent the Markov Model (MM) by means of the language and the tools of the Systems Dynamics (SD), and the results are obtained by simulation. As an example, an RMSS with three components, failure rates associated with the Weibull distribution and repair rates associated with the Log-logistic distribution is developed. This example serves to identify the advantages and disadvantages of a ESDM to make model a RMSS and evaluate some reliability measures.

Application of Markov renewal theory and semi‐Markov decision processes in maintenance modeling and optimization of multi‐unit systems

AbstractIn this paper, a condition‐based maintenance model for a multi‐unit production system is proposed and analyzed using Markov renewal theory. The units of the system are subject to gradual deterioration, and the gradual deterioration process of each unit is described by a three‐state continuous time homogeneous Markov chain with two working states and a failure state. The production rate of the system is influenced by the deterioration process and the demand is constant. The states of the units are observable through regular inspections and the decision to perform maintenance depends on the number of units in each state. The objective is to obtain the steady‐state characteristics and the formula for the long‐run average cost for the controlled system. The optimal policy is obtained using a dynamic programming algorithm. The result is validated using a semi‐Markov decision process formulation and the policy iteration algorithm. Moreover, an analytical expression is obtained for the calculation of the mean time to initiate maintenance using the first passage time theory.

Markov-Modulated Linear Regression Parameter Estimation Using a Convolution of Exponential Densities

Markov-modulated linear regression model is a special case of the Markov-additive process (𝒀, 𝑱) = {(𝒀(𝒕), 𝑱(𝒕)), 𝒕 ≥ 𝟎}, where component J is called Markov, and component Y is additive and described by a linear regression. The component J is a continuous-time homogeneous irreducible Markov chain with the known transition intensities between the states. Usually this Markov component is called the external environment or background process. Unknown regression coefficients depend on external environment state, but regressors remain constant. This research considers the case, when the Markov property is not satisfied, namely, the sojourn time in each state is not exponentially distributed. Estimation procedure for unknown model parameters is described when it’s possible to represent transition intensities as a convolution of exponential densities. An efficiency of such an approach is evaluated by a simulation.

Context-Aware Quantification for VANET Security: A Markov Chain-Based Scheme

Recently, the quantification of VANET security has drawn significant attention due to the lack of standard computational metrics. The salient features of VANET, such as highly dynamic connections, sensitive information sharing, and unreliable fading channels, make the security quantification challenging. Accurate measurement for VANET security depends on the sufficient understanding of “context”, or making sense of the states, environment, or situation. This article proposes a context-aware security quantification scheme for VANET based on the Markov chain. Firstly, a Homogeneous Continuous-Time Markov Chain (HCTMC)-based security state model is designed for VANET. The value of each state of the HCTMC is determined with a value function that incorporates the security strength of transmitted data, dynamic and randomness of the vehicular channel, and transmission delay of the current situated environment of VANETs. Finally, the state transition matrix is derived based on the Homogeneous Discrete-Time Markov Chain (HDTMC) and Homogeneous Poisson Process (HPP). Simulation results show that the security quantification method enables the VANET’s system to adopt context-aware defense strategies according to the situated environment.

Bounding inferences for large-scale continuous-time Markov chains: A new approach based on lumping and imprecise Markov chains

If the state space of a homogeneous continuous-time Markov chain is too large, making inferences becomes computationally infeasible. Fortunately, the state space of such a chain is usually too detailed for the inferences we are interested in, in the sense that a less detailed—smaller—state space suffices to unambiguously formalise the inference. However, in general this so-called lumped state space inhibits computing exact inferences because the corresponding dynamics are unknown and/or intractable to obtain. We address this issue by considering an imprecise continuous-time Markov chain. In this way, we are able to provide guaranteed lower and upper bounds for the inferences of interest, without suffering from the curse of dimensionality.

Multi-mission selective maintenance modelling for multistate systems over a finite time horizon

In the life cycle of a long-life structure, multiple maintenances are often required to ensure safety or normal performance. This article developed a multi-mission selective maintenance optimization model for a system with several multistate components over a finite time horizon. The model aims to minimize the total cost by determining the optimal intervention time and the corresponding maintenance option of each component. The system reliability requirement is taken as the constraint. The degradation process of each component is characterized by a homogeneous continuous time Markov chain in this model. Considering the uncertainties that exist during the deterioration–maintenance process, the condition state of each component at any time is described by a state probability vector. In addition, the components in the system are considered to be structurally dependent during the maintenance process, and a directed graph is used to represent the precedence relations among these components. A differential evolution algorithm is utilized to solve this complicated optimization model. A numerical example and an engineering application are presented to illustrate the availability of the proposed model.

Synchronization of Nonlinearly and Stochastically Coupled Markovian Switching Networks via Event-Triggered Sampling.

This paper studies the exponential synchronization problem for a new array of nonlinearly and stochastically coupled networks via events-triggered sampling (ETS) by self-adaptive learning. The networks include the following features: 1) a Bernoulli stochastic variable is introduced to describe the random structural coupling; 2) a stochastic variable with positive mean is used to model the coupling strength; and 3) a continuous time homogeneous Markov chain is employed to characterize the dynamical switching of the coupling structure and pinned node sets. The proposed network model is capable to capture various stochastic effect of an external environment during the network operations. In order to reduce networks' workload, different ETS strategies for network self-adaptive learning are proposed under continuous and discrete monitoring, respectively. Based on these ETS approaches, several sufficient conditions for synchronization are derived by employing stochastic Lyapunov-Krasovskii functions, the properties of stochastic processes, and some linear matrix inequalities. Numerical simulations are provided to demonstrate the effectiveness of the theoretical results and the superiority of the proposed ETS approach.

Statistical Identification of Markov Chain on Trees

The theoretical study of continuous-time homogeneous Markov chains is usually based on a natural assumption of a known transition rate matrix (TRM). However, the TRM of a Markov chain in realistic systems might be unknown and might even need to be identified by partially observable data. Thus, an issue on how to identify the TRM of the underlying Markov chain by partially observable information is derived from the great significance in applications. That is what we call the statistical identification of Markov chain. The Markov chain inversion approach has been derived for basic Markov chains by partial observation at few states. In the current letter, a more extensive class of Markov chain on trees is investigated. Firstly, a type of a more operable derivative constraint is developed. Then, it is shown that all Markov chains on trees can be identified only by such derivative constraints of univariate distributions of sojourn time and/or hitting time at a few states. A numerical example is included to demonstrate the correctness of the proposed algorithms.

Transient performance analysis of smart grid with dynamic power distribution

Transient performance analysis of power distribution network (PDN) after a failure occurrence could facilitate the better design of smart grid. Researchers have proposed analytical models and the numerical solutions to analyze the PDN's transient behaviors by applying homogeneous continuous-time Markov chain (CTMC). However, the PDN system may be time-varying during a failure recovery. Then, the system restoration process evolves as a non-homogeneous CTMC (NHCTMC) on a finite state space. This paper seeks to analyze the transient performance of such a time-varying system from a failure occurrence until the system's full restoration. This restoration process consists of multiple phases which are sequential or in parallel. We apply piecewise constant approximation method to derive the formulas for computing state transient probabilities and then derive the computation formulas for the metrics of interest. A case study is conducted to apply the proposed approach to analyze the transient performance of a simple distribution automation network. This network was derived from a real power distribution network.

Markov-modulated multivariate linear regression

The article concerns parameter estimation for the Markov-modulated multivariate linear regression model. It is supposed that the parameters of the linear regression are dependent from states of a random environment. The last is described as a continuous-time homogeneous irreducible Markov chain with known parameters. The procedure of estimating the regression parameters is established.