semi-Markov Reward Processes Research Articles

Resilience analysis of multi-state systems with time-dependent behaviors

Most of existing resilience models assume that system performances are continuous. In this paper, we consider resilience modeling and analysis for multi-state systems, whose performances are characterized by discrete, rather than continuous variables. A non-homogeneous Semi-Markov reward process model is developed for resilience analysis of multi-state systems. In the developed model, system performance changes, caused by either disruptive events or system recoveries, are modeled as state transitions, and rewards are used to model financial losses incurred during and after the disruptions. Four resilience metrics are defined to quantify different aspects of resilience. As the developed model is non-homogeneous, it can capture time-dependent system behaviors and their impact on system resilience. An efficient resilience analysis algorithm is also designed based on linear interpolation and implemented using vectorization. The computational benefits of the developed algorithm are demonstrated through two numerical experiments. We apply the developed method on two practical case studies, an oil tank farm and a re-configurable computing system. The results show that the developed methods can quantify resilience of multi-state systems accurately and efficiently.

On the functional central limit theorem for first passage time of nonlinear semi-Markov reward processes

In this article we examine the functional central limit theorem for the first passage time of reward processes defined over a finite state space semi-Markov process. In order to apply this process for a wider range of real-world applications, the reward functions, considered in this work, are assumed to have general forms instead of the constant rates reported in the other studies. We benefit from the martingale theory and Poisson equations to prove and establish the convergence of the first passage time of reward processes to a zero mean Brownian motion. Necessary conditions to derive the results presented in this article are the existence of variances for sojourn times in each state and second order integrability of reward functions with respect to the distribution of sojourn times. We finally verify the presented methodology through a numerical illustration.

Semi-Markov Disability Insurance Models

In this article, we present a stochastic model for disability insurance contracts. The model is based on a discrete time non homogeneous semi-Markov process (DTNHSMP) to which the backward recurrence time process is introduced. This permits a more exhaustive study of disability evolution and a more efficient approach to the duration problem. The use of semi-Markov reward processes facilitates the possibility of deriving equations of the prospective and retrospective mathematical reserves. The model is applied to a sample of contracts drawn at random from a mutual insurance company.

Discrete Time Non-Homogeneous Semi-Markov Processes Applied to Models for Disability Insurance

In this paper, we present a stochastic model for disability insurance contracts. The model is based on a discrete time non-homogeneous semi-Markov process (DTNHSMP) to which the backward recurrence time process is introduced. This permits a more exhaustive study of disability evolution and a more efficient approach to the duration problem. The use of semi-Markov reward processes facilitates the possibility of deriving equations of the prospective and retrospective mathematical reserves. The model is applied to a sample of contracts drawn at random from a mutual insurance company.

Strong Law of Large Numbers for Nonlinear Semi-Markov Reward Processes*

Strong Law of Large Numbers for Nonlinear Semi-Markov Reward Processes*

Strong Law of Large Numbers for Nonlinear Semi-Markov Reward Processes

Strong Law of Large Numbers for Nonlinear Semi-Markov Reward Processes

An Algorithmic Approach to Discrete Time Non-homogeneous Backward Semi-Markov Reward Processes with an Application to Disability Insurance

In this paper semi-Markov reward models are presented. Higher moments of the reward process is presented for the first time applied to in time non-homogeneous semi-Markov insurance problems. Also an example is presented based on real disability data. Different algorithmic approaches to solve the problem is described.

Estimating Software Rejuvenation Schedules in High-Assurance Systems

Software rejuvenation is a preventive maintenance technique that has been extensively studied in recent literature. In this paper, we extend the classical result by Huang et al. (1995), and in addition propose a modified stochastic model to generate the software rejuvenation schedule. More precisely, the software rejuvenation models are formulated via the semi-Markov reward process, and the optimal software rejuvenation schedules are derived analytically in terms of the reward rate. In particular, we consider the two special cases: steady-state availability and expected cost per unit time in the steady state. Further, we develop non-parametric algorithms to estimate the optimal software rejuvenation schedules, provided that the statistically complete (unsensored) sample data of failure time is given. In numerical examples, we compare two models from the viewpoints of system availability and economic justification, and examine asymptotic properties for the statistical estimation algorithms.

Partially observable semi-Markov reward processes

The main objective of this paper is to investigate the conditional behavior of the multivariate reward process given the number of certain signals where the underlying system is described by a semi-Markov process and the signal is defined by a counting process. To this end, we study the joint behavior of the multivariate reward process and the multivariate counting process in detail. We derive transform results as well as the corresponding real domain expressions, thus providing clear probabilistic interpretation.

Partially observable semi-Markov reward processes

The main objective of this paper is to investigate the conditional behavior of the multivariate reward process given the number of certain signals where the underlying system is described by a semi-Markov process and the signal is defined by a counting process. To this end, we study the joint behavior of the multivariate reward process and the multivariate counting process in detail. We derive transform results as well as the corresponding real domain expressions, thus providing clear probabilistic interpretation.

Sojourn times in semi-Markov reward processes: application to fault-tolerant systems modeling

The paper considers fault-tolerant systems with operational periods in which the system works, and repair periods in which the system performs a reconfiguration from a recovering point. The aim is to obtain information about the successive operational periods before a fatal breakdown. In order to be able to deal with arbitrary probability distributions for the holding times in each state of the system, it is assumed that the system is modeled by a semi-Markov process. In this paper the distribution of the n th operational period is given. It often happens that the user wants to make a distinction among the operational states in order to get more detailed information. The use of rewards on the model states is then allowed, i.e. the results are presented for semi-Markov reward processes. An example illustrates this work.

Performability analysis using semi-Markov reward processes

M.D. Beaudry (1978) proposed a simple method of computing the distribution of performability in a Markov reward process. Two extensions of Beaudry's approach are presented. The authors generalize the method to a semi-Markov reward process by removing the restriction requiring the association of zero reward to absorbing states only. The algorithm proceeds by replacing zero reward nonabsorbing states by a probabilistic switch; it is therefore related to the elimination of vanishing states from the reachability graph of a generalized stochastic Petri net and to the elimination of fast transient states in a decomposition approach to stiff Markov chains. The use of the approach is illustrated with three applications. >

Non-homogeneous semi-Markov reward processes and their numerical aspects : R. De Dominicis and R. Manca, ∗Università di Napoli, Italy

Non-homogeneous semi-Markov reward processes and their numerical aspects : R. De Dominicis and R. Manca, ∗Università di Napoli, Italy

Iterative Aggregation-Disaggregation Procedures for Discounted Semi-Markov Reward Processes

The equation v = q + Mv, where M is a matrix with nonnegative elements and spectral radius less than one, arises in Markovian decision processes and input-output models. In this paper, we solve the equation using an iterative aggregation-disaggregation procedure that alternates between solving an aggregated problem and disaggregating the variables, one block at a time, in terms of the aggregate variables of the other blocks. The disaggregated variables are then used to guide the choice of weights in the subsequent aggregation. Computational experiments on randomly generated and inventory problems indicate that this algorithm is significantly faster than successive approximations when the spectral radius of M is near one, and is slower in unstructured problems with spectral radii in the neighborhood of 0.8. The algorithm appears promising for large structured problems, where it can often reduce computational time and main memory storage requirements and offer greater robustness to initial values.

A stochastic theory of pension dynamics

Based on a semi-Markov reward process, a stochastic theory of pension dynamics is developed to characterize the ultimate benefits (and costs) to be derived by a group of similar workers from their career membership in pension plans. This characterization is in terms of a number of basic functions representing the expected value and variability of benefits and costs. Analytical and numerical investigation of these functions provide useful insights into the effects of plan types, vesting rules, coverage, and portability.